With the advent of LLMs to the general public, I’m scrambling to pick the right tools to capture and structure my data, notes, thoughts, documents etc in a (as) future-proof way (as possible). I just learned the term PKM, and this seems that’s what I’m after. I don’t have as much insight and knowledge as a lot of you here, so please bare with me.
Been using Airtable for a a few years now and love what it has enabled for me. I use it for basically everything, but starting to really see it’s limitations.
My thinking for Airtable right now is: use it mainly for automation, SOPs and storing datasets, but not for taking and organizing documents, notes and thoughts.
Because of that I’m looking for something to complement and ideally cross-communicate/integrate with Airtable really well.
Been looking at some options just know, but anxious about investing to much time in sub-optimal solutions I’ll later regret.
Been looking so far at:
Notion (most user-friendly and widely supported?),
Coda (feels like a direct competitor to Notion but loosing the battle?),
Tana (new, quite technical, graph-based architecture, promising but to much overlap with Airtable?) and
Roam (quite technical, graph based architecture, but questions about staying power and development?).
Something tells me these graph based note-taking app have something that fit’s the A.I. paradigm better. Not sure (at all) if I’m correct - just an uneducated feeling - but I think it comes from seeing Microsoft incorporating a graph in their Office365 A.I. solution too.
Hoping wiser/smarter/more knowledgeable heads have clearer thoughts on the matter!
PKM is a quagmire not unlike the mental sate of Stanley Kubric following a 6-day bender at Coachella with Joaquin Phoenix and Gary Busey.
I’ve written a little about this topic, but I’ll leave you with one parting thought. Imagine a PKM system and process void of any PKM tools. That’s what you probably want. For decades we have assumed there’s an app for PKM. What if there isn’t? What if there shouldn’t be?
Lol, you’re either humble-bragging or you quoted the wrong sentence as it seems you very much do have clearer thoughts on the matter. Btw, did your Grammarly malfunction or is Stanley Kubric someone I should know ;)?
But to the point, thanks for the write-ups you shared. As a complete newbie in a lot of this stuff, I already feel better orientated in this quagmire of PKM by reading them.
About your “parting thought”…… I feel much more at ease now to just pick one of these apps as part of my stack…but……only if I can beta-test FreeFlow…
A lot has happened in AI in the past 90 days. FreeFlow is a victim of the new pace of technology. The premise may still have legs, but the implementation will change dramatically if it happens at all.
Significant tech advances occurred about every 18 to 36 months in the previous decade. In the first year of the current decade, we had three significant advances (one every four months). In the past six months, we’ve had four significant advances (~every six weeks). Innovators haven’t quite adjusted to the new AGI pace.
Ideas that seemed rational or practical just eight weeks ago are irrational or impractical today. On the chance that you might make something better. here’s the source code to both modules.
Create Embedding
// Name: FreeFlow (create embedding)
// Shortcut: command /
// Description: push new content to the vector database
import "@johnlindquist/kit";
import fetch from 'node-fetch';
// configure openai
let { Configuration, OpenAIApi } = await npm("openai")
let configuration = new Configuration({
apiKey: await env("OPENAI_API_KEY"),
})
let openai = new OpenAIApi(configuration)
var cPineconeAPIKey = "<apiKey here>";
var endpoint = "https://<your pinecone project name here>.svc.us-east1-gcp.pinecone.io/vectors/upsert";
let input = await getSelectedText();
if (input) {
//
// get a title of the input
//
let getTitle = async (input) => {
let response = await openai.createCompletion({
"prompt" : "Generate a brief, one-sentence blog post title for this text: " + input,
"model" : "text-davinci-003",
"temperature": 0,
"max_tokens" : 256
})
return response;
}
let titleText = await getTitle(input);
var titleData = await titleText.data.choices[0].text;
//
// get the keywords of the input
//
let getKeywords = async (input) => {
let response = await openai.createCompletion({
"prompt" : "Generate five comma separated keywords without quotes in lower case from this text: " + input,
"model" : "text-davinci-003",
"temperature" : 0,
"max_tokens" : 256
})
return response;
}
let keywordText = await getKeywords(input);
var keywordData = await keywordText.data.choices[0].text;
//
// create embedding and post it to the vector database
//
let embedding = async (embeddingText) => {
let response = await openai.createEmbedding({
"input" : embeddingText,
"model" : "text-embedding-ada-002",
})
return response;
}
//
// get and parse the embedding
//
let embeddingVectors = await embedding(titleText + "\n" + input + "\n" + keywordText);
var data = await embeddingVectors.data.data[0].embedding;
//
// store the embedding
//
// define the vector id
let vectorID = generateVectorID(input);
let content = await editor("VectorID: " + vectorID + "\n\nTitle: " + titleData.trim() + "\n\n" + input.trim() + "\n\nKeywords: " + keywordData.trim());
if (content) {
// create the payload;
let payload = {
vectors: [
{
"id": vectorID,
"metadata" : {
"title" : titleData,
"content" : input,
"keywords" : keywordData
},
"values" : data
}
],
"namespace": "freeflow"
}
// post the new vector
const response = await fetch(endpoint, {
method: 'POST',
headers: {
'Api-Key' : cPineconeAPIKey,
'Content-Type' : 'application/json'
},
'body' : JSON.stringify(payload)
});
const body = await response.text();
setSelectedText(vectorID + ": " + input);
}
}
//
// create vector id
//
function generateVectorID(str) {
var token = (Math.random() * 46656) | 0;
token = ("000" + token.toString(36)).slice(-3);
return token.toUpperCase();
}
Query Embedding
// Name: FreeFlow (create embedding)
// Shortcut: command \
// Description: query content from the vector database
import "@johnlindquist/kit";
import fetch from 'node-fetch';
// configure openai
let { Configuration, OpenAIApi } = await npm("openai")
let configuration = new Configuration({
apiKey: await env("OPENAI_API_KEY"),
})
let openai = new OpenAIApi(configuration)
var cPineconeAPIKey = "<pinecone API key here>";
var cPineconeEndpoint = "https://<pinecone project name here>.svc.us-east1-gcp.pinecone.io/query";
let input = await getSelectedText();
if (input) {
//
// create embedding and query it from the vector database
//
let embedding = async (input) => {
let response = await openai.createEmbedding({
"input": input,
"model": "text-embedding-ada-002",
})
return response;
}
//
// get and parse the embedding
//
let embeddingVectors = await embedding(input);
var data = await embeddingVectors.data.data[0].embedding;
//
// query the embedding
//
// create the payload;
var payload = {
"vector": data,
"topK": 10,
"includeMetadata": true,
"includeValues": false,
"namespace": "freeflow"
}
// get the related vectors
const response = await fetch(cPineconeEndpoint, {
method: 'POST',
headers: {
'Api-Key' : cPineconeAPIKey,
'Content-Type' : 'application/json'
},
'body' : JSON.stringify(payload)
});
const body = await response.text();
var oItems = JSON.parse(body).matches;
// seletion i more than one
var bestResult;
var aMD = getMDResults(oItems);
var md_ = aMD[0];
var aKeywords = aMD[1];
let selector = await div(md(md_));
// await div(md(`# You selected ${selector}`))
if ((selector >= 0) && (selector <= 9)) {
// listed item was selected
bestResult = (selector) ? oItems[Number(selector)].metadata.content.trim() : "";
} else if (selector) {
// keyword was selected
var oKeywordItems = await keywordFilter(selector, oItems);
md_ = getMDResults(oKeywordItems)[0];
selector = await div(md(md_));
bestResult = (selector) ? oItems[Number(selector)].metadata.content.trim() : "";
} else {
// do nothing
}
if (bestResult) {
setSelectedText(bestResult);
}
}
//
// keyword filter
//
async function keywordFilter(selector, oItems) {
var aItems = [];
for (var i in oItems)
{
if ((oItems[i].metadata.title.toString().toLowerCase().indexOf(selector) > -1) ||
(oItems[i].metadata.content.toString().toLowerCase().indexOf(selector) > -1) ||
(oItems[i].metadata.keywords.toString().toLowerCase().indexOf(selector) > -1)
) {
aItems.push(oItems[i]);
}
}
return(aItems);
}
//
// keyword filter query
//
async function keywordFilterQuery(data, keyword) {
// create the payload;
var payload = {
"vector" : data,
"filter" : {"keywords" : {$in : "solar"}},
"topK" : 5,
"includeMetadata" : true,
"includeValues" : false,
"namespace" : "freeflow"
}
// get the related vectors
const response = await fetch(cPineconeEndpoint, {
method: 'POST',
headers: {
'Api-Key' : cPineconeAPIKey,
'Content-Type' : 'application/json'
},
'body' : JSON.stringify(payload)
});
const body = await response.text();
var aItems = await JSON.parse(body).matches;
return(aItems);
}
//
// get md results
//
function getMDResults(aItems) {
var md_ = "";
for (var i = 0; i < aItems.length; i++) {
// get the title
var thisTitle = aItems[i].metadata.title.trim();
// get the content
var aContentParagraphs = aItems[i].metadata.content.toString().split("\n");
var thisContent = aItems[i].metadata.content.trim().replace(/\n/ig, "\n\t");
// get the keywords
var aKeywords = aItems[i].metadata.keywords.trim().split(",");
if (aKeywords) {
var thisKeywords = "";
for (var j in aKeywords) {
thisKeywords += "[" + aKeywords[j].trim() + "](submit:" + aKeywords[j].trim() + "), ";
}
}
md_ += "- ### " + ((aItems[i].score) * 100).toFixed(1) + "%: [" + thisTitle + "](submit:" + i + ")\n\n\t" + thisContent + "\n\n\t" + thisKeywords + "\n";
}
return([md_, aKeywords]);
}
Ahh, that guy! Almost forgot about him. The Sam Altman of the HAL-9000 LLM model. Wonder when the monolith gets discovered.
That probability is not very high. While @Kuovonne feels like a teenager. She is probably in tenth grade, while I’m just starting out in 6th grade not knowing what I want/should do.
I’m looking forward to play around with your code none the less and slowly learn my way around this new tech a bit.
To close off, I’m really appreciating all things you’re generously and vigorously sharing with others around here, and various other places. Really helps me grappling to hold on and achieve a bit of sense-making in all these developments.
