In this article I will chronicle various ways I tried to integrate AI (==LLM) into LifeLab, a personal note taking app, and show ways it failed.

Reasons against AI integration

I was initially strongly against integrating it, because:

1. It is easy and unrewarding from a hobbyist-coder standpoint. Integrating LLM is 1 npm install away and I didn't imagine many interesting vibe-decisions to be had. I did not think I can make a unique take on it so it would just become another tackled on feature. It reminded me of people who install openclaw just so they can claim they work with openclaw, without doing anything interesting with it.
2. It goes against the spirit of taking notes as a thinking tool. Just making LLM do the thinking and planning for you defeats the purpose of writing stuff down in the first place. I feel a lot of "chat with your notes" style integrations just seem like pasting your diary into chatGPT interface and don't have actionable outcomes.

Reasons for integration

I changed my mind for two and a half reasons.

1. If it were a truly malleable app, it should be able to be edited using Claude from within itself without using claude code externally. I really wanted to prove this approach would work and demonstrate a automagical experience where I ask AI to add a visualization from data and it would just do it, no need to copy paste from another chat window.
2. If everything is a block in a database, can we dynamically compose prompts from blocks? Can this system evolve? I thought I could make a interesting spin on LLM from this.
3. Hubris. Maybe I can make something more interesting than the Notion Assistant or CraftDocs assistant chat windows? Maybe my block api design is powerful enough to create something cool? Both companies likely have experienced UX designers, user surveys and analytics data to figure out optimal user flows and needs, but I got taste and enthusiasm. This goes against the first point I had but I am fickle.

The first thing I did was storing the api key serverside, and exposing an API via my notebook object. Then I can call the configured AI from code blocks. Code blocks can already query and edit other blocks, and be installed as hooks to trigger on system and user events. I could make dynamic experiences like guesstimating number of calories from photos by tagging a photo with ":macros:" tag. After adding a tag, a registered codeblock automatically takes the content of the block, an image, and feeds it to an LLM with a hardcoded prompt. Afterwards it updates the block title with a description, and the properties as calories. That worked fairly elegantly. I think a self contained workflow that can be triggered at will is a great use case.

I thought if we polish this a bit more, nb.ai() can be a great first class utility, even though it defeats the purpose of coding a bit. It is so much more convenient than installing huggingface and loading a model locally. Under this block is a photo I had claude automatically describe and try to estimate calories.

Then, I thought I can make an interesting way to chat with your notes. I added a chat button with a dynamic prompt composed of all blocks on the current page. LifeLab has an api surface to also search and edit blocks and I exposed all of those calls as tools to the model, which would be cooler than just "Chat with your notes". The theme was "AI and user use the same api surface to work on notes", but it wasn't so useful because the AI would just query all notes and run out of context. There was a lot of prompting needed to turn this into a good experience.

I also did not like the transient nature of the chats. You can save a chat as a block by tapping a button but by default it disappears on refresh. I felt I am burning tokens and not storing them but that would later turn out to be a hoarding problem.

What truly killed this feature was when my wife told me it looks like a customer support chat. I deleted it immediately afterwards.

Following the previous fiasco, I tried to implement ai as a first class block. LifeLab currently has note blocks for text content, code blocks for.. code, mermaid diagram blocks, html blocks, image blocks, each has a different way to visualize and modify its content. Why not AI block? It would persist the entire conversation, allowing me to continue chats if needed. It would be collapsible and have a sense of belonging by being a explicit block on a page.

It made absolutely sense on a conceptual level, but the experience was horrific, because it would have to fit (the overly verbose) AI output in a fixed height block, so it was a lot of scrolling. There was also no iteration. The AI block can update and create blocks but could not edit them because the prompt was only created and populated at the beginning of the chat, there was no easy way to refresh. Additionally, it kept hallucinating APIs and code that wouldn't work, making it a painful experience.

When I removed the chat field, I did not realize some tasks are transient and I ended up with this workflow where I create AI blocks, prompt them it to create a block with code, and then delete the original AI block, which is even worse than before.

As a minor compromise to avoid the humiliation of customer support chat, I added a inline slash command that updates the existing block only. Since most of the changes I actually want is programming, I added the command to the code block. By typing /ai it shows a small prompt menu to describe my change and then the AI gives me a diff which I can see and confirm.

This was incredibly practical. Usually, each block is a self contained script with no dependencies, so it's all you need to feed to the model. The diff view is useful at a glance, the reject/apply pattern is familiar... it's familiar to claude code. Even in a self-hosted, personal note taking and scripting app, I want claude code. Claude code is the new default state of coding. I implemented a language server and autocomplete to show function calls, but what I really wanted the whole time is to prompt the notebook API spec into claude and have it generate code for me, making all my previous work somewhat pointless.

It's hard to pretend AI-assisted coding is not a thing and stubbornly make a tool that supports old fashioned coding, when it's everywhere. So I leaned into it. We now have ghetto claude code in the code blocks which are no longer used for coding, just for prompting claude to write code for us.

Finally, I have had massive problems maintaining the API documentation. Since it's vibe generated and vibe verified, it often ended up incorrect and drifting apart. Claude would imagine and describe functionality that was not based on the real API behavior. Every time I found a fix, I would need to rebuild the rust code, redeploy docker and wait for my app to relaunch.

One major help was to turn the hardcoded documentation into text blocks, and aggregate them using tags. These are now dynamic and I can change the documentation from within the app without having to relaunch.

Finally, one useful AI-adjacent feature is a transcription block by using soniox speech API. I never heard of them before and initially wanted to use eleven labs before, because some of my coworkers went there, but ElevenLabs had notably worse API and did not do translation. Soniox is working really well! I use transcription blocks to transcribe AND translate chinese text for learning purposes. The resulting transcriptions can also be mined using code blocks to make flashcards, and I can implement a spaced repetition system within LifeLab to memorize them.

In conclusion, I learned a bit about AI use different patterns and by avoiding the customer service chat slop, I stepped into a worse puddle of compressing the entire thing into a tiny block and ended up realizing the chat pattern has value and use for times you need to quickly edit things.

I learned not all conversations require saving and that I should not hoard information for no reason. After this reflection, I ended up with a new pattern -> Embed AI inline exactly where it is needed the most. No need for a separate block or chat window. This scopes the capabilities and expectations better.

I also learned that claude-code-like coding is the absolute future and no one cares about my stupid autocomplete suggestions with code snippets, when they can just prompt entire codebases into existence.

I still wonder if there is value in a archivist / historian style agent that maintains your notes, but I have not yet found a compelling use cases worth obsessing over.

I have also added 1 example of the system prompt that is collected and composed into the system and can be modified during runtime. it describes calling AI from code blocks.