malleability vs AI

I occasionally beg my friend to try Lifelab and give me feedback. I really wanted lifelab to be malleable for end user to be able to do any sort of view or computation on top of their notes. I painstakingly wrote pages of notebook API specs and a tutorial how to write lua and python code. There is dynamic autocomplete in codemirror. I added a coding assistant within the app which also loads the same API specs. There is a way to write renderer code that loads any jsons and gives you charts, we got interactive buttons and basically anything I could think a user would need to write their own mini app without ever leaving lifelab.

Instead, we had this exchange:

Sixhobbits: "Huh, the calendar view on the sidebar is missing. I asked claude to bring it back." me: "Wait-"

That almost made me quit the whole thing, because I realized no malleable system I ever design could match this kind of flexibility and power. I also thought the friction of learning my arbitrary set of APIs is infinitely higher than just prompting claude to modify it without reading a single line of code.

Even non-technical users will soon be able to just describe what they want and agents can either take apart existing code, or just tailor perfect solutions to their needs. What value could this possibly bring them?

Let's put down some different categories of malleability of a note taking app. This is not an exhaustive feature set of an app, which can be richer. It is about the ability for end user to modify their app behavior without external tooling. For example, if you have obsidian and use claude code to modify the raw notes, it is not a part of obsidain.

** Level 1: Closed app The experience is packaged and fully crated by the developer. (Hopefully) they thought of all the use cases their app should do, and more importantly what their app should NOT do. There are different interfaces that might or might not scratch that one itch you have to write. Take it or leave it, it is a finished product with a singular vision

** Level 2: Query and template system There is a integrated way to make templates for repetitive notes, like daily journals. You can compose queries and save them, and maybe even have a live rendering of the query for a quasi dashboard. It is no longer just notes, but things that can create notes. Again, this is still curated and highly polished, with some expected user behavior for the templates.

** Level 3: External plugins The developer exposes an API to access the internal system. There is some sort of contract between the app and the plugins and maybe an SDK or sample project. Ideally, the contract ensures the plugin doesn't have footguns and the API surface allows performant queries on top of the data. Apart from the API surface, plugins are free to do whatever. Obsidian is here.

** Level 4: Scriptable environment Large parts of the apps are exposed in the API, or even large parts of the app use the same API to process content. Users can write arbitrary code that can query notes, or also just skip notes completely and do whatever they want. The constraint is the capability of the programming language but given enough tenacity, they can run DOOM in it. The best examples in my opinion are Tiddlywiki, Trilium notes, Silverbullet or Emacs.

** Level 5: Full source available You have the full source code and there are frankly no restrictions on what modifications you can (or should) do. You can change the font, or the backend, or the sync engine. The moment you have full access to the code, there is no experience the developer can curate and share, because they are competing with your own vision. Your software becomes a bootstrapping theme for whatever they want to make, all your thoughts become an obstacle.

Level 5 was always there, but it was gatekept by the insane difficulty of programming. Guys and gals who could write basic javascript would gloat on stupid normies who had no choice but to beg for features on Github issues. (Those people would also be known as the most ungrateful and borderline toxic audience a open source developer can be exposed to)

Only very dedicated people would send time and effort to learn the behaviors of a scriptable environment to make it their own. The reward was incredible productivity and a sense of freedom and accomplishment I have yet to feel in a corporate setting.

Programming was hard, having taste was harder and the overlap between both was fairly small, which meands you ended up with a lot of more technical apps that end users would struggle using. No wonder Apple design awards are so coveted, the competition was fairly small once you hit a certain pedigree and polish and it's hard to justify spending that kind of effort on open source software.

Even a seasoned software developer has to learn a new programming language to write a obsidian plugin or chrome plugin which could be a painful process even if you know what you are doing. Tinkering is a major time waste, which creates this entire class of (excellent) programmers who only use the most basic tools on default settings.

Designing a railoaded system that allows end users to slightly modify their app without the whole thing collapsing used to be foolish errand, because you could sell them specialized apps instead.

But with advent of Claude Code, the table has flipped irreversibily. Just getting the source is now the easiest possible way to run and modify any software. Claude can just read all the docs (and git commits) and figure out how to run it. It can trace the code and modify the exact code needed and autonomously make sure it builds. It also is patient enough and can patiently parse the dumbest user request, I don't think any developer can compete with that.

I even found at work, a coworker liked a visualization app someone else made and instead of discussing and collaborating together (or making a feature request), he just forked the entire app and started it as his own thing. I was frankly shocked and I suspect this is an increasing reality in software, where any sort of minimal friction can lead to people just taking what you have and re-gpt-it. It is a both humbling and demoralizing realization.

Now, lifelab was supposed to be a level 4 note taking app. I thought I can make the experience of modifying your app more polished than existing approaches (malleability UX) but I suspect AI completely overrode that. No amount of scaffolding will beat a prompt!

I even found myself opening claude code in my github repo, asking it to write a code block, which I then pasted into the app, completely bypassing my intended flow of user writing code (or asking the in-app AI assistant to write code). It sucked.

I did some soul searching, and I could either do closed source from a hidden repository (disgusting), or fully lean into this new AI-malleability world. What I mean is, instead of making user-facing-tooling to modify the app, we should make AI-facing tooling instead. This is somewhat exciting, because it is uncharted territory and a new contract between developers and end users. I want to allow everyone with claude code to modify (...some!) parts of the app, without having to read the source code and rebuild the app.

How how do we allow modifying of the app via some interface, without having to access the source code? Luckily we already have a lot of mechanisms where lifelab queries the database for JSX UI code and dynamically loads it, so as long as an AI agent has access to the database and enough documentation, it should be able to succesfully add new elements. Secondly, the AI agent should know what are looking at right now, and it should be able to gently point us to a different part. I would propose MCP as a solution to both.

I used to push against MCP servers a lot as boomer tech given claude works well with skills (skills + CLI), and I still think some uses of MCP are fairly incorrect, but here is my proposal for a AI-malleability-fist app:

• If you have the desktop app installed, Claude code should be able to read AND modify the entire UI state of the app. If it goes to a page, the UI should change in tandem.
• Claude code (or any other AI agent!) should be able to write, execute and validate code blocks for automation, and author rendering code to attach to blocks which can plot. The App should share sufficient information with via MCP to succeed. (Close to having the source code, without exposing it) It should be able to query blocks, but also decide between making in-app code and do adhoc analysis.
• Lifelab should move as much functionality into this API interface as possible.
• Lifelab should dynamically load whatever frontend UI code is present in the database
• I don't want to maintain an in-app assistant, because users are opinionated about what models they use and they might want to connect it to their daily work, instead of switching between "my flavor" of AI assistant, lifelab should be fully open to whatever AI assistant they are using instead.