devmil
devmil
Making the agents learn
Making the agents learn
As you might have noticed, I have been working on plenty of projects lately.
First the Desktop framework comparison, followed by experimenting with Qt and Rust and then the switch back to Flutter, including a bunch of applications that I was missing for Linux and macOS in the quality I wanted them to be.
All those projects follow the same pattern. Depending on the programming language, I have a set of utilities, a common architecture and a defined way of implementing the applications.
Initially, I did it by just telling the agent: “Here is this other project, please take the architecture, DI system, logging, error handling, translation system and everything that is generic and apply it here” as one of the first steps.
This worked quite well as long as there were 2 or maybe 3 projects. The shared knowledge only lived in the source code and markdown files of specific projects.
On top of that, I noticed that there are classes of errors that the agents make all over again, ranging from simple UI glitches like a wrong animation in custom context menus to more complex issues around multi-window handling.
Those issues started to pile up and I was in desperate need of a shared “agent knowledge” system.
Looking for “off the shelf” solutions
I started to look for a solution that works “out of the box”, but the ones I found were either way too complex for my setup or lacking one basic requirement that I had.
I thought about how a knowledge system can work, be reliable and stay up to date. My biggest concern was agents just dropping knowledge bits from their context into the knowledge system, which would ultimately bloat it and finally make it unusable.
So I was looking for a system that separates the knowledge “finding” from maintaining the knowledge system.
After some back and forth and a couple of evenings discussing with my agent, I came up with a solution that works quite well for me, so I want to share it here.
The knowledge system
The system is quite straightforward. I called it “Foundry” and it lives in its own Git repository, so Git is basically the memory of the system. The knowledge is a structure of markdown files that link to each other. It contains small knowledge bits (e.g. “When you encounter this kind of crash, then here is what the solution has been for other projects”) and bigger things like a “Flutter profile” that contains the mentioned blueprint for how Devmil Flutter projects are structured.
Every agent gets a small hook for the knowledge system that explains how to retrieve information and how to propose additions or changes.
The entry point is an index that only routes to small, focused documents. This way an agent only loads what it actually needs for the task at hand instead of the whole knowledge base.
If an agent comes to the conclusion that whatever knowledge it built might be valuable for other agents or projects as well, it can propose a change to the knowledge system. This is basically a branch that contains a markdown file with a defined structure and content. And that’s it.
To prevent the bloat I was afraid of, an agent is only allowed to propose something if the problem showed up in more than one project or task, if the failure was costly enough to justify a check, or if existing knowledge turned out to be wrong or outdated. Product-specific behavior, personal preferences and workarounds where nobody knows why they work are not welcome.
Here is a real (shortened) example from one of my Flutter apps:
# A post-frame callback does not request a frame
- Scope: `flutter`
- Source: GitLama command-line remote open
- Verification status: `complete`
## Summary
WidgetsBinding.addPostFrameCallback only queues work for the next frame;
it does not schedule one. An effect deferred that way from a stream event,
IPC message or timer in an idle window waits until something else repaints,
possibly never. [...]
## Evidence
A command forwarded from the command line reached HomeScreen through a
broadcast stream. [...] A widget test pumping 12 frames never ran the callback. [...]
## Recommended Foundry change
Extend profiles/flutter/navigation-effects.md [...]
## Validation
The widget test failed with the post-frame deferral and passes with direct dispatch. [...]Another agent then goes through the proposals and decides if this is a valid change / addition, and if it is, it also decides where in the knowledge system this bit needs to be placed. While doing this, the knowledge maintainer agent can also rearrange things and restructure the index if necessary.
A review is final: a proposal is either accepted or rejected, there is no back and forth with the agent that proposed it. If a proposal isn’t solid yet, it gets rejected and a later session can propose it again with better evidence. And I still have the final say: nothing gets merged without my OK.
To make sure that the system works and to see if there is some optimization potential, I added an observability layer that tracks those agent sessions and records how many pages of the knowledge system were loaded and how many of them were actually helpful, while also tracking a rough estimation of tokens used per context. This way I can see if the knowledge system is actually helping the agents to be more efficient and if it is worth the effort to maintain it.
