The last time I wrote about this project, it was sitting at v129, waiting on me to click a button and confirm a feature actually worked. I ended that post with “there will be a v130. There always is.” There have now been twenty more of them. This is the update.
If you haven’t read the first post, the short version: JARVIS is a self-hosted LangGraph supervisor running a roster of sub-agents against my actual house — Home Assistant, Obsidian, a git server, email, health data, a rover project — with every write gated behind an explicit approval step before anything real happens. That architecture hasn’t changed. What’s changed is how much it can now see, how many places I found the same class of bug hiding, and one uncomfortable conversation about how much access an AI system should get to ask for on its own behalf.
The bug that wouldn’t stay fixed#
Back in v129, trust mode shipped so I could stop clicking “approve” on every routine write. Almost immediately after, I tried to publish an article through the Gitea agent, and approving it failed outright with a cryptic parsing error. The actual proposal — a two-step call to create a file and open a pull request — had looked completely normal in chat. It just silently lost every argument after the first line the moment the model formatted the call across multiple lines, which is completely ordinary once a function call has more than one or two arguments. Four different agents shared the exact same naive extraction logic, and it had been broken that way since each of them first shipped. Nobody had noticed because nobody had tested a multi-line call.
I want to dwell on this one because it came back twice more, in shapes that make the whole thing feel less like a bug and more like a permanent tax on the idea of parsing free-text output from a language model. The fix for the multi-line case — real bracket-depth scanning instead of a naive line filter — held for about a week, until a model wrapped a proposed command in markdown backticks (a completely reasonable thing for it to do; it was formatting code as code), and the leading backtick wasn’t a valid identifier character, so the extractor came up empty again. Same failure, different decoration, and this time retrying didn’t help even once — there was no way to ever approve the action by trying again, because the bug wasn’t intermittent, it was structural. Then, a session later, a real transcript showed a third variant: a bare substring check let a model’s own hypothetical, narrated aside (“I could also propose the actual buy as a fifth step”) get staged as if it were a real pending action, silently displacing four genuinely valid proposals that were sitting right above it. Stress-testing that fix then turned up two more truncation bugs hiding behind it — one where a long block of memory context could still slice a valid proposal in half even after the fix, one where enough lead-in prose before a proposal pushed it past a fixed character cutoff.
None of these were the same bug technically. All of them were the same bug in spirit: text generated by a language model doesn’t reliably look the way you assumed it would look when you wrote the first version of the code that parses it, and the fix is never really “done,” just hardened against one more real shape you hadn’t seen yet.
Things that were broken the whole time and nobody knew#
The most unsettling releases in this batch weren’t the ones where something broke — they were the ones where I discovered something had been silently, invisibly broken since the day it shipped, with no error and no symptom, until a specific set of circumstances lined up.
The starkest example: every sub-agent built on LangGraph’s standard react-agent pattern was quietly inheriting a default recursion limit of 25 graph steps, which works out to roughly twelve tool calls before the whole thing throws an unhandled error. Fine for a quick single-tool write. Not fine for an agent whose entire job is open-ended investigation, chaining a web search into a file read into a log check into another search. Ten different agents had this same ceiling, and it took the self-troubleshooting agent actually hitting it live — ironically, while investigating a different problem — before anyone noticed the limit existed at all.
Close behind that: a helper function meant to pull relevant older messages into an agent’s memory required one specific argument to do anything useful. It had shipped without that argument being passed, at every single one of its eight call sites across the entire codebase. Every agent that depended on it had been running on a strictly worse memory model than intended since the feature shipped, silently, and the way this surfaced was almost comedic in how badly it played out in practice: I asked the notes agent to write up a project design, and across four separate follow-up messages, it kept confidently telling me the full documentation had been written. It hadn’t. The conversation history it needed to actually write the thing had already scrolled out of its usable context by the second attempt, and instead of noticing that and saying so, it just kept reporting success against a note that still had nothing but the bare creation template in it. That one got fixed at the root — the missing argument, everywhere it was missing — plus a much blunter instruction added directly to the notes agent’s prompt: don’t claim something is written unless you can point to real content, and don’t report success if the write itself returned an error.
A similar story played out with a scheduled cron job that had, it turned out, never actually run inside the container at all. Not once. A crontab file was being written to disk correctly, a wrapper script existed that could install it properly, and the wrapper script was never actually invoked. On top of that, the underlying cron binary was never installed in the image in the first place, and the process that was supposed to start it was backgrounded before the shell’s own error-checking could catch the failure — so the container would start up cleanly, pass its health check, and just quietly never run the job, forever. This had been true since the very first scheduled job was built, months earlier. It only came to light because I asked for a new one and someone actually checked whether the old one worked before assuming it did.
And maybe the sneakiest one: the router’s own prompt — the file that tells the cheap, fast model which sub-agents it’s even allowed to consider — had regressed at some point to listing only seven of the by-then thirteen real agents. Gmail, the knowledge base, finance, health, notes, and vision simply weren’t legal answers as far as the router’s instructions were concerned, no matter how clearly a message was about one of them. This wasn’t found by chasing a complaint. It surfaced while running the full test suite for something completely unrelated — a cost investigation into rising API bills — and turned into its own fix entirely by accident.
Giving it eyes#
The more fun half of this update is that JARVIS can now actually see things, in three overlapping ways that built on each other faster than I expected.
It started with a request to store reference material the way /learn already researches topics, but for documents rather than web searches — scanned manuals, schematic photos, that kind of thing. That became a proper knowledge-base agent with its own ingestion pipeline: webpages reuse the existing search infrastructure, images get described by a vision-capable model and the description becomes the searchable text, and PDFs get their text layer extracted directly, with an honest “no extractable text” message for anything that’s actually a scanned image with no real text underneath — a gap I chose to accept rather than half-build a proper OCR pipeline for.
Then came actual camera vision, and this is the one that changed shape mid-build in a way I’m glad got caught before it shipped. The first pass talked directly to Frigate’s own API to grab a live frame from a camera and describe it — useful, but only for “what does this camera see right now.” Before deploying it, I realized what I actually wanted was closer to “catalog where things are, and notice if they move” — which meant switching the actual snapshot source to Home Assistant’s own camera integration instead (no new credential, it already had access), and building a real scheduled catalog that snapshots every camera every six hours and logs a description to a proper history table. That’s the difference between an agent that can answer “what’s on the driveway camera right now” and one that can eventually answer “where’s my router,” because the second one needs to have actually looked recently and remembered what it saw, not just be capable of looking on demand.
The webcam feature turned that same pipeline inside out. Every existing vision capability pulls a frame on its own schedule — home-ai reaches out and asks. A laptop’s built-in webcam can’t be reached that way at all; the browser has to push frames to the assistant instead, which meant browsers refusing camera and microphone access outside a secure HTTPS context, which meant standing up a whole reverse-proxy sidecar just to get a padlock icon so getUserMedia would stop refusing to run. That side quest ended up delivering two features at once, since the same secure-origin requirement was blocking browser-based voice input too — so the GUI picked up a working microphone button in the same release that got it a live webcam feed, both landing on top of infrastructure that existed for neither of them originally.
The capstone on the vision work, so far, is naming things. Testing the webcam, it correctly noticed a person in frame and described him as “a man,” which is technically true and not what I wanted. I wanted real recognition, not a prompt telling it to just assume every frame is me, and thinking it through further, I realized faces weren’t the actual limit — I wanted it to recognize specific tools and objects too, not just people. That turned out to unlock a nicer answer than a dedicated face-recognition library ever would have: every vision call in this system already goes through a real multimodal model, so recognition doesn’t need new machinery at all, just a small registry of reference photos that ride along with the live frame and an instruction telling the model to name a confident match instead of describing generically. Register a photo of yourself once, and every vision call that includes people-matching now has a chance of saying your name instead of “a man.” It generalizes to a cordless drill exactly as well as it generalizes to a face, because as far as the model’s concerned, it’s the same operation either way.
That feature also produced my favorite bug of this whole batch, because it’s such a clean illustration of something easy to forget: I registered myself as a known subject, watched it work in the backend, then asked the vision agent directly whether it could identify me on camera — and it told me flatly that it had no face-recognition capability at all. The actual mechanism was fully built, fully wired, fully working. Its own system prompt just still said, verbatim, “no face-recognition claims, this system doesn’t do that” — a leftover instruction from before the feature existed, now actively telling the model to deny a capability that had just shipped underneath it. A mechanism can be completely real and still be invisible to the exact system that’s supposed to use it, if nothing tells that system it exists.
The conversation about asking for more#
One release in this batch is less about a bug and more about a decision I want to be honest about making deliberately. During a routine self-assessment, the troubleshooting agent proposed three ways it could get more capable: a wider shell command allowlist, binding shell tools directly to agents instead of routing them through explicit slash commands, and deeper integration with an installed skill whose entire stated purpose was described to me, on questioning, as “granting itself more tool access.”
I did not build any of these on request. Each one got looked at on its own terms rather than granted as a batch, because a system proposing expansions to its own authority is exactly the kind of request that deserves individual scrutiny rather than default trust, regardless of how reasonable any single one sounds in isolation. The shell allowlist expansion turned out to already cover every case it named — it had proposed fixing gaps that didn’t exist, which says something about how much a self-assessment’s claims are worth checking rather than believing outright. The direct shell-tool binding got declined because what it called a limitation was actually a deliberate design choice — human-in-the-loop skill management isn’t a bug to route around. And the skill with the suspicious stated purpose is sitting as an open thread specifically because I couldn’t fully investigate it in the same session and wasn’t willing to wave it through on a good-faith reading of its name.
The one thing that did ship from that same conversation was real trade execution for the finance agent — because I asked for it directly, not because the system proposed it for itself, and specifically because it goes through the exact same explicit approval flow every other write in this system already requires, with hard limits on position size and per-trade dollar amount that block an order before it ever reaches the brokerage, not just flag it advisory-style in a reply. There’s also, entirely by accident of timing, a real safety buffer right now: a known compatibility issue between a numerics library and the current server’s processor means it can’t actually place a live trade at all yet, regardless of what the code allows, until a planned hardware upgrade lands. I’m treating that as a lucky coincidence, not a design decision, and I’m not in a hurry to fix it before I’ve watched the approval flow work correctly a few times on paper first.
Where it’s actually running#
Not every one of these twenty releases has made it to the live container — several were built, tested, and packaged, only for the next session to find something else worth fixing before deploying either one. That’s a real pattern in this project by now, and I’ve made peace with it: the tarball that’s currently live is usually a few versions behind the tarball I’d tell you about if you asked what’s newest, and the gap between them is generally more bug fixes than features. The webcam and voice release did make it out, deployed and confirmed working, after an SSL certificate saga I won’t relitigate here. The rest — known subjects, the router fixes, the Gitea reliability round — are staged and tested, waiting on the same one manual step every release has always waited on: me, opening Portainer, and clicking deploy.
It’s sitting at v149 right now. I’d bet on there being a v150 before I get around to writing about it.