Every sufficiently ambitious home automation project eventually asks the same dangerous question: what if the house just had an AI running it? Mine is called Home-AI internally, JARVIS to everyone who talks to it, and — briefly, after a persona rename I still haven’t fully forgiven — FRIDAY for a while. It’s a self-hosted LangGraph supervisor with a roster of sub-agents underneath it, running in Docker on a Proxmox box, fronted by a custom chat HUD and wired into Home Assistant’s voice pipeline. It controls lights and alarms, reads and writes my Obsidian vault, plays music, patches files in my self-hosted Gitea, drafts and sends email, tracks a rover project through a ROS sandbox, watches my health vitals via my phone’s sensor data, and — as of the last release — remembers to research topics on a schedule when I ask it to get smarter about something.
None of that shipped at once. What follows is the story of roughly a month of releases, numbered v117 through v129, and what actually breaks when you hand an LLM a supervisor role over a house full of real systems.
The shape of the thing#
Before the war stories, the architecture, because it explains almost every bug that follows. A router — a cheap, fast model — looks at each message and decides which sub-agent should handle it: home_assistant for smart-home control, notes for Obsidian, ops for the server itself, robotics for the rover, gitea for the git server, gmail for email, finance, medic, doctor for self-troubleshooting, remote for other machines on the network. Every agent that can write anything — flip a switch, commit a file, send an email — doesn’t get to just do it. It emits a WRITE_INTENT line describing exactly what it wants to do, I approve or decline it in chat, and only then does a separate, deterministic dispatcher actually execute it. No LLM sits in the write path itself. That single design decision is the reason a hallucinating model has never actually broken anything in my house — it can lie to me in conversation, but it can’t act on the lie without my say-so.
Voice is even more locked down: anything spoken through Home Assistant’s pipeline is read-only by design, full stop, regardless of what mode the rest of the system is in.
That safety architecture held up. Almost everything else got found the hard way.
The agent that wasn’t there#
The first real lesson showed up at v118, and it’s the one that kept resurfacing for the next month. I asked JARVIS, in plain conversation, whether it could access my Obsidian notes. It could — the notes agent has real, working tools for exactly that. But the answer I got back was a fabricated “obsidian CLI,” invented vault contents, and at one point a claimed SSH read of a config file that doesn’t exist. Three different sessions, three different confident lies.
The actual bug was almost embarrassingly small: notes had a working tool and a working dispatch entry, but nobody had added its name to the router’s own prompt or its code-level agent allowlist. As far as the part of the system responsible for picking an agent was concerned, notes simply didn’t exist. So a plain-language question about Obsidian got routed to whichever agent seemed closest — ops, doctor, home_assistant — and each of them, having no real tool for the job, just answered anyway. Not maliciously. Not even inconsistently with how language models behave by default. They were asked a question, they had no mechanism to say “that’s not my job,” so they generated a plausible-sounding answer instead.
Fixing the immediate bug was one line in an allowlist. Fixing the pattern took longer, and it recurred twice more before it was actually closed. A follow-up audit at v118 found the same kind of drift in three other places: a planner with its own separate agent roster that had quietly fallen out of sync, a naming collision guard missing four agent names entirely (meaning a custom agent could have silently overwritten the real Gitea integration with no warning), and a dormant crash bug in custom-agent dispatch that had just never been triggered by anything real. Every one of these was the same root cause wearing a different hat: a system with six or seven places that all need to agree on “which agents exist,” and no single source of truth enforcing that they do.
The real fix didn’t land until v125, and it’s the one I’m proudest of in the whole series. Instead of patching the router again, I added a static “capability boundary” block to the shared prompt prefix every single agent already loads before it does anything else — a plain statement of what each agent’s domain actually is, with an instruction to say so and name the right agent instead of guessing when a request clearly isn’t its job. One change, inherited by every agent automatically, closing the hallucination path at its source rather than chasing it symptom by symptom. There’s now a regression test that fails loudly if a new agent ever gets added to the router without also being named in that boundary block — the same lesson, finally enforced by machinery instead of memory.
Small router, big consequences#
Not every routing bug was about missing agents. At v119 I asked a follow-up question about a document — something like “I take it you mean rover and not river” — and it got routed to the robotics agent, purely because the word “rover” appeared in it. The router was seeing each message in total isolation, with zero awareness that a conversation was already in progress. One overlapping keyword was enough to knock it sideways.
The fix was giving the router the same memory recall every sub-agent already had, plus an explicit instruction: if a message reads like a short correction or follow-up to whatever the previous turn was about, stay put unless something clearly signals a new topic. Small fix, but it’s the kind of thing that only shows up once you’re actually living with the system day to day — nobody designs a router test case around “what if the user says a word that means two different things depending on context,” until they say exactly that sentence at 9pm and watch it go sideways.
A smaller, funnier version of the same problem hit at v124: I asked to troubleshoot “the MCP server,” meaning Home Assistant’s own built-in Model Context Protocol integration. The ops agent, having no concept of what that phrase meant, guessed it was a Docker container, picked the only host it knew about, and searched for something that has never existed. Two-line fix, same lesson: an agent that doesn’t know what a phrase means will guess rather than say so, unless something specifically tells it not to.
The dependency pin that took down voice control#
Not every incident was a hallucination. At v123, voice control broke outright — every spoken command started failing with an “internal dependency error” buried in an MCP tool-discovery call. The proximate cause was almost farcical: a Python package called mcp shipped a genuine stable 2.0 release, restructuring internals that a library one layer down depended on without ever putting an upper bound on the version it would accept. A routine pip install silently walked onto the new major version and broke a code path that every single voice command runs through, with no warning anywhere in the build.
This wasn’t even a new class of mistake — the exact same shape of bug had already taken down a different dependency chain back at v97. The fix, again, was small: pin the transitive dependency with an upper bound, the same way its neighbor already was. But the lesson is the one worth keeping — an unbounded floor pin (>=x, no ceiling) on anything you don’t control isn’t a fix you make once, it’s a liability that resurfaces the moment that package’s own dependencies move. It’s now called out explicitly in the project’s running list of gotchas, specifically so I check every transitive dependency the next time I pin something, not just the one that broke.
The refresh button that ate a Friday#
My favorite bug in the whole series, because it took two releases to actually fix, and the first attempt was a completely correct diagnosis of the wrong scope. I reported that refreshing the chat window kept replacing my live conversation with old content from the previous Friday — “I think it hit a limit,” was my read on it, and that was exactly right. Voice-tagged turns were being restored on every page load with a count cap but no time bound, and since voice gets used sparsely, the same fifteen-turn-old block from Friday kept winning that cap indefinitely.
That got fixed at v127 — add a time window, problem solved, or so it seemed. I redeployed and reported almost immediately that it was “deleting chat history again on refresh.” The v127 fix had been completely correct about the bug it targeted and had missed the actual complaint: typed chat had never been restored on page load at all, by original design — the code comment literally said it should behave “the same as typed turns already have,” which was true, and also not what anyone actually wants from a chat window. Refreshing the page had always wiped whatever was actively being discussed; v127 just changed what stale content replaced it with. The real fix, at v128, was building a proper unified “recent chat turns” endpoint covering both typed and voice history together, with the voice-only poll re-seeded from it so nothing got duplicated once live polling resumed.
The lesson generalizes past this one bug: when someone describes a symptom in vague terms — “it’s resetting,” “it hit a limit” — verify the fix actually covers the full scope of what’s visibly wrong before calling it closed. The first plausible mechanism you find and fix might genuinely be a real bug, and still not be the bug.
Giving it more hands, carefully#
Between the incident responses, the feature list grew steadily, and always with the same write-approval discipline applied to whatever got added. v120 gave the Gitea agent an actual file-write tool — before that it could open issues and comment, but never get code onto a branch, so “project partner” stopped at triage. v121 added a full Gmail integration, OAuth2 refresh tokens and all, defaulting to drafts over sends because a bad draft is an inconvenience and a bad send is not. v122 added /learn — track a topic, get a real synchronized web-research pass immediately, and a standing weekly refresh after that, specifically because I’d once asked what areas JARVIS could research to get better at helping me, gotten a list back, agreed with it, and watched nothing actually happen. Now agreeing with the list triggers the research.
The most recent release, v129, is the one that finally addressed something that had been quietly annoying me the whole time: every single write, no matter how routine, needed its own individual approval click. I asked for a way to approve a class of actions for a while rather than one at a time. What I got is a time-boxed trust window — thirty minutes by default, always expiring, never a permanent toggle — that auto-approves writes from the three agents doing routine, recoverable work (ops, Gitea, notes), while explicitly hard-exempting destructive operations even during an active trust window: no auto-approved rm, no auto-approved reboot, no auto-approved note deletion, regardless of trust state. Two domains were left out of trust mode entirely on purpose — the robotics agent, because it can move actual hardware, and the self-modification agent, because it edits JARVIS’s own source. Neither was part of what I asked for, and both felt like the wrong place to introduce an unprompted default toward less friction.
Building that feature also surfaced a bug that had apparently existed since the very first release of the planning feature: approving a multi-step /plan had been crashing with a bare TypeError on every single attempt, because the function backing it only ever accepted one argument while everything calling it passed three. Nothing had ever tested the actual approval path, so it shipped broken and stayed broken for who knows how many releases until building something adjacent to it forced a direct reproduction.
What a month of this actually teaches you#
Reading back through thirteen releases in a row, the pattern isn’t “AI systems are unreliable,” which is both true and useless as a lesson. It’s narrower than that: a multi-agent system is only as trustworthy as the agreement between all the places that need to agree on what each agent can do, and that agreement rots by default unless something enforces it. Every hallucination in this whole series traces back to a gap between what an agent could actually do and what the rest of the system believed about it — and every real fix was either closing that specific gap, or building something (a regression test, a shared prompt block, a set-equality check) that makes the next gap fail loudly instead of quietly.
The write-approval boundary is the other half of why any of this was safe to run against a real house in the first place. Agents are allowed to be wrong in conversation. They are never allowed to be wrong in execution without a human in the loop first — and thirteen releases of hallucinated tools, misrouted requests, and one genuinely production-breaking dependency incident later, that boundary is the one thing that never had to be patched.
It’s currently sitting at v129, waiting on me to actually go trigger a write and click “Approve & Trust” to confirm the amber pill shows up like it’s supposed to. There will be a v130. There always is.