Last time I wrote about this project, JARVIS was sitting at v149 and I said I’d bet on a v150 showing up before I got around to writing about it. It’s at v172 now. Twenty-two releases in about a month, which is a pace that tells you less about how much free time I have and more about how much a system like this surfaces once you actually start living with it day to day.
This update covers more ground than the last two, so I’m picking four threads instead of staying strictly chronological: a bug where one agent genuinely started talking to the user as if it were a different agent, a shared-speaker bug that took five releases and ended in an honest non-fix, a decision to tear up the “narrow specialist for everything” architecture that’s been the backbone of this project since the beginning, and a new agent that actually designs and validates physical parts instead of just answering questions about them.
The agent that forgot which agent it was#
Every conversational agent in this system threads recent turns back into its own message history so it has context across a conversation — ask about a trip, then ask a follow-up, and it remembers what you said two messages ago. That’s been true since early in the project. What I hadn’t thought hard enough about: when the router legitimately switches which agent is handling the conversation between turns — which happens constantly, by design, every time you change subjects — the new agent’s history still contains the old agent’s reply, with nothing marking whose words they were.
Here’s the transcript that made this impossible to ignore:
you: Morning briefing
travel: I appreciate the persistence, Sir, but I'm still the wrong agent for this task...
You'll want to ask the home_assistant agent for your morning briefing...
you: Today's weather
home_assistant: Sir, I don't have weather tools available to this agent. The travel agent is
configured for flight/hotel research, not live weather data... you'll want to ask the
home_assistant agent -- they have access to your local weather sensors...Read that second line again. home_assistant — correctly routed, this time — replied by telling me to go ask home_assistant, about itself, in the third person, while also bringing up a travel agent nobody mentioned. It wasn’t confused about who it was. It was reading the travel agent’s prior reply out of its own message history and continuing from there as if it had said it.
The fix was to tag cross-agent history with a note — “this reply was from a different specialist agent (travel), not you” — whenever a turn’s source agent doesn’t match whoever’s answering now. Turns from the same agent pass through untouched. It’s a plain string instruction, not a structural guarantee, which is the honest caveat in the release notes: it relies on the model actually reading and respecting the tag the same way it respects every other instruction in these prompts.
That same transcript hid a second, worse bug underneath it. Every agent’s capability-boundary redirect — “that’s not my job, ask X instead” — had gotten in the habit of ending with a trailing courtesy question: “Shall I help you with anything travel-related instead?” And the router’s continuity logic treats a question-ending reply as the strongest signal it has: whatever comes next is probably answering that question, so stay with the same agent. A capability redirect isn’t actually a pending question — there’s nothing to answer, I just want my original unrelated request handled — but because it ended in a ?, every short reply after it got glued back onto the wrong agent. That’s how “all voice requests are routing to the travel agent now” happens: not one bad reply, but a self-reinforcing loop with nothing in the transcript that ever breaks it on its own.
The fix was two-layered on purpose: stop agents from ending redirects with a trailing question at all (closing it at the source), plus a regex backstop that refuses to tag a redirect-shaped reply as “awaiting an answer” even if a future prompt slips a question back in. Belt and suspenders, because the first bug already proved that a rule stated once in a prompt isn’t something I can fully trust a live model to never violate.
Five releases chasing a stutter#
This one I’m including because it’s a good example of what debugging looks like when you don’t actually have log access to the thing misbehaving.
The bug: ask JARVIS to play music by voice, the track starts, and then a spoken “Sorry, Jarvis appears to be offline” plays over it and the music stops. The shared speaker is the problem — Home Assistant’s own voice-assistant pipeline uses the same device for music and for its own spoken replies, so whatever text JARVIS returns gets spoken right on top of whatever’s already playing. The existing fix, from months earlier, was to make JARVIS return a genuinely empty string on a successful music turn, so there’d be nothing for the pipeline to speak.
Except an empty string, it turned out, isn’t actually silent on this setup — something downstream treats “the agent sent nothing back” as “the agent failed” and substitutes its own canned apology. v157’s fix was a zero-width space: a string that’s non-empty (survives every falsy check) but renders as literally nothing audible. Shipped, and it didn’t fix it. v160 added a pragmatic mitigation on top — re-issue a media_play command a few seconds after the voice turn, regardless of what’s actually causing the interruption, on the theory that nudging playback back on would paper over the symptom whatever the real mechanism turned out to be. Also didn’t fix it.
By v161 I stopped guessing. Instead of a third blind attempt, that release shipped pure instrumentation: log the speaker’s exact state immediately before the resume command fires, plus whether the command itself actually succeeded. No behavior change at all — just enough visibility that the next live test would produce a real answer instead of another hypothesis. The honest root cause, confirmed afterward by Gavin’s own side-by-side test (identical request typed into the GUI plays fine; the same request spoken gets interrupted every time): the voice satellite hardware’s reference firmware has no audio mixing between music and announcements at all. Any voice-pipeline turn forces the speaker’s audio channel off music and onto announcement/TTS, independent of what text comes back. It’s not a home-ai bug in the sense of broken code — it’s a hardware/firmware limitation that two code-side fixes and a mitigation couldn’t paper over, because the thing causing the interruption was never in home-ai’s control to begin with.
I don’t have a clean ending for this one yet. That’s sort of the point of including it — not every bug resolves into a satisfying fix, and shipping “we now know exactly what’s happening and exactly why our last two fixes didn’t touch it” is sometimes the correct release, even though it feels like admitting defeat.
Fifteen specialists walk into a NAS question#
For most of this project, every request has been routed to one of a growing list of narrow specialist agents — home_assistant, travel, doctor, ops, finance, and so on — each bound to a tight, specific toolset. That’s a deliberate design choice, and it’s worked well for things each specialist is actually built for. It stopped working well the moment questions started drifting outside anyone’s lane.
The transcript that forced the issue: I asked about LoRa garden sensors and a better mesh node for the network. It went to the robotics agent, which — having no better framework to reach for — turned it into a ROS base station design. A question about pricing a 16TB NAS went to ops, which has no web access, so it couldn’t actually look anything up. A planter-box question went to inventory, which answered fine but leaked a raw internal history tag — [Note: this reply was from a different specialist agent...] — straight into its reply, an artifact of the very fix described above. None of these agents declined. They each answered, confidently, from whatever narrow frame they had, which meant the automatic redirect-on-decline mechanism from a few releases back never had a chance to kick in — it only helps when an agent admits it’s the wrong one.
The diagnosis, stated plainly in the release notes: every specialist runs on a comparatively old model with a few turns of trimmed history, so of course a general-knowledge question answered through that narrow a lens comes out worse than just asking a frontier model directly. The fix wasn’t another case-by-case router patch — it was building a new jarvis agent, running a current frontier model with read-only access to web search, the knowledge base, and project context, and making it the default destination for anything that doesn’t specifically need a specialist’s tools. Talking about a topic goes to jarvis now; only actually doing something in a specialist’s domain routes there. It also triggered a long-overdue model refresh across the whole roster — several agents had been sitting on a year-old model generation simply because nobody had gone back and updated the config after the fact.
That reframing kept going. A few days later it became “Jarvis seems less intelligent than Claude or Gemini — how do we fix that” turning into an explicit identity shift: JARVIS is now framed in its own routing prompt as a general-purpose personal assistant, not just a home-automation bot that happens to also answer other things. That came with a real tasks agent for to-dos, reminders, and calendar events (replacing a scattered mess of half-working calendar tools bolted onto the home_assistant agent), the ability to export a real document — a PDF packing list, a DOCX draft — straight from a chat request, and the first deliberate exception to the “voice never writes” rule that’s held since early in the project: a spoken “remind me to check the oven in 10 minutes” or “put the dentist on my calendar Tuesday” now executes immediately, read back for confirmation, because there’s no Approve button to tap when you’re talking to a speaker across the room.
Teaching it to actually build things#
The last thread is the one I’m most pleased with, because it’s the first time this project has produced an agent that outputs something you can hold.
The old freecad agent could run scripts against a 3D modeling tool, but it had no real design discipline — it would execute whatever code it wrote and call it done, once exporting a literal placeholder cube when asked for something more ambitious. It’s been replaced outright by an engineer agent, modeled explicitly on a proper design-review loop: build a model, render it, critique the render, check physical parts against each other for interference, validate the result, and only then hand over files. It also now covers circuit design — SKiDL for netlists, real KiCad commands for schematics, electrical rule checks, and Gerber exports — which means a request like “what resistor do I need for a red LED at 5V and 10mA, and give me a netlist” is now something it can actually do end to end, not just describe.
The validation step is the part I’d call genuinely good engineering discipline: every build gets checked for exactly one solid per part (catching the classic bug where a feature that was supposed to fuse into the main body instead ends up floating, unattached, inside the model), a watertight mesh, and a fit check against the actual bed sizes of the printer and CNC it has to work with. None of this was theoretical — it was build-tested against a real containerized FreeCAD/KiCad/CadQuery image, including an interference check that caught a genuine 0.29mm³ overlap between a lid and a fillet that would have been a frustrating surprise on the print bed. A Workbench tab followed a day later: a real three.js viewer for the resulting STL files, a project browser, a checks panel showing the validation badges, and a scoped chat that only talks to the engineer about whatever project is currently open.
Where it sits now#
A few smaller things are worth a mention without a whole section: JARVIS’s knowledge base and self-authored notes moved off a local SQLite table and onto a proper second-brain tool (Open Notebook), with Obsidian staying as the place I write things by hand rather than where the system writes to by default. And the very last release in this batch, v172, found and fixed a pair of stacked bugs that meant every attempt to have JARVIS create a Home Assistant automation had been silently broken from the start — a schema mismatch, and underneath it, API endpoints that simply don’t exist in real Home Assistant, confirmed against a real instance running in Docker rather than just reasoned about from documentation. Two genuinely separate bugs, both wearing the same disguise: a vague apology from the model that blamed “formatting” for a failure neither bug had anything to do with.
It’s sitting at v172 right now. Given the pace of the last month, I’d be surprised if that’s still true by the time anyone reads this.