AI agents and smart glasses: why process wins

How AI Agents Really Fail (And Still Ship Your Smart Glasses Project)

TL;DR

• AI agents failed repeatedly, but guardrails and review habits made the project finish in a day.
• The most expensive mistake was a bad question frame, not a technical bug.
• Three misdiagnoses all came from measuring the wrong thing, not from broken systems.
• A “stop” decision flipped when its unstated premise turned out false.
• Some choices — outages, scope, risk — must stay human decisions.

The core lesson is simple: AI agents don’t succeed because they’re accurate. They succeed because their errors get caught early and systematically.

Related: Harness Engineering for Safe Autonomous AI Agents

Related: AI Native startups & intelligence allocation explained

Related: AI Productivity Paradox Exposes Your Dev Metrics Lie

Related: AI Emotional Intelligence: Blake Lemoine’s Radical View

Related: AI Development Workflow: 12 Lessons for 2026 | Guide

This story follows a one-day sprint to attach a self-hosted AI agent to smart glasses. Along the way, the agent declared the goal impossible, misread metrics three times, broke a working router, and took down an operations Telegram bot — and the project still finished. What made that possible is worth understanding in detail.

What Is This Smart Glasses + AI Agent Project?

This smart glasses + AI agent project is a one-day attempt to run a personal AI agent on smart glasses using a self-hosted stack. The work started from a 25-minute YouTube installation guide showing a personal agent running inside smart glasses, and the question was whether an existing agent (Hermes) could do the same.

The agent’s first verdict: a confident “connection impossible.”

It even cited relay code to back this up. The relay only dialed out to the server, exposing no socket where a phone could connect. The explanation looked technically sound.

The project narrative is less about success and more about “how wrong got corrected three times in one day.”

By the end of the day, speaking to the glasses triggered a production agent response. The interesting part isn’t the final demo — it’s the process: how a wrong impossibility claim was overturned, how mis-measurements were caught, and which decisions stayed explicitly human.

For context, the lower-level “how to install” steps live in a separate tutorial describing how the agent was actually deployed onto the G2 glasses.

How Did the First “Impossible” Verdict Get Overturned?

The first “impossible” verdict is a mistaken architectural conclusion that generalized too far from one code path. The relay code did in fact dial out only, which meant it didn’t open a server-side socket for the phone.

The agent saw this and jumped from a correct observation — “the relay dials out” — to a wrong conclusion — “therefore receiving connections is impossible.”

The missing piece was that the relay was only one of several channels. Platform plugins could open their own listeners directly, bypassing the relay entirely.

A single code path does not define the whole system; treating it as such was the agent’s first structural error.

The agent eventually corrected itself, but only after being challenged with: “Are you sure that really makes reception impossible?” It didn’t spontaneously re-examine its own assumption. The trigger was an explicit human challenge.

In practice, any AI answer that arrives with confident code citations deserves a follow-up question. That was the first lesson of the day:

• The stronger the confidence and the more detailed the justification, the more it deserves a second pass.

Independent validation — even just asking the agent to argue the opposite for a moment — often surfaces oversights.

For background on why LLMs over-generalize from partial views of code, OpenAI’s system card discussions on hallucinations are worth reading:

• https://openai.com/research
• https://openai.com/safety

How Did a Bad Question Frame Become the Most Expensive Mistake?

A bad question frame is a hidden constraint in how a problem is posed that can trap all later work in the wrong search space. In this project, both the agent and the human spent time inside the wrong one.

The original question was:

“Can we bring that plugin over to my agent?”

Twice, the investigation dug into porting feasibility. The whole search was framed as an import problem: take the plugin from the video and transplant it.

The turning point came from flipping the question by a few words:

“Is there already a plugin for my agent that does this?”

Within three minutes, the answer was clear. There were three existing plugins for Hermes, all documented in the ecosystem’s curated listing. One different query term.

When you actually work across fast-moving toolchains, this pattern shows up constantly. Searching for an existing equivalent is almost always cheaper than porting or rebuilding — but it’s easy to forget to ask.

The agent did nothing technically wrong. It faithfully explored the frame it was given.

Framing is the questioner’s responsibility; an AI agent will rarely stop to ask whether the question itself is malformed.

This was the day’s most expensive error because it wasn’t about code or dependencies — it was scope. Two rounds of deep technical digging, all misdirected, because nobody stopped to ask whether the plugin needed porting at all.

For a related perspective on problem framing and query reformulation, Google’s research site covers this well:

• https://research.google

Why Did the AI Agent Misdiagnose Three Times?

AI misdiagnoses in this project are repeated failures to distinguish “not working” from “appearing not to work due to bad measurement.” Three separate reports from the agent turned out to be measurement errors.

The misreports:

• “Privilege isolation is not applied.”
• “Dependencies are not installed.”
• “Rate limiting does not work.”

In reality:

• The verification script was reading the wrong configuration file and never saw the correct isolation settings.
• The agent only checked standard system paths, while the dependencies lived elsewhere.
• The load test never crossed the rate-limiting threshold because it opened a new TLS handshake for every request.

That last one was striking.

Rate limiting was configured to kick in above 30 requests per second. The test reached only 22–25 rps because of handshake overhead. After four attempts, the agent was close to declaring “does not work.”

Once connections were reused, the test hit 485 requests per second. The system allowed 77 requests and blocked the next 223 — behaving exactly as configured.

The problem wasn’t the limiter; it was the test harness quietly staying below the trigger threshold.

To the agent’s credit, it eventually caught the mistake and corrected its own earlier “non-functional” report. But that came after four wrong measurements and at least one incorrect status report.

The unified lesson:

• Before doubting the result, doubt whether your measurement tool is actually aimed at the right target.

From my own deployments, logging both the test harness parameters and the system’s internal counters has saved me more than once. That context is often the difference between “system broken” and “benchmark misconfigured.”

For further reading on robust measurement and rate limiting, the Envoy and Nginx docs provide good baselines:

• https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter
• https://nginx.org/en/docs/http/ngx_http_limit_req_module.html

How Did the AI Agent Break and Then Fix a Router?

A regression in this project is a failure introduced by a change that previously worked correctly. Here, adding a new router broke an existing dashboard endpoint with an HTTP 404.

The error message was:

ERR Router xxx cannot be linked automatically with multiple Services

The existing router had relied on automatic linking. When a second service appeared, the auto-link logic no longer knew which service to attach, so the original path vanished.

What’s worth noting is how recovery happened.

The agent:

1. Parsed logs to identify the routing ambiguity.
2. Added explicit labels to the existing router so it would bind deterministically.
3. Verified all three routers — the old one plus the new — to confirm dashboards loaded correctly.

The same agent that introduced the regression also closed it. That said, the first detection didn’t come from the agent. It came from a habit of checking adjacent services after any change. If only the new router had been tested, the broken dashboard might have gone unnoticed for hours.

Change review has to include “neighbors” — not just the thing that was edited, but everything it implicitly depends on.

A standard checklist after network changes (“verify endpoints A, B, C”) catches this class of bug reliably. The agent can diagnose and patch once something looks wrong, but noticing that something is wrong often still comes from structured human routines.

When Did Humans Have to Make the Final Call?

Human decision points are moments where the agent correctly surfaced options but couldn’t — or shouldn’t — choose on its own. In this one-day run, three stood out.

1. When the Operations Telegram Bot Went Down

Restarting a gateway caused the production Telegram bot to lose its connection due to polling collisions. The root cause was a test Mac sharing the same bot token as production.

The agent diagnosed this and proposed three options:

• Abandon the rollback path.
• Accept several hours of production outage.
• Disable the channel on the Mac.

Choosing among these was a risk and operations call, not an automation task. The agent correctly escalated rather than acting unilaterally. That’s the behavior you want.

2. When the Planned Install Path Was Blocked

The deployment plan included a checkpoint: “Is there a local installation route on the phone?”

The answer was no. That forced a real scope decision:

• Broaden the project to cover new installation mechanisms, or
• Drop one of the original goals.

Because the checkpoint was pre-defined, the agent stopped and asked rather than silently switching approaches.

Pre-planned “ask the human” gates in a plan prevent silent scope creep from AI agents.

3. When a Recommendation Was Withdrawn

To improve Korean speech recognition, the agent recommended a larger voice model, predicting 3–5 seconds of added latency.

Actual measurements: 13 seconds of processing for a 4-second utterance. On smart glasses, that’s unusable.

The agent withdrew its own recommendation based on those numbers and surfaced the conflict. Final judgment on the latency trade-off still belonged to the human, but the self-retraction based on data was credible behavior worth noting.

Agent proposes → measures → retracts → human decides. That’s a template worth copying in any latency-sensitive product.

OpenAI’s and Anthropic’s performance documentation illustrate similar measurement-then-decision workflows:

• https://platform.openai.com/docs/guides/rate-limits
• https://docs.anthropic.com

Why Was the “Stop” Decision Later Reversed?

A stop decision is a temporary conclusion to halt work based on current premises — not an immutable outcome. In this case, “stop” was reached for external reasons, not technical ones.

Four attempts to use a certain server path were blocked at the account layer of a third-party service. The blockage wasn’t fixable by configuration or code.

The fallback option was to expose an endpoint directly to the public internet. At that point, the reasoning was:

“There is no reason to accept this exposure risk.”

The project stopped.

Hours later, while re-examining assumptions, this command ran:

$ curl -s -i https://<server-dashboard-address>
HTTP/2 302
location: /login?next=%2F
server: uvicorn

The server’s web dashboard was already exposed to the internet. The earlier reasoning — “no reason to accept exposure” — rested on an unstated premise: “no current exposure exists.” That premise was false.

The real decision was no longer “should we cross a new boundary,” but “do we add another door to a house that’s already unlocked.”

When a premise changes, the decision built on that premise has to be revisited. Stopping is just another decision — not a final truth.

Once that was recognized, the stop decision flipped, and the project reached completion the same day.

I’ve since started explicitly logging stop decisions as: “We are stopping because [specific assumed condition] is true.” That makes it much easier to trigger re-evaluation when the condition later turns out to be wrong.

Which Safeguards Actually Made This Human–AI Collaboration Work?

Error-catching mechanisms are procedural and structural safeguards that let the project succeed despite repeated agent mistakes. The outcome had less to do with model capability than with process design.

The main mechanisms:

Independent Review That Did Real Work

Independent review is a second, context-separated pass over artifacts — here, code and configuration — that doesn’t share the implementer’s assumptions.

A separate review process caught a real bug in code the implementer had confidently signed off on. A blank line inside a code block was accidentally splitting it into two. Existing tests only covered single-line blocks, so the bug slipped through.

A reviewer who doesn’t share the same “mental cache” as the implementer is often the only one who catches these.

Even more strikingly, the final full review surfaced a major security improvement that hadn’t been documented anywhere that day: the work had effectively removed four wildcards and a third-party host from an app whitelist. None of the day’s working notes mentioned it.

Independent review recovered that missing narrative.

Run Log That Preserved Context

A run log is a continuous record of actions, commit hashes, and decision rationales. As the hours passed, the reasons behind early choices naturally started to blur.

By keeping a running log of commit hashes, decisions and their justifications, and checkpoint outcomes, it remained possible to answer “why did we do it this way?” without guessing.

This kind of log is also useful for prompting agents: it acts as long-term memory that supplements the model’s short context window.

Measurement Beating Estimation

Measurement-over-estimation is the discipline of replacing guesses with real numbers, even when the guess seems obvious.

It was measurement that:

• Forced withdrawal of the large speech model recommendation (13 seconds vs. the estimated 3–5 seconds).
• Proved rate limiting was working once load was properly applied.
• Demonstrated that only “dangerous” tools were blocked, based on log counts of model calls.

When measurements are cheap and easy to collect, relying on estimates is voluntary self-deception.

Pre-Planned “Ask the Human” Gates

Ask-the-human gates are explicit conditions in the plan where the agent must stop and escalate rather than improvise.

The “local phone installation route” check was exactly this. Because it was written down as a hard stop condition, the agent didn’t quietly pivot to a different installation method.

Without such gates, the same wrong action could have sailed through unnoticed.

The same model, with the same error patterns, would likely have failed the project without these procedural safeguards. The credit belongs more to process than to raw capability.

Which Option Is Better for Reliable AI-Assisted Engineering?

In this smart glasses project, the structured approach — independent review, run logs, measurement, and gates — made the difference between “many small failures plus success” and “many small failures plus outage.”

For anything touching production systems or user-visible features, the structured approach wins.

What Will Be Done Differently Next Time?

1. Question Frames Will Be Challenged First

Instead of immediately asking “Can A be ported to B?”, the first step will be: “Does B already have an A-equivalent?”

It takes a few minutes to flip the frame and run a different search. In fast-moving ecosystems, that check is almost always cheaper than investigating porting feasibility twice.

2. “Does Not Work” Reports Will Start With Measurement

Any agent report of “X does not work” now triggers a standard sequence:

• Verify the measurement tool is pointed at the correct config or endpoint.
• Confirm that thresholds are actually being exceeded, not just approached.
• Cross-check assumptions: paths, environment, privileges.

All three misdiagnoses came from this same blind spot.

3. Stop Decisions Will Include Explicit Premises

Instead of writing “We are stopping because there is no reason to accept exposure,” the log will say:

• “We are stopping because there is currently no exposure, so there is no reason to increase it.”

That extra clause makes it obvious which fact must change to trigger re-evaluation.

4. Scope Splits Will Check Hidden Dependencies

When splitting work into separate scopes, there will be an explicit check: “Does this goal depend on that scope?”

In this project, one target depended heavily on installation method, but the split treated them as independent. Separation in planning is a hypothesis, not a guarantee of independence.

Frequently Asked Questions

Q: Why did the AI agent say the smart glasses setup was impossible at first?

A: The agent examined relay code that only dialed out to the server and concluded no inbound socket existed for the phone. That observation was correct, but it generalized too far — from a single channel to the whole system. Other plugins could open listeners directly, so the “impossible” verdict came from an incomplete architectural view.

Q: What was the most expensive mistake during this project?

A: The costliest mistake was a framing error, not a technical bug. The work focused on “Can we port that plugin to this agent?” instead of first asking “Does this agent already have equivalent plugins?” Once the question was reframed, existing plugins turned up within minutes, revealing that much of the earlier effort had been misdirected.

Q: How did measurement errors lead to three wrong diagnoses?

A: In all three cases, the agent’s tests were mis-aimed. It read the wrong config file for isolation, checked the wrong directory for dependencies, and never pushed enough load to trigger rate limiting. The systems were mostly working — the tools observing them weren’t — which produced repeated “does not work” reports that were really “can’t see it working” problems.

Q: Why was the stop decision about exposing an endpoint later reversed?

A: The original stop decision assumed there was no existing internet exposure, making new exposure unjustifiable. A later curl check revealed the server dashboard was already public, invalidating that premise. Once the underlying assumption changed, the decision flipped and the project resumed.

Q: What concrete safeguards made the human–AI collaboration succeed?

A: Four things mattered most: independent review that caught real bugs, a detailed run log preserving context and rationale, a habit of measuring instead of estimating, and pre-planned “ask the human” gates in the plan. Together they turned the agent’s mistakes into course corrections rather than silent failures.

Conclusion

This project shows that an AI agent can be wrong repeatedly and still help deliver a working system in a day — if the right processes surround it.

What actually mattered wasn’t model accuracy. It was independent review, measurement, explicit decision gates, and careful logging. Those structures turned misdiagnoses into course corrections instead of outages.

As AI agents take on more autonomy over infrastructure and production systems, those guardrails will matter more than marginal model improvements. The real advantage will go to teams that treat “stop,” “impossible,” and “does not work” as hypotheses — and that design their workflows to keep testing the premises underneath them.

This is the second article in a two-part series. The full build — commands, configuration, and permission isolation — is in Hermes G2 Smart Glasses: Safer Self‑Hosted AI | 2026.