Your voice agent sounds fine. Look at what it wrote down.

Two lines from a call log, same second:

06:48:36  [HEARD] 'Keep talking.'
06:48:36  [FSM] record_answer stage=10_payment_check payment_arrangement='plan'

The caller said "Keep talking." The record says they agreed to a payment plan.

The obvious reading is that a model made a mistake, and models make mistakes, so who cares. What matters is what that row looks like afterwards.

It looks completely normal. payment_arrangement='plan' is a valid value in a field that expects exactly that value. Nothing downstream flags it, because there is nothing to flag. Open the record a week later and you see a student who chose a payment plan.

The lie is well formed, which is why none of the things I was watching caught it.

I found it by accident. I was chasing a different bug, one you can hear, where the agent would go quiet and then apologise for something that hadn't happened. Diagnosing that meant instrumenting the call properly for the first time, and the whole of it came down to one identifier. I stamped the same call ID on the speech-recognition event and on the write that followed it. Before that I had two logs. After it I had a transcript sitting beside its own record. That's the only reason those two lines ended up next to each other on my screen.

I wasn't looking for this. I don't think I'd ever have gone looking for it, because the channel I was watching was my ears.

Here is the fuller excerpt, because the shape of it matters more than the single line:

06:48:08  [HEARD] 'Keep talking.'
06:48:08  [FSM] record_answer stage=10_payment_check deposit_confirmed='no'
06:48:16  [SAID] "And for the remaining amount, will you be going on a payment
                  plan, or paying upfront?"
06:48:36  [HEARD] 'Keep talking.'
06:48:36  [FSM] record_answer stage=10_payment_check payment_arrangement='plan'

The same two words, twenty-eight seconds apart, produced two different fabricated financial commitments. The second was built on the premise of the first. The agent decided the deposit was unpaid, then asked how the balance would be handled, then decided that had been answered too.

Once I had the two columns side by side, the count went up. Across two calls:

One of those landed conservatively and the other invented a payment plan nobody had agreed to. The direction was luck either way: the same mechanism writes 'yes' exactly as readily as 'no'.

The people on these calls were me or a tester. "Real call" reads as "real customer", so I want the difference on the record. This is a pre-production system for a regulated training provider, and no student gets called until the data agreement is signed. That is what testing is for.

Note what the caller was doing in every one of those rows. Speaking. Just not answering. The audio was real, the transcription was broadly right, and the model still produced a value that had no source in the words.

That failure has a name in the literature, and it sits in the search-agent work rather than anywhere near voice: Contextual Information Policy Optimization for Search Agents calls it prior-driven reasoning, where the model confirms rather than retrieves. It answers from the shape of the conversation instead of from the words in it. Asked whether the deposit was confirmed, my agent produced the answer that a call going well would have.

It took me two months to be able to say what was actually wrong, and when I finally did it came out as a warning rather than a lesson. The words were: the excess guard and framework you end up building are there "to help fix something that you may have caused with the way you are trying to validate what you've got out of a user."

They say you create your own demons. In coding, sometimes you do.

What follows is what I found, in the order I found it.

I picked a live speech to speech model on purpose. Audio in, audio out, one model, no pipeline in the middle. It's fast, it's easy to stand up, and I believed it sounded better than anything I could assemble myself. My working belief was that it could run the whole call script end to end without mistakes, and that this was the shortcut.

That turned out to be wrong, in the way the log above shows. So I put a state machine underneath it to make the call repeatable, and I put checks around the fields that mattered.

Then the calls got slower, and worse.

The reason took me a long time to see, and I had it slightly wrong even when I got there. My first version of this story was that my checks added a transcription step and a wait. The wait was already there. The framework holds a turn open until the transcript finalises, and it always had.

What changed is that I could no longer skip it.

A live model doesn't need the text to answer you. It hears audio and it speaks audio. What I had missed is that the framework underneath it was waiting for that transcript anyway, on every single turn, and had been the whole time.

So I didn't add latency. I lost the right to ignore it. That distinction took me a fortnight and an argument with my own logs, and it's the one I'd want a reader to have for free.

I did not measure any of this. For weeks the whole judgement was a mood: the agent "felt worse than before".

There is a real argument on the other side of this. The production voice AI field is moving fast and most of it is aimed squarely at the half of the trade I gave up. An AWS session, Voice Agents That Handle Interrupts, is built around 200 millisecond response budgets. Latent Space ran a panel in August called Forward Deployed: Voice AI on what works in 2026, on latency budgets, tool reliability and handoff design. If your agent's job is to answer questions and sound good doing it, that work is the frontier and none of this applies to you. My agent's job was to come away with a set of facts a human being would act on.

There are three levels of strictness on the fields the agent captures, and they are better read as different evidence requirements than as a quality ladder:

That reads like a design. Each level exists because something specific broke, and they arrived one at a time, in the order the damage did. Written up as a considered hierarchy it would have looked better and taught you nothing, because the useful information is which failures forced which level.

The shape of the strict check, in the only form that is mine to publish:

# ILLUSTRATIVE ONLY. This is not my implementation, which isn't public.
# The shape is the part worth copying, not the code.

def may_write(field, value, utterance, turn, agent_playback):
    if field.tier is not STRICT:
        return True

    # 1. Is this the caller, or my own voice coming back at me?
    #    Decided on timing, not on words: a turn that opens while
    #    the agent is still playing audio cannot be an answer to it.
    if began_during(turn, agent_playback):
        return False

    # 2. Does the caller's polarity actually match the value?
    #    "no" and "not really" must never become deposit_confirmed = yes.
    if not polarity_agrees(value, utterance):
        return False

    # 3. Are the content words there at all, including numbers
    #    written either way ("two" and "2")?
    if not content_words_support(value, utterance):
        return False

    return True

Three checks, and the first one has nothing to do with language. I'll come back to that.

The part that genuinely surprised me came later. A paper from August, Decomposed Entailment for Factuality Checking and Hallucination Detection, describes the same structure from the other direction: break generated content into atomic claims, check each against the source, and aggregate so that a single confidently contradicted claim flips the whole verdict. One atomic claim is one field. The source is what the caller said. The asymmetric aggregation is my strictest level, where one contradiction refuses the write.

I had not read it. The value here is the convergence rather than any insight of mine: I got there by being burned repeatedly.

The part I got wrong, and it is probably costing you too

Everything above sorts fields by how much a wrong value hurts. That axis is real. The one that would have saved me the time is a different one.

All five strict fields are packaged and handed to the CRM when the call ends. That is what they are for. The legislative obligation is that the student answered the question. The second obligation, that my system verify the answer while they were still on the phone, is one I invented for myself and nobody imposed.

The transcript at the top of this article refutes the easy version of that claim. Look at it again. The agent records the deposit as unpaid, and its very next question is "for the remaining amount". That field steered the turn. So at least one of the five genuinely does need to be right before the call can continue, and I would not have been able to tell you which ones without going and reading the transcripts, because I never asked.

Those are two different obligations, and I spent seconds on the difference between them without noticing. The recording is retained. The transcript is retained. Most of those fields could have been extracted and corroborated after the call, at no conversational cost at all.

The axis I should have sorted by was whether the call has to change course because of the value.

Recording consent is the exception, and it earns its seconds. You can't lawfully continue a call on an unconfirmed consent, and if it fails you need to know now rather than in a batch job at midnight. A payment arrangement does not have that property, and I treated them the same.

Where I think this goes

Two months in, my read is that live models sound extraordinary and charge you for it, and that I could not get repeatable calls and naturally sounding ones out of the same box. What I think works better is to stop buying the audio-native behaviour whole and to synthesise it instead: read the caller's tone and emotion, generate explicit emotion direction, and hand that to a good text-to-speech model.

Since I drafted this I have moved the voice across. It sits in the deployed agent now and has taken nothing but test calls. The first thing I learned is that I had the difficulty in the wrong place. The text-to-speech model runs the same voice as the live one I was running. It's her voice, and the timbre I thought I was buying was portable all along.

What doesn't come across for free is the judgement. Whatever the live model was doing to decide when to slow down or warm up, it was doing without me, and I never had to write any of it down. It arrives when I write it, as tags, ahead of time.

That sounds like a downgrade and I think it's the opposite. Her warmth was a behaviour I couldn't see, couldn't measure and couldn't correct. As tags it's an artifact: I can read it, diff it, and be wrong about it in a way that shows up.

I'll tell you it works when I have measured it. Everything above is a record of what my impressions of this system were worth.

The worst thing I found was not in the packaging fields.

The strictest level of checking exists to protect consent. Those are the fields with scripted wording, a compliance review attached, and a specific legal weight. And they were being answered by the agent itself.

Follow the arrows and look for the point where the check could have known. In the version that shipped, there was not one. By the time the audio reached the transcription step it had no origin attached, so the guard was asked to judge the meaning of a sentence that had already lost the only property that mattered, which is who said it.

That is the first check in the code above, and it had been built for a different bug. It works on timing: a caller turn that begins while the agent is still playing audio is treated as contamination, whatever the words in it turn out to be. That costs me the eager caller who says "okay" over the top of the question, and on this one field I would rather ask them again than record a consent they may not have given. Note where the fix had to live. Underneath the language layer, which had no way to tell.

On a call that has to be recorded lawfully, the system was capable of generating its own permission to record it, in its own voice, while the person on the other end said nothing at all. It reached every hard consent field.

It was a flaw in the build.

Three qualifications belong in the same breath as that sentence. It was pre-production. No student was ever called. It has been found and it has been fixed.

But I don't want to make it sound smaller than it was, and I especially don't want to tell you it was easy to catch. It ran for weeks. It was found by accident while I was chasing something else, and I have caught myself since describing it as an obvious bug. It became obvious the moment I saw it, which is a completely different property.

Consent is the field you'd assume is safe. It has a script. It has a reviewer. You can hear it being asked correctly on the recording. Everything about it looks defended.

Now back to that first check, the one with nothing to do with language. Echo caused a second bug that looks nothing like this one. Before any of the consent work, it was making the agent appear deaf: her own audio read back as the caller speaking, and my watchdog concluded a turn had gone unanswered and made her apologise for it. I then shipped a fix, and the fix passed. When I measured it properly, thirty of the thirty two cases it had cleared as "she answered" had the supposed caller turn beginning within a hundred milliseconds of the agent's own speech, median about ten. That is a machine hearing itself, quietly validating a repair that did not work.

Two different diseases, then. The packaging fields were fabricated from real speech that did not answer the question. The consent fields were fabricated from the agent's own voice. From the outside, in the record, they are indistinguishable, which is the only reason I found them at the same time.

I should also say what happened when I shipped the fix for the first one. The root cause was that the input transcription was hallucinating and nothing constrained it, so I pinned the recogniser's language. The parameter I used is supported on one of the two API modes and raises an exception on the other, which is the one I was running. It shipped at 07:14 and I found it at 07:35 when I tried to make a test call and she never spoke. Not degraded. Silent, on every call.

That sent me back to the architecture, and I spent the next stretch genuinely unsure whether the state machine was the right way to run a voice agent, and whether the live model was the right way either.

There is a fair objection to all of this, which is that verification layers have their own problems. Mitigating Scoring Bias in LLM-as-a-Judge is fairly damning about position and length bias, and Evidence Lock Before Commitment shows that a judge locked to a frozen view of the evidence gets worse rather than better. That criticism doesn't miss. My answer is narrow: what I built is a deterministic comparison between the words the caller said and the value about to be written, and it does not get an opinion. That does not make it correct. It fails in a way I can trace.

There's a hard limit on how long the model will hold a session. On an eleven minute call, mine hit it at about nine minutes, and the reconnection path was rejected every time it tried. The caller waited eighty seconds in silence and then hung up. Orientation calls are scripted for fifteen to thirty minutes, so that is a ceiling.

I solved it. The stored handle that was supposed to resume the session had been rejected on every retry, so the fix was to stop trusting it: when the warning arrives, a fresh session is opened and the conversation so far is replayed into it.

The handover takes long enough that the caller experiences it as a long pause, which is to say it feels exactly like the failure it fixes. It's still sitting on my desk.

I said earlier that I never measured the cost, that I only felt it. That was not quite true, and the truth is more uncomfortable.

I had a measurement the whole time. It told me everything was fine.

I would run a test and get gaps of about a second. Then the same changes would go to production and I would sit through four or five seconds of silence waiting for the agent to answer. Going back through the logs to write this, I found something else recorded. From the first clean full call: normal turns running 0.9 to 1.9 seconds from the caller finishing to her voice starting, and three slow turns at 3.7, 3.8 and 7.1 seconds.

Which is to say the rig, the register and my memory were describing three different things, and I had been reading them as one number.

What I carried away from those calls was the tail. The median is what the dashboard carried away. Nine good turns and one seven second silence is experienced as an agent that hangs, and my summary statistics were quietly averaging away the only turns anyone would remember.

There is a second reason the numbers disagreed, and it is structural. The gap is measured from the caller's speech ending to the agent's first audio. Written down like that it sounds unambiguous. It isn't, because the caller's speech ending is a decision the system makes. Everything before that decision sits outside the number: the voice detector holding on to be sure you had finished, the endpointing delay, and on a call routed over SIP, transcripts arriving anywhere from six to fourteen seconds late.

My first instinct was the wrong one, and it's probably yours too. I assumed my measurement was broken, so I went and improved my measurement. That's a satisfying kind of problem to have, because you can fix it. It didn't close the gap, because the missing time was not in a component I owned. It only became visible once I had the platform's own logs.

I did find real defects in the test rig, eight of them, and they are worth knowing about because they are funny in the way only your own work can be. The clip pack I used to simulate a caller had been calibrated against clips that were themselves artificially degraded with the same recipe, which is a simulation calibrated against a simulation. A sort order meant every run I thought was using the new twelve clip pack was actually playing four retired clips first and only reaching the new ones at turn five, and it was invisible because the label in the results was synthesised rather than read from the filename. The recording path captured about 0.43 of real time, so my test audio was running fast and roughly an octave high.

All real, all fixed, and every one could've been fixed with the four seconds still unaccounted for. A broken instrument is the reassuring version of this story: you find the bug, you fix it, your numbers are true again. My rig was not broken in the way that story needs. It was doing its job on a stimulus that was never the thing I was worried about.

Where all of that landed, in the harness's own record, with the caveat that the harness was being repaired across the same period: from zero of six runs passing and dying at turn two, to five of eleven and then four of six runs passing, 94.6 to 96.8 per cent of turns OK, and false apologies from six to zero. The agent is deployed on Google Cloud Run in the Sydney region and takes real telephony calls. It doesn't yet run a full orientation end to end. The first clean full call reached stage eight of eighteen in 4.7 minutes, and the session ceiling in the previous section is the reason.

There is one more layer, and it's the one I didn't expect to be writing down.

All that instrumentation existed for a reason, and the reason was that I was building this with an AI coding assistant and I couldn't trust what it told me. It would report a fix as working. Sometimes the fix worked for a different reason than the one it gave me. Sometimes it hadn't worked at all. So I stopped reading the reports and started measuring the system directly, working backwards from whatever I could observe.

Which is precisely what I had to do with the voice agent, and for the same reason: a report I could not check, arriving in a form that gave me nothing to check it against. I had that problem twice, at two different layers of the same project, and I didn't notice they were the same problem until I sat down to write this.

Three fixes, three prices, and it took me until now to notice I had paid all of them in the same currency.

Making the record trustworthy put a wait that had always been there onto the path the caller sits in, and I pay some of that today, though less than I thought and less than I needed to. Blocking the fabrications cost a completely silent agent for twenty one minutes, which I fixed and do not pay any more. Surviving the session limit costs a pause that reads as the bug it repairs, and I pay that one in full.

Three prices, all of them billed to the same account: how the call feels to the person on the phone. Not money, not compute, not engineering time.

Which brings me back to the two lines at the top. When this system was broken, I could not hear it. Now that parts of it are fixed, I cannot hear that either, because a correct handover and a failed one sound the same from the caller's seat.

The ear is useless in both directions.

The check that finally found mine took one call. I put the transcript beside what the agent had written down and read the two columns together, starting with the field that would have embarrassed me most if it were wrong. It's also the check I should have run in week one, on a system I was proud of, for a reason I couldn't have given you at the time.