Speech in and speech out

A voice feature is a chain: audio in, transcription, a model, speech back out. This section is about what that chain does to latency and where it breaks, because people notice a pause longer than about a second and every stage adds one. The two hard parts are streaming every stage so the delays overlap instead of adding up, and deciding when someone has finished speaking, which is a judgement that feels either sluggish or rude whenever it is wrong.

Speech to text#

Transcription models such as Whisper, which is open source, and hosted APIs turn audio into text. They handle clear speech well and struggle with noise, overlapping speakers, heavy accents, and domain words such as product names. Many services can take a list of expected terms to improve those.

The measure is word error rate: insertions plus deletions plus substitutions, divided by the number of words actually spoken. A headline figure of five per cent sounds fine and tells you almost nothing, because the errors are not evenly spread. They concentrate on exactly the words that carry the meaning — names, numbers, dates, product codes — and a transcript that is ninety-five per cent right with the order number wrong is a hundred per cent useless. Measure word error rate separately on the entity types your feature depends on, and treat the overall number as a sanity check rather than a target.

The voice pipeline#

microphone -> speech to text -> model -> text to speech -> speaker
              (streaming)        (streaming)  (streaming)

In conversation, people notice a pause of more than about a second. Each stage adds delay, so every stage has to stream: transcribe while the person talks, start the model as soon as they stop, and start speaking the first sentence while the rest is still being generated.

That is easy to say and worth costing out, because the difference between the two arrangements is not a refinement. Measure everything from the moment the caller stops talking.

# Every number is measured from the moment the caller stops speaking.
ENDPOINT  = 0.7      # silence you wait for before deciding they have finished
STT_TAIL  = 0.3      # transcription still to do once the audio stops
TTFT      = 0.6      # model time to first token
GEN_RATE  = 45       # tokens per second
REPLY     = 60       # tokens in the reply
TTS_TTFB  = 0.25     # time to the first audio sample once text starts arriving
FIRST_CLAUSE = 12    # tokens before the first sentence can be spoken

serial = (ENDPOINT + STT_TAIL + TTFT + REPLY / GEN_RATE
          + REPLY / GEN_RATE + TTS_TTFB)         # TTS waits for the whole reply
streamed = (ENDPOINT + STT_TAIL + TTFT
            + FIRST_CLAUSE / GEN_RATE + TTS_TTFB)

print(f"stage after stage: {serial:.2f}s of silence before the caller hears anything")
print(f"every stage streamed: {streamed:.2f}s")
print(f"same model, same words, {serial - streamed:.2f}s of the difference is pipeline shape\n")

print("where the streamed 2.12s goes:")
for name, cost in [("endpoint silence", ENDPOINT), ("transcription tail", STT_TAIL),
                   ("model time to first token", TTFT),
                   ("first clause generated", FIRST_CLAUSE / GEN_RATE),
                   ("speech synthesis starts", TTS_TTFB)]:
    print(f"  {name:<28}{cost:>5.2f}s {cost/streamed:>6.0%}")
print(f"  {'total':<28}{streamed:>5.2f}s")

# stage after stage: 4.52s of silence before the caller hears anything
# every stage streamed: 2.12s
# same model, same words, 2.40s of the difference is pipeline shape
#
# where the streamed 2.12s goes:
#   endpoint silence             0.70s    33%
#   transcription tail           0.30s    14%
#   model time to first token    0.60s    28%
#   first clause generated       0.27s    13%
#   speech synthesis starts      0.25s    12%
#   total                        2.12s

Two things to take from the breakdown. Streaming is not an optimisation you do later — it is 2.4 seconds, which is the difference between a usable phone line and one people hang up on. And even with everything streamed you are still at 2.12 seconds, over the target, so the remaining work is in the largest line: endpointing at a third of the budget, and it is the only item you change by policy rather than by buying a faster component.

Knowing when someone has finished#

Deciding that the user has stopped talking is harder than it sounds. Wait too long and the system feels slow; too short and it interrupts people who pause to think. Users also interrupt the assistant, so playback must stop immediately when they speak.

A fixed silence threshold trades one failure directly against the other: every hundred milliseconds you shave off the wait is a hundred milliseconds faster and a few more people cut off mid-sentence. The way out is to stop treating it as one number. Silence after a completed sentence means something different from silence after the word my, so use the partial transcript as well as the audio — a clause that sounds finished can be endpointed aggressively, one ending on a preposition or a digit should wait. People reading out a card number or a postcode pause in the middle of it, and that single case accounts for a large share of complaints about being interrupted.

Barge-in is the other half and it is a plumbing problem rather than a modelling one. When the caller speaks over the assistant you must stop playback within about two hundred milliseconds, discard the audio still buffered downstream, and — the part that gets forgotten — truncate the conversation history at the point the caller actually heard. If you keep the whole generated reply in the transcript, the model believes it said things the caller never heard, and every later turn is built on that false record.

Speech-to-speech models#

Some providers now offer models that take audio in and produce audio out directly. They can respond faster and keep tone of voice, at the cost of less control over the text in the middle, which makes logging, checking and evaluating harder.

The trade is worth stating plainly. What you gain is the endpoint silence and transcription tail off the budget, and prosody that survives — hesitation, emphasis, the difference between a question and a statement. What you lose is the text in the middle, and that text is where your guardrails, your logs, your evals and your tool-calling logic all live. A hybrid is common: audio in and out for the conversation, with a transcript produced alongside it for the record, accepting that the transcript is a description of what happened rather than the thing that was checked before it happened.

Cost#

Voice is priced per minute of audio or per audio token, and conversations are long. Estimate cost per call minute early, including silence, which some services bill for.

# Substitute your provider's numbers. The shape is what transfers.
STT_PER_MIN   = 0.006     # dollars per minute of audio transcribed
TTS_PER_MCHAR = 16.00     # dollars per million characters synthesised
IN, OUT       = 3.00, 15.00   # dollars per million text tokens

CALL_MIN      = 6         # a support call
TURNS         = 14        # exchanges in it
HISTORY_GROWTH = 220      # tokens the transcript grows by each turn
SYSTEM        = 1200      # system prompt and tools, resent every turn
REPLY_TOKENS  = 70
CHARS_PER_TOKEN = 4

stt = CALL_MIN * STT_PER_MIN
tts = TURNS * REPLY_TOKENS * CHARS_PER_TOKEN / 1e6 * TTS_PER_MCHAR
sent = sum(SYSTEM + HISTORY_GROWTH * t for t in range(TURNS))
llm_in, llm_out = sent / 1e6 * IN, TURNS * REPLY_TOKENS / 1e6 * OUT

for name, cost in [("transcription", stt), ("model input", llm_in),
                   ("model output", llm_out), ("speech synthesis", tts)]:
    print(f"  {name:<18} " "$" f"{cost:>7.4f}")
total = stt + llm_in + llm_out + tts
print(f"  {'call total':<18} " "$" f"{total:>7.4f}   "
      "$" f"{total/CALL_MIN:.4f} a minute")
print(f"\n{sent:,} text tokens sent across {TURNS} turns for a transcript that ends at "
      f"{HISTORY_GROWTH*TURNS:,} tokens:")
print(f"the history is resent every turn, which is "
      f"{sent/(SYSTEM+HISTORY_GROWTH*TURNS):.1f}x the size of the thing itself.")
print(f"10,000 calls a month: " "$" f"{total*10_000:,.0f}")

#   transcription      $ 0.0360
#   model input        $ 0.1105
#   model output       $ 0.0147
#   speech synthesis   $ 0.0627
#   call total         $ 0.2239   $0.0373 a minute
#
# 36,820 text tokens sent across 14 turns for a transcript that ends at 3,080 tokens:
# the history is resent every turn, which is 8.6x the size of the thing itself.
# 10,000 calls a month: $2,239

The audio is not the expensive part. Transcription and synthesis together are 44 per cent of the call; the single largest line is model input, because the transcript is resent on every turn and fourteen turns of a growing conversation bill 8.6 times the conversation's final size. That is where prompt caching earns its keep on voice, and it is why the instinct to shop around for cheaper transcription is usually aimed at the wrong line of the bill.

What to skip#

Skip building your own voice activity detection; the ones bundled with the streaming APIs are good and the interesting problem is the policy on top, not the detector. Skip evaluating on clean studio recordings, which tell you nothing about a call from a car. And skip the perfect first version: ship with a fast route to a person, measure how often people take it, and use those calls as the test set.