Asked about Speech in and speech out
Design a voice assistant for a restaurant's phone line.
An answer that works
Streaming speech to text, a model with tools for menu, hours and bookings, streaming text to speech, with a latency target of about a second per turn. Handle interruptions, confirm bookings by reading details back, transfer to a person on request or when confidence is low, and log transcripts for review.
Then they push further
Callers keep getting cut off mid-sentence. What is wrong?
End-of-turn detection is too aggressive, treating a thinking pause as the end. Lengthen the silence threshold, or use a model that judges whether the sentence sounds finished, and measure the trade-off against response delay.
How would you evaluate it?
Recorded test calls with scripted scenarios and noise, measuring task success such as correct bookings, per-turn latency, interruption handling, and transcription errors on names and times. Then review a sample of real calls weekly.
What gets marked up
Marked up for a latency budget across streaming stages, turn-taking, and a human fallback.
Reading an answer is not being able to give one. The section that teaches this is Speech in and speech out, 3 hours of Multimodal input.
Learn this properly