Multimodal input interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Images and documents as input
Read the section that teaches thisIf this comes up in an interview
When would you use a multimodal model instead of OCR?
When layout carries meaning: tables, forms, charts, and any scanned or photographed document. It replaces OCR plus layout reconstruction plus extraction with one call, and handles the cases where text extraction loses the structure entirely.
What are the failure modes of image extraction?
Dense tables where values drift between columns, small or low-contrast text where it guesses, counting, and precise spatial relationships. All of them produce confident wrong answers rather than reporting difficulty, so numeric output needs cross-checking.
How does an image become something a language model can read?
It is cut into a grid of patches, typically 14 or 16 pixels square, and each patch is projected through one linear layer into a vector the same width as a text embedding. The sequence of those vectors is what the model attends over, alongside the text. The count grows with area, so doubling resolution multiplies patches by four, and the encoder pools the grid down before the language model sees it.
Then they push further
Roughly what does a page of A4 cost?
Order of a couple of thousand tokens at the resolution most providers cap at, which is several times the prompt around it. Anything above about 1568 pixels on the long edge is downscaled before counting, so a 300 dpi scan costs exactly what a 150 dpi one does.
What gets marked up. Strong answers say patches and say that cost scales with area. Weak answers treat the image as an opaque attachment with a flat price.
How do you control the cost of image input?
Resize before sending, since most documents are readable well below original resolution and image tokens are expensive. Send one page per call rather than batching many, which also improves accuracy.
Speech in and speech out
Read the section that teaches thisIf this comes up in an interview
Design a voice assistant for a restaurant's phone line.
Streaming speech to text, a model with tools for menu, hours and bookings, streaming text to speech, with a latency target of about a second per turn. Handle interruptions, confirm bookings by reading details back, transfer to a person on request or when confidence is low, and log transcripts for review.
Then they push further
Callers keep getting cut off mid-sentence. What is wrong?
End-of-turn detection is too aggressive, treating a thinking pause as the end. Lengthen the silence threshold, or use a model that judges whether the sentence sounds finished, and measure the trade-off against response delay.
How would you evaluate it?
Recorded test calls with scripted scenarios and noise, measuring task success such as correct bookings, per-turn latency, interruption handling, and transcription errors on names and times. Then review a sample of real calls weekly.
What gets marked up. Marked up for a latency budget across streaming stages, turn-taking, and a human fallback.
These are the questions for one topic. The whole question bank covers every topic on the site.