Multimodal input

A great deal of what people want processed is not text: scanned invoices, screenshots, forms, slides, a photograph of a whiteboard. A model that takes images can often replace the whole OCR-then-layout-then-extract pipeline that used to be required, and it is far better wherever position on the page carries meaning. This section covers how images are sent and what they cost in tokens, when a multimodal model genuinely beats a text pipeline, and where it is unreliable in the worst way — reading a number confidently that is not there.

How an image becomes tokens#

A language model has no way to read pixels. What happens instead is that the image is cut into a grid of small squares called patches, typically fourteen or sixteen pixels on a side. Each patch is flattened and pushed through a single linear layer, which turns it into a vector of the same width as a text embedding. From there the model does what it always does: attends over a sequence of vectors. The picture has become tokens, and it sits in the same sequence as your words.

import torch
import torch.nn as nn

PATCH, DIM = 14, 1152                 # patch edge in pixels, embedding width

patchify = nn.Conv2d(3, DIM, kernel_size=PATCH, stride=PATCH)

for edge in (224, 512, 1092, 1568):
    image = torch.zeros(1, 3, edge, edge)          # one RGB image, edge x edge
    grid = patchify(image)                          # [1, DIM, edge/14, edge/14]
    patches = grid.flatten(2).transpose(1, 2)       # [1, n_patches, DIM]
    n = patches.shape[1]
    print(f"{edge:>5}x{edge:<5} grid {grid.shape[2]:>3}x{grid.shape[3]:<3} "
          f"= {n:>5} patches  -> tensor {tuple(patches.shape)}")

#   224x224   grid  16x16  =   256 patches  -> tensor (1, 256, 1152)
#   512x512   grid  36x36  =  1296 patches  -> tensor (1, 1296, 1152)
#  1092x1092  grid  78x78  =  6084 patches  -> tensor (1, 6084, 1152)
#  1568x1568  grid 112x112 = 12544 patches  -> tensor (1, 12544, 1152)

Two things follow from those shapes. The count grows with area rather than with the long edge, so doubling the resolution multiplies the patches by four — 224 pixels is 256 patches and 1568 pixels is 12,544. And twelve thousand vectors is more than any model wants to spend on one picture, so a vision encoder pools that grid down before handing it over. The number you are billed for is therefore smaller than the raw patch count and set by the provider, not by this arithmetic. The shape of the relationship is the part that transfers.

What an image actually costs#

Providers publish an estimator rather than an internal patch count. A common shape is area divided by a constant, with anything beyond a long edge of about 1568 pixels scaled down before it is counted. Substitute your own provider's constants; the conclusions below survive it.

MAX_EDGE = 1568        # long edge above which the provider downscales for you
DIVISOR = 750          # published estimator: tokens ~= width * height / divisor
IN_PER_MTOK = 3.00     # dollars per million input tokens -- substitute yours
PROMPT_TOKENS = 700    # the system prompt and the question, as text

def image_tokens(w, h):
    scale = min(1.0, MAX_EDGE / max(w, h))
    w, h = round(w * scale), round(h * scale)
    return w, h, round(w * h / DIVISOR)

print("dpi   submitted    actually sent   tokens   x the text   $/1000 pages")
for dpi in (72, 100, 150, 200, 300):
    w, h = round(8.5 * dpi), round(11 * dpi)          # one US Letter page
    sw, sh, tok = image_tokens(w, h)
    print(f"{dpi:>3}   {w:>4}x{h:<6} {sw:>6}x{sh:<6} {tok:>7,}   {tok/PROMPT_TOKENS:>7.1f}x   "
          f"{tok / 1_000_000 * IN_PER_MTOK * 1000:>9.2f}")

w, h, tok = image_tokens(2550, 3300)
print(f"\none 300 dpi page is {tok:,} image tokens next to {PROMPT_TOKENS} text tokens: "
      f"{tok/(tok+PROMPT_TOKENS):.1%} of the call")
print(f"the same page at 150 dpi costs the same {tok:,}, "
      f"because both were downscaled to {w}x{h} before counting")
lo = image_tokens(round(8.5*100), round(11*100))[2]
print(f"at 100 dpi it is {lo:,}, a {1 - lo/tok:.0%} saving, "
      f"on a page whose body text is still legible")

# dpi   submitted    actually sent   tokens   x the text   $/1000 pages
#  72    612x792       612x792        646       0.9x        1.94
# 100    850x1100      850x1100     1,247       1.8x        3.74
# 150   1275x1650     1212x1568     2,534       3.6x        7.60
# 200   1700x2200     1212x1568     2,534       3.6x        7.60
# 300   2550x3300     1212x1568     2,534       3.6x        7.60
#
# one 300 dpi page is 2,534 image tokens next to 700 text tokens: 78.4% of the call
# the same page at 150 dpi costs the same 2,534, because both were downscaled to 1212x1568 before counting
# at 100 dpi it is 1,247, a 51% saving, on a page whose body text is still legible

Two practical facts come out of that table. Sending a 300 dpi scan buys you nothing over 150 dpi, because it is downscaled before it is counted — you paid to upload it and got the same tokens. And one ordinary page is several times the size of the prompt wrapped around it, so on a call carrying a page and a question the image is around 78 per cent of the input. Rewording the prompt is rounding error. Halving the resolution is not.

Sending an image#

import base64

with open("invoice.png", "rb") as f:
    encoded = base64.b64encode(f.read()).decode()

response = client.chat([{
    "role": "user",
    "content": [
        {"type": "image", "source": {"type": "base64",
                                     "media_type": "image/png",
                                     "data": encoded}},
        {"type": "text", "text": "Extract the invoice number and total."},
    ],
}])

When it beats a text pipeline#

Traditional document processing is OCR, then layout reconstruction, then extraction. A multimodal model can often replace all three, and it is dramatically better on the cases where layout carries meaning.

• Scanned documents and photographs, where there is no text layer at all.
• Tables, which flatten into meaningless runs of numbers in text extraction.
• Forms, where the association between a label and its value is spatial.
• Charts and diagrams, which text extraction simply loses.
• Screenshots, which are increasingly what users send.

The common thread is that the model is reading a layout rather than a string. It is reliable at reading ordinary printed text at a sensible resolution, at finding the value that belongs to a label wherever on the page that label sits, at describing what a chart shows, at reading an interface screenshot well enough to say which button is disabled, and at answering a question about a page without being told where on the page to look. None of those need a bounding box, a template, or a rule about which column the total lives in, and that absence is the whole reason the approach took over.

Where it is unreliable#

• Dense tables with many columns. Values drift between columns and the output looks entirely plausible.
• Small or low-contrast text, where it will guess rather than decline.
• Precise counting of items in an image.
• Exact spatial relationships and measurements.

The pattern in those failures is that the model produces a confident answer rather than reporting difficulty. Anything numeric extracted from an image should be validated against something else: a total that must sum, a format that must match, a range it must fall in.

It is worth being precise about why. A text pipeline that cannot read a character gives you a blank or a replacement symbol, which is a signal. A model always produces its most likely continuation, and for a smudged digit that is a digit. There is no field in the response that says this one was hard, no equivalent of an OCR confidence score, and asking the model how sure it is produces a number that is not calibrated against anything. So the check has to be external, and it has to be arithmetic or a constraint rather than a second opinion.

Document extraction, which is most of the work#

Invoices, receipts, forms, statements, delivery notes and claim documents are the bulk of what gets built with these models. The shape is always the same: an image in, a schema out, and a check that the numbers agree with each other before anything is written down. The check is the part that gets skipped, and it is the only part that catches the failure that matters.

def check(extracted):
    """Every numeric field from an image is a guess until something else agrees."""
    problems = []
    computed = sum(round(r["qty"] * r["unit_price"], 2) for r in extracted["lines"])
    if abs(computed - extracted["subtotal"]) > 0.01:
        problems.append(f"lines sum to {computed:.2f}, "
                        f"subtotal says {extracted['subtotal']:.2f}")
    if abs(extracted["subtotal"] + extracted["tax"] - extracted["total"]) > 0.01:
        problems.append("subtotal + tax does not equal total")
    for r in extracted["lines"]:
        if abs(r["qty"] * r["unit_price"] - r["amount"]) > 0.01:
            problems.append(f"{r['sku']}: qty x unit_price is not the amount shown")
    return problems

extracted = {
    "lines": [
        {"sku": "A-100", "qty": 3,  "unit_price": 12.50, "amount": 37.50},
        {"sku": "B-220", "qty": 10, "unit_price": 4.00,  "amount": 40.00},
        {"sku": "C-337", "qty": 2,  "unit_price": 95.00, "amount": 190.00},
    ],
    "subtotal": 267.50,
    "tax": 53.50,
    "total": 321.00,
}
print("clean extraction:", check(extracted) or "no problems")

# Now the realistic failure: one row came back with the unit price and the amount
# swapped. Column drift, not a hallucination, and every value is a real number
# that appears on the page.
drifted = {**extracted, "lines": [dict(r) for r in extracted["lines"]]}
drifted["lines"][2]["unit_price"], drifted["lines"][2]["amount"] = 190.00, 95.00
for p in check(drifted):
    print("caught:", p)

# clean extraction: no problems
# caught: lines sum to 457.50, subtotal says 267.50
# caught: C-337: qty x unit_price is not the amount shown

Notice what the drifted extraction looks like without the check. Every field is populated, every value appears somewhere on the page, the schema validates, and the total is right. Schema validation passes an extraction that is wrong by 190 pounds, because a schema constrains shape and the failure here is in the values. Cross-field arithmetic is the only thing that catches it, and where the document has no arithmetic to check — a form, a set of addresses — the honest options are a second extraction compared against the first, or a person.

Against a specialist model#

The comparison people expect is that OCR is cheap and a large model is expensive. At current prices that is no longer where the difference lies, and it is worth doing the arithmetic rather than repeating the assumption.

IN, OUT = 3.00, 15.00        # dollars per million tokens in / out. Substitute yours.
OCR_PER_PAGE = 0.0015        # what your OCR vendor charges per page
IMAGE_TOKENS = 1247          # a 100 dpi Letter page, from the table above
TEXT_TOKENS = 650            # the same page once OCR has turned it into text
PROMPT, REPLY = 700, 300     # instructions, and the JSON that comes back

def dollars(tok_in, tok_out):
    return tok_in / 1e6 * IN + tok_out / 1e6 * OUT

direct = dollars(IMAGE_TOKENS + PROMPT, REPLY)
viaocr = OCR_PER_PAGE + dollars(TEXT_TOKENS + PROMPT, REPLY)

print(f"multimodal, page as an image   " "$" f"{direct:.5f} per page   "
      "$" f"{direct*100_000:>8,.0f} per 100k pages")
print(f"OCR first, then the same model " "$" f"{viaocr:.5f} per page   "
      "$" f"{viaocr*100_000:>8,.0f} per 100k pages")
print(f"difference                     {direct/viaocr:.2f}x")

breakeven = direct - dollars(TEXT_TOKENS + PROMPT, REPLY)
print(f"\nOCR has to cost less than " "$" f"{breakeven:.5f} a page to be the cheaper route.")
print(f"At " "$" f"{OCR_PER_PAGE:.4f} it is, by " "$" f"{breakeven - OCR_PER_PAGE:.5f} a page.")
print(f"\nAlso worth noticing: {dollars(0, REPLY)/direct:.0%} of the multimodal call "
      f"is the reply, not the image.")

# multimodal, page as an image   $0.01034 per page   $   1,034 per 100k pages
# OCR first, then the same model $0.01005 per page   $   1,005 per 100k pages
# difference                     1.03x
#
# OCR has to cost less than $0.00179 a page to be the cheaper route.
# At $0.0015 it is, by $0.00029 a page.
#
# Also worth noticing: 44% of the multimodal call is the reply, not the image.

Three per cent. At a hundred thousand pages the whole argument is worth twenty-nine dollars, and 44 per cent of the multimodal call is the JSON coming back rather than the page going in — so the lever on cost is a tighter output schema, not the choice of front end. That reframes the decision entirely. Pick the image route because it handles layout, needs no template per document type and is one component instead of three. Pick OCR because you already have it, because you need character-level coordinates for a highlight overlay, or because the volume is high enough that latency and throughput matter more than accuracy on hard pages.

Latency is the real asymmetry. A page of image tokens has to be prefilled before a single output token appears, so a multimodal extraction of a page typically takes seconds where a specialist OCR pass takes tens of milliseconds. For a batch job overnight that is irrelevant. For anything a user is waiting on, or a queue of millions of pages, it decides the design.

Practical patterns#

One page per call     better accuracy than sending twenty pages
Resize first         large savings, usually no accuracy loss
Schema the output    structure guaranteed, values still need checking
Cross-check numbers  line items must sum to the stated total
Keep the image       so a human can verify a disputed extraction

What to skip: skip fine-tuning a vision model before you have measured what the general one gets wrong, because the answer is usually a prompt or a resolution problem. Skip elaborate image pre-processing — deskewing, thresholding, denoising — which was necessary for OCR and is mostly unnecessary here. And skip sending the highest resolution you have; the table above is the reason.

A voice feature is a chain: audio in, transcription, a model, speech back out. This section is about what that chain does to latency and where it breaks, because people notice a pause longer than about a second and every stage adds one. The two hard parts are streaming every stage so the delays overlap instead of adding up, and deciding when someone has finished speaking, which is a judgement that feels either sluggish or rude whenever it is wrong.

Speech to text#

Transcription models such as Whisper, which is open source, and hosted APIs turn audio into text. They handle clear speech well and struggle with noise, overlapping speakers, heavy accents, and domain words such as product names. Many services can take a list of expected terms to improve those.

The measure is word error rate: insertions plus deletions plus substitutions, divided by the number of words actually spoken. A headline figure of five per cent sounds fine and tells you almost nothing, because the errors are not evenly spread. They concentrate on exactly the words that carry the meaning — names, numbers, dates, product codes — and a transcript that is ninety-five per cent right with the order number wrong is a hundred per cent useless. Measure word error rate separately on the entity types your feature depends on, and treat the overall number as a sanity check rather than a target.

The voice pipeline#

microphone -> speech to text -> model -> text to speech -> speaker
              (streaming)        (streaming)  (streaming)

In conversation, people notice a pause of more than about a second. Each stage adds delay, so every stage has to stream: transcribe while the person talks, start the model as soon as they stop, and start speaking the first sentence while the rest is still being generated.

That is easy to say and worth costing out, because the difference between the two arrangements is not a refinement. Measure everything from the moment the caller stops talking.

# Every number is measured from the moment the caller stops speaking.
ENDPOINT  = 0.7      # silence you wait for before deciding they have finished
STT_TAIL  = 0.3      # transcription still to do once the audio stops
TTFT      = 0.6      # model time to first token
GEN_RATE  = 45       # tokens per second
REPLY     = 60       # tokens in the reply
TTS_TTFB  = 0.25     # time to the first audio sample once text starts arriving
FIRST_CLAUSE = 12    # tokens before the first sentence can be spoken

serial = (ENDPOINT + STT_TAIL + TTFT + REPLY / GEN_RATE
          + REPLY / GEN_RATE + TTS_TTFB)         # TTS waits for the whole reply
streamed = (ENDPOINT + STT_TAIL + TTFT
            + FIRST_CLAUSE / GEN_RATE + TTS_TTFB)

print(f"stage after stage: {serial:.2f}s of silence before the caller hears anything")
print(f"every stage streamed: {streamed:.2f}s")
print(f"same model, same words, {serial - streamed:.2f}s of the difference is pipeline shape\n")

print("where the streamed 2.12s goes:")
for name, cost in [("endpoint silence", ENDPOINT), ("transcription tail", STT_TAIL),
                   ("model time to first token", TTFT),
                   ("first clause generated", FIRST_CLAUSE / GEN_RATE),
                   ("speech synthesis starts", TTS_TTFB)]:
    print(f"  {name:<28}{cost:>5.2f}s {cost/streamed:>6.0%}")
print(f"  {'total':<28}{streamed:>5.2f}s")

# stage after stage: 4.52s of silence before the caller hears anything
# every stage streamed: 2.12s
# same model, same words, 2.40s of the difference is pipeline shape
#
# where the streamed 2.12s goes:
#   endpoint silence             0.70s    33%
#   transcription tail           0.30s    14%
#   model time to first token    0.60s    28%
#   first clause generated       0.27s    13%
#   speech synthesis starts      0.25s    12%
#   total                        2.12s

Two things to take from the breakdown. Streaming is not an optimisation you do later — it is 2.4 seconds, which is the difference between a usable phone line and one people hang up on. And even with everything streamed you are still at 2.12 seconds, over the target, so the remaining work is in the largest line: endpointing at a third of the budget, and it is the only item you change by policy rather than by buying a faster component.

Knowing when someone has finished#

Deciding that the user has stopped talking is harder than it sounds. Wait too long and the system feels slow; too short and it interrupts people who pause to think. Users also interrupt the assistant, so playback must stop immediately when they speak.

A fixed silence threshold trades one failure directly against the other: every hundred milliseconds you shave off the wait is a hundred milliseconds faster and a few more people cut off mid-sentence. The way out is to stop treating it as one number. Silence after a completed sentence means something different from silence after the word my, so use the partial transcript as well as the audio — a clause that sounds finished can be endpointed aggressively, one ending on a preposition or a digit should wait. People reading out a card number or a postcode pause in the middle of it, and that single case accounts for a large share of complaints about being interrupted.

Barge-in is the other half and it is a plumbing problem rather than a modelling one. When the caller speaks over the assistant you must stop playback within about two hundred milliseconds, discard the audio still buffered downstream, and — the part that gets forgotten — truncate the conversation history at the point the caller actually heard. If you keep the whole generated reply in the transcript, the model believes it said things the caller never heard, and every later turn is built on that false record.

Speech-to-speech models#

Some providers now offer models that take audio in and produce audio out directly. They can respond faster and keep tone of voice, at the cost of less control over the text in the middle, which makes logging, checking and evaluating harder.

The trade is worth stating plainly. What you gain is the endpoint silence and transcription tail off the budget, and prosody that survives — hesitation, emphasis, the difference between a question and a statement. What you lose is the text in the middle, and that text is where your guardrails, your logs, your evals and your tool-calling logic all live. A hybrid is common: audio in and out for the conversation, with a transcript produced alongside it for the record, accepting that the transcript is a description of what happened rather than the thing that was checked before it happened.

Cost#

Voice is priced per minute of audio or per audio token, and conversations are long. Estimate cost per call minute early, including silence, which some services bill for.

# Substitute your provider's numbers. The shape is what transfers.
STT_PER_MIN   = 0.006     # dollars per minute of audio transcribed
TTS_PER_MCHAR = 16.00     # dollars per million characters synthesised
IN, OUT       = 3.00, 15.00   # dollars per million text tokens

CALL_MIN      = 6         # a support call
TURNS         = 14        # exchanges in it
HISTORY_GROWTH = 220      # tokens the transcript grows by each turn
SYSTEM        = 1200      # system prompt and tools, resent every turn
REPLY_TOKENS  = 70
CHARS_PER_TOKEN = 4

stt = CALL_MIN * STT_PER_MIN
tts = TURNS * REPLY_TOKENS * CHARS_PER_TOKEN / 1e6 * TTS_PER_MCHAR
sent = sum(SYSTEM + HISTORY_GROWTH * t for t in range(TURNS))
llm_in, llm_out = sent / 1e6 * IN, TURNS * REPLY_TOKENS / 1e6 * OUT

for name, cost in [("transcription", stt), ("model input", llm_in),
                   ("model output", llm_out), ("speech synthesis", tts)]:
    print(f"  {name:<18} " "$" f"{cost:>7.4f}")
total = stt + llm_in + llm_out + tts
print(f"  {'call total':<18} " "$" f"{total:>7.4f}   "
      "$" f"{total/CALL_MIN:.4f} a minute")
print(f"\n{sent:,} text tokens sent across {TURNS} turns for a transcript that ends at "
      f"{HISTORY_GROWTH*TURNS:,} tokens:")
print(f"the history is resent every turn, which is "
      f"{sent/(SYSTEM+HISTORY_GROWTH*TURNS):.1f}x the size of the thing itself.")
print(f"10,000 calls a month: " "$" f"{total*10_000:,.0f}")

#   transcription      $ 0.0360
#   model input        $ 0.1105
#   model output       $ 0.0147
#   speech synthesis   $ 0.0627
#   call total         $ 0.2239   $0.0373 a minute
#
# 36,820 text tokens sent across 14 turns for a transcript that ends at 3,080 tokens:
# the history is resent every turn, which is 8.6x the size of the thing itself.
# 10,000 calls a month: $2,239

The audio is not the expensive part. Transcription and synthesis together are 44 per cent of the call; the single largest line is model input, because the transcript is resent on every turn and fourteen turns of a growing conversation bill 8.6 times the conversation's final size. That is where prompt caching earns its keep on voice, and it is why the instinct to shop around for cheaper transcription is usually aimed at the wrong line of the bill.

What to skip#

Skip building your own voice activity detection; the ones bundled with the streaming APIs are good and the interesting problem is the policy on top, not the detector. Skip evaluating on clean studio recordings, which tell you nothing about a call from a car. And skip the perfect first version: ship with a fast route to a person, measure how often people take it, and use those calls as the test set.