Images and documents as input

A great deal of what people want processed is not text: scanned invoices, screenshots, forms, slides, a photograph of a whiteboard. A model that takes images can often replace the whole OCR-then-layout-then-extract pipeline that used to be required, and it is far better wherever position on the page carries meaning. This section covers how images are sent and what they cost in tokens, when a multimodal model genuinely beats a text pipeline, and where it is unreliable in the worst way — reading a number confidently that is not there.

How an image becomes tokens#

A language model has no way to read pixels. What happens instead is that the image is cut into a grid of small squares called patches, typically fourteen or sixteen pixels on a side. Each patch is flattened and pushed through a single linear layer, which turns it into a vector of the same width as a text embedding. From there the model does what it always does: attends over a sequence of vectors. The picture has become tokens, and it sits in the same sequence as your words.

import torch
import torch.nn as nn

PATCH, DIM = 14, 1152                 # patch edge in pixels, embedding width

patchify = nn.Conv2d(3, DIM, kernel_size=PATCH, stride=PATCH)

for edge in (224, 512, 1092, 1568):
    image = torch.zeros(1, 3, edge, edge)          # one RGB image, edge x edge
    grid = patchify(image)                          # [1, DIM, edge/14, edge/14]
    patches = grid.flatten(2).transpose(1, 2)       # [1, n_patches, DIM]
    n = patches.shape[1]
    print(f"{edge:>5}x{edge:<5} grid {grid.shape[2]:>3}x{grid.shape[3]:<3} "
          f"= {n:>5} patches  -> tensor {tuple(patches.shape)}")

#   224x224   grid  16x16  =   256 patches  -> tensor (1, 256, 1152)
#   512x512   grid  36x36  =  1296 patches  -> tensor (1, 1296, 1152)
#  1092x1092  grid  78x78  =  6084 patches  -> tensor (1, 6084, 1152)
#  1568x1568  grid 112x112 = 12544 patches  -> tensor (1, 12544, 1152)

Two things follow from those shapes. The count grows with area rather than with the long edge, so doubling the resolution multiplies the patches by four — 224 pixels is 256 patches and 1568 pixels is 12,544. And twelve thousand vectors is more than any model wants to spend on one picture, so a vision encoder pools that grid down before handing it over. The number you are billed for is therefore smaller than the raw patch count and set by the provider, not by this arithmetic. The shape of the relationship is the part that transfers.

What an image actually costs#

Providers publish an estimator rather than an internal patch count. A common shape is area divided by a constant, with anything beyond a long edge of about 1568 pixels scaled down before it is counted. Substitute your own provider's constants; the conclusions below survive it.

MAX_EDGE = 1568        # long edge above which the provider downscales for you
DIVISOR = 750          # published estimator: tokens ~= width * height / divisor
IN_PER_MTOK = 3.00     # dollars per million input tokens -- substitute yours
PROMPT_TOKENS = 700    # the system prompt and the question, as text

def image_tokens(w, h):
    scale = min(1.0, MAX_EDGE / max(w, h))
    w, h = round(w * scale), round(h * scale)
    return w, h, round(w * h / DIVISOR)

print("dpi   submitted    actually sent   tokens   x the text   $/1000 pages")
for dpi in (72, 100, 150, 200, 300):
    w, h = round(8.5 * dpi), round(11 * dpi)          # one US Letter page
    sw, sh, tok = image_tokens(w, h)
    print(f"{dpi:>3}   {w:>4}x{h:<6} {sw:>6}x{sh:<6} {tok:>7,}   {tok/PROMPT_TOKENS:>7.1f}x   "
          f"{tok / 1_000_000 * IN_PER_MTOK * 1000:>9.2f}")

w, h, tok = image_tokens(2550, 3300)
print(f"\none 300 dpi page is {tok:,} image tokens next to {PROMPT_TOKENS} text tokens: "
      f"{tok/(tok+PROMPT_TOKENS):.1%} of the call")
print(f"the same page at 150 dpi costs the same {tok:,}, "
      f"because both were downscaled to {w}x{h} before counting")
lo = image_tokens(round(8.5*100), round(11*100))[2]
print(f"at 100 dpi it is {lo:,}, a {1 - lo/tok:.0%} saving, "
      f"on a page whose body text is still legible")

# dpi   submitted    actually sent   tokens   x the text   $/1000 pages
#  72    612x792       612x792        646       0.9x        1.94
# 100    850x1100      850x1100     1,247       1.8x        3.74
# 150   1275x1650     1212x1568     2,534       3.6x        7.60
# 200   1700x2200     1212x1568     2,534       3.6x        7.60
# 300   2550x3300     1212x1568     2,534       3.6x        7.60
#
# one 300 dpi page is 2,534 image tokens next to 700 text tokens: 78.4% of the call
# the same page at 150 dpi costs the same 2,534, because both were downscaled to 1212x1568 before counting
# at 100 dpi it is 1,247, a 51% saving, on a page whose body text is still legible

Two practical facts come out of that table. Sending a 300 dpi scan buys you nothing over 150 dpi, because it is downscaled before it is counted — you paid to upload it and got the same tokens. And one ordinary page is several times the size of the prompt wrapped around it, so on a call carrying a page and a question the image is around 78 per cent of the input. Rewording the prompt is rounding error. Halving the resolution is not.

Sending an image#

import base64

with open("invoice.png", "rb") as f:
    encoded = base64.b64encode(f.read()).decode()

response = client.chat([{
    "role": "user",
    "content": [
        {"type": "image", "source": {"type": "base64",
                                     "media_type": "image/png",
                                     "data": encoded}},
        {"type": "text", "text": "Extract the invoice number and total."},
    ],
}])

When it beats a text pipeline#

Traditional document processing is OCR, then layout reconstruction, then extraction. A multimodal model can often replace all three, and it is dramatically better on the cases where layout carries meaning.

• Scanned documents and photographs, where there is no text layer at all.
• Tables, which flatten into meaningless runs of numbers in text extraction.
• Forms, where the association between a label and its value is spatial.
• Charts and diagrams, which text extraction simply loses.
• Screenshots, which are increasingly what users send.

The common thread is that the model is reading a layout rather than a string. It is reliable at reading ordinary printed text at a sensible resolution, at finding the value that belongs to a label wherever on the page that label sits, at describing what a chart shows, at reading an interface screenshot well enough to say which button is disabled, and at answering a question about a page without being told where on the page to look. None of those need a bounding box, a template, or a rule about which column the total lives in, and that absence is the whole reason the approach took over.

Where it is unreliable#

• Dense tables with many columns. Values drift between columns and the output looks entirely plausible.
• Small or low-contrast text, where it will guess rather than decline.
• Precise counting of items in an image.
• Exact spatial relationships and measurements.

The pattern in those failures is that the model produces a confident answer rather than reporting difficulty. Anything numeric extracted from an image should be validated against something else: a total that must sum, a format that must match, a range it must fall in.

It is worth being precise about why. A text pipeline that cannot read a character gives you a blank or a replacement symbol, which is a signal. A model always produces its most likely continuation, and for a smudged digit that is a digit. There is no field in the response that says this one was hard, no equivalent of an OCR confidence score, and asking the model how sure it is produces a number that is not calibrated against anything. So the check has to be external, and it has to be arithmetic or a constraint rather than a second opinion.

Document extraction, which is most of the work#

Invoices, receipts, forms, statements, delivery notes and claim documents are the bulk of what gets built with these models. The shape is always the same: an image in, a schema out, and a check that the numbers agree with each other before anything is written down. The check is the part that gets skipped, and it is the only part that catches the failure that matters.

def check(extracted):
    """Every numeric field from an image is a guess until something else agrees."""
    problems = []
    computed = sum(round(r["qty"] * r["unit_price"], 2) for r in extracted["lines"])
    if abs(computed - extracted["subtotal"]) > 0.01:
        problems.append(f"lines sum to {computed:.2f}, "
                        f"subtotal says {extracted['subtotal']:.2f}")
    if abs(extracted["subtotal"] + extracted["tax"] - extracted["total"]) > 0.01:
        problems.append("subtotal + tax does not equal total")
    for r in extracted["lines"]:
        if abs(r["qty"] * r["unit_price"] - r["amount"]) > 0.01:
            problems.append(f"{r['sku']}: qty x unit_price is not the amount shown")
    return problems

extracted = {
    "lines": [
        {"sku": "A-100", "qty": 3,  "unit_price": 12.50, "amount": 37.50},
        {"sku": "B-220", "qty": 10, "unit_price": 4.00,  "amount": 40.00},
        {"sku": "C-337", "qty": 2,  "unit_price": 95.00, "amount": 190.00},
    ],
    "subtotal": 267.50,
    "tax": 53.50,
    "total": 321.00,
}
print("clean extraction:", check(extracted) or "no problems")

# Now the realistic failure: one row came back with the unit price and the amount
# swapped. Column drift, not a hallucination, and every value is a real number
# that appears on the page.
drifted = {**extracted, "lines": [dict(r) for r in extracted["lines"]]}
drifted["lines"][2]["unit_price"], drifted["lines"][2]["amount"] = 190.00, 95.00
for p in check(drifted):
    print("caught:", p)

# clean extraction: no problems
# caught: lines sum to 457.50, subtotal says 267.50
# caught: C-337: qty x unit_price is not the amount shown

Notice what the drifted extraction looks like without the check. Every field is populated, every value appears somewhere on the page, the schema validates, and the total is right. Schema validation passes an extraction that is wrong by 190 pounds, because a schema constrains shape and the failure here is in the values. Cross-field arithmetic is the only thing that catches it, and where the document has no arithmetic to check — a form, a set of addresses — the honest options are a second extraction compared against the first, or a person.

Against a specialist model#

The comparison people expect is that OCR is cheap and a large model is expensive. At current prices that is no longer where the difference lies, and it is worth doing the arithmetic rather than repeating the assumption.

IN, OUT = 3.00, 15.00        # dollars per million tokens in / out. Substitute yours.
OCR_PER_PAGE = 0.0015        # what your OCR vendor charges per page
IMAGE_TOKENS = 1247          # a 100 dpi Letter page, from the table above
TEXT_TOKENS = 650            # the same page once OCR has turned it into text
PROMPT, REPLY = 700, 300     # instructions, and the JSON that comes back

def dollars(tok_in, tok_out):
    return tok_in / 1e6 * IN + tok_out / 1e6 * OUT

direct = dollars(IMAGE_TOKENS + PROMPT, REPLY)
viaocr = OCR_PER_PAGE + dollars(TEXT_TOKENS + PROMPT, REPLY)

print(f"multimodal, page as an image   " "$" f"{direct:.5f} per page   "
      "$" f"{direct*100_000:>8,.0f} per 100k pages")
print(f"OCR first, then the same model " "$" f"{viaocr:.5f} per page   "
      "$" f"{viaocr*100_000:>8,.0f} per 100k pages")
print(f"difference                     {direct/viaocr:.2f}x")

breakeven = direct - dollars(TEXT_TOKENS + PROMPT, REPLY)
print(f"\nOCR has to cost less than " "$" f"{breakeven:.5f} a page to be the cheaper route.")
print(f"At " "$" f"{OCR_PER_PAGE:.4f} it is, by " "$" f"{breakeven - OCR_PER_PAGE:.5f} a page.")
print(f"\nAlso worth noticing: {dollars(0, REPLY)/direct:.0%} of the multimodal call "
      f"is the reply, not the image.")

# multimodal, page as an image   $0.01034 per page   $   1,034 per 100k pages
# OCR first, then the same model $0.01005 per page   $   1,005 per 100k pages
# difference                     1.03x
#
# OCR has to cost less than $0.00179 a page to be the cheaper route.
# At $0.0015 it is, by $0.00029 a page.
#
# Also worth noticing: 44% of the multimodal call is the reply, not the image.

Three per cent. At a hundred thousand pages the whole argument is worth twenty-nine dollars, and 44 per cent of the multimodal call is the JSON coming back rather than the page going in — so the lever on cost is a tighter output schema, not the choice of front end. That reframes the decision entirely. Pick the image route because it handles layout, needs no template per document type and is one component instead of three. Pick OCR because you already have it, because you need character-level coordinates for a highlight overlay, or because the volume is high enough that latency and throughput matter more than accuracy on hard pages.

Latency is the real asymmetry. A page of image tokens has to be prefilled before a single output token appears, so a multimodal extraction of a page typically takes seconds where a specialist OCR pass takes tens of milliseconds. For a batch job overnight that is irrelevant. For anything a user is waiting on, or a queue of millions of pages, it decides the design.

Practical patterns#

One page per call     better accuracy than sending twenty pages
Resize first         large savings, usually no accuracy loss
Schema the output    structure guaranteed, values still need checking
Cross-check numbers  line items must sum to the stated total
Keep the image       so a human can verify a disputed extraction

What to skip: skip fine-tuning a vision model before you have measured what the general one gets wrong, because the answer is usually a prompt or a resolution problem. Skip elaborate image pre-processing — deskewing, thresholding, denoising — which was necessary for OCR and is mostly unnecessary here. And skip sending the highest resolution you have; the table above is the reason.