Validating at the boundary
By the end: Untrusted data becomes trusted objects in exactly one place in your system.
- Where validation belongs
- Parsing model output safely
- Schemas as the single source of truth
- Failing usefully
New to typing & validation? A type says what kind of value something is: a number, a piece of text, a record with particular fields. Validation is code that checks a value really has that shape before anything downstream trusts it. In Python, type hints write the expectation down and a library such as Pydantic enforces it while the program runs, turning an unchecked blob of data into an object whose fields are guaranteed to exist. Why it matters
There are a few points in any system where data crosses in from somewhere you do not control, and a model's reply is now one of them. This section is about checking it in exactly one place per edge and handing typed objects to everything inward of that, so nothing further in has to wonder whether a field is there. Scatter those checks and they drift apart, which is how one codebase ends up with three incompatible ideas of what a missing value means.
One place, at the edge#
Every system has boundaries where data arrives from somewhere you do not control: an HTTP request, a file, a database, a model response. Validate at each boundary and pass typed objects inward. Validating deeper means checking the same thing in several places, and those checks drift apart.
HTTP request -> validate -> typed object -> business logic
Model output -> validate -> typed object -> business logic
Past the arrow, nothing needs to ask whether a field exists.Model output is untrusted input#
Treat a model response exactly as you would a form submission from a stranger. It is usually well formed and occasionally is not, and the failure should be caught where it arrives rather than three functions later.
from pydantic import BaseModel, Field, ValidationError
class Extraction(BaseModel):
invoice_number: str
amount: float = Field(gt=0)
currency: str = Field(min_length=3, max_length=3)
due_date: str | None = None # genuinely optional
def extract(text: str) -> Extraction:
raw = call_model(text)
try:
return Extraction.model_validate_json(raw)
except ValidationError as e:
raise ModelOutputError(f"could not parse extraction: {e.errors()}") from eThere are four distinct ways that call can go wrong, and they want different handling. Knowing which one you have is the difference between a retry that works and a retry loop that burns money reaching the same answer.
What went wrong What it looks like What to do
not JSON at all json_invalid, "Expecting value" strip prose, then retry
JSON, wrong shape missing / extra field retry with the error
right shape, impossible value amount: -5, currency 'POUNDS' retry with the error
right shape, plausible, wrong invoice_number: 'INV-0001' validation cannot helpThe first row is common enough to handle rather than retry. A model asked for JSON frequently returns JSON wrapped in a sentence of explanation, or inside a fenced code block, and both parse as invalid. Strip that before validating, and reserve the retry for cases where there is genuinely no JSON in the reply.
import json, re
FENCE = chr(96) * 3 # the triple backtick a model wraps JSON in
def extract_json(text: str) -> str:
"""Return the first complete JSON value in a model reply, as a string."""
t = text.strip()
fenced = re.search(FENCE + r"(?:json)?\s*(.*?)" + FENCE, t, re.S)
if fenced:
t = fenced.group(1).strip()
start = min((i for i in (t.find("{"), t.find("[")) if i != -1), default=-1)
if start == -1:
raise ValueError("no JSON object or array in model output")
obj, _end = json.JSONDecoder().raw_decode(t[start:]) # ignores trailing prose
return json.dumps(obj)
extract_json('Here is the JSON:\n{"amount": 12.5}\nHope that helps.') # {"amount": 12.5}
extract_json(FENCE + 'json\n{"amount": 12.5}\n' + FENCE) # {"amount": 12.5}
extract_json("I am not able to help with that.")
# ValueError: no JSON object or array in model outputStructured output does not remove the boundary
Constrained decoding and tool-use schemas make malformed JSON rare, and they change nothing about the other three rows. The value can still be out of range, the enum can still be a plausible-looking member you did not define if the schema allowed a free string, and nothing at all checks that the invoice number is the one on the invoice. Validate anyway: it costs microseconds and it is the only place a wrong value can be stopped.
One schema, two jobs#
The same model definition can generate the JSON schema you send to the API and validate the response that comes back. That is worth more than it sounds: the contract and the check cannot disagree, because they are the same declaration.
schema = Extraction.model_json_schema()
response = client.generate(prompt, response_schema=schema)
result = Extraction.model_validate_json(response.text){"properties": {"invoice_number": {"title": "Invoice Number", "type": "string"},
"amount": {"exclusiveMinimum": 0, "type": "number"},
"currency": {"maxLength": 3, "minLength": 3, "type": "string"},
"due_date": {"anyOf": [{"type": "string"}, {"type": "null"}],
"default": null}},
"required": ["invoice_number", "amount", "currency"],
"title": "Extraction", "type": "object"}Two things follow. Give every field a description, because that text is carried into the schema and is the only instruction the model gets about what the field means; amount: float with no description is guessed at, while a description saying the total including tax, in the invoice's own currency is not. And keep the schema small. A schema with forty fields and deep nesting gets a worse fill rate than three calls with a dozen fields each, and it is harder to tell which part went wrong.
Optional means optional#
Required fields force invention
Marking a field required when the source might not contain it does not make the data appear. It makes the model invent a plausible value, and validation will happily accept it because it is the right type. Mark genuinely optional fields optional, and tell the model null is acceptable.
The same argument applies to the range checks. Field(gt=0) on an amount is worth having because a negative total is impossible and catching it is free. Field(le=500) on a refund is a different kind of rule: it is a policy limit, and it belongs in the model definition precisely because that is the one place every caller passes through. Where a genuine value might exceed it, a validator that flags for review is better than one that rejects, because a rejection here loses a real refund.
from typing import Literal
class Extraction(BaseModel):
invoice_number: str = Field(description="As printed on the invoice, e.g. INV-20412")
amount: float = Field(gt=0, description="Total including tax")
currency: Literal["GBP", "USD", "EUR"] # an enum, not a free string
due_date: str | None = Field(
default=None,
description="ISO date. Null if the invoice does not state one.",
)Fail with something actionable#
try:
extraction = Extraction.model_validate_json(response.text)
except ValidationError as e:
for err in e.errors():
field = ".".join(str(p) for p in err["loc"])
log.warning("validation_failed", extra={
"field": field,
"problem": err["msg"],
"run_id": run_id,
})# What errors() actually contains, for
# {"invoice_number": "INV-1", "amount": -5, "currency": "POUNDS"}
[
{"type": "greater_than", "loc": ("amount",),
"msg": "Input should be greater than 0", "input": -5},
{"type": "string_too_long", "loc": ("currency",),
"msg": "String should have at most 3 characters", "input": "POUNDS"},
]Log the type as well as the message, because type is a stable machine-readable code and msg is prose that changes between versions. Counting failures by field and type is what turns this from an error log into a signal: if ninety percent of your failures are currency, the prompt needs to name the three currencies, and no amount of retrying will discover that for you.
Do not log the whole invalid payload by default
The thing that failed validation is untrusted input, and it may contain a customer's personal details or an entire document. Log the field names, the error types and an id; keep the payload behind a flag, or store it in a review queue with the same access controls as the source data.
Finally, decide what failure means before it happens, and make it a route rather than an exception someone else catches. Retry once with the specific error fed back, then fail cleanly: store nothing, put the case in a review queue or return an explicit error, and record it. The one option that is never acceptable is continuing with partial data.
Where people go wrong
Catching a validation error and continuing with partial data. A half-valid object flowing onward causes damage somewhere unrelated and much later, which is far more expensive to diagnose than an immediate failure.
If this comes up in an interview
Where should validation happen?
At every boundary where data enters from somewhere you do not control, and only there. Past that point everything downstream can assume the shape is correct, which keeps the checks in one place instead of scattered and drifting.
Why is model output untrusted input?
It is generated rather than guaranteed. It is usually the shape you asked for and sometimes is not, and a schema only guarantees structure, never that the values are real. Treat it like a form submission from a stranger.
What is the benefit of generating a JSON schema from your validation model?
The contract you send to the API and the check you run on the response come from one declaration, so they cannot drift apart. Change the shape once and both follow.
Build this: Take a model call that returns JSON and put a validated boundary around it. Then feed it three malformed responses and produce a distinct actionable message for each.
Watch it explained
This is one section of Typing & validation, 11 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 7 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.