Typing & validation

There are a few points in any system where data crosses in from somewhere you do not control, and a model's reply is now one of them. This section is about checking it in exactly one place per edge and handing typed objects to everything inward of that, so nothing further in has to wonder whether a field is there. Scatter those checks and they drift apart, which is how one codebase ends up with three incompatible ideas of what a missing value means.

One place, at the edge#

Every system has boundaries where data arrives from somewhere you do not control: an HTTP request, a file, a database, a model response. Validate at each boundary and pass typed objects inward. Validating deeper means checking the same thing in several places, and those checks drift apart.

HTTP request  ->  validate  ->  typed object  ->  business logic
Model output  ->  validate  ->  typed object  ->  business logic

Past the arrow, nothing needs to ask whether a field exists.

Model output is untrusted input#

Treat a model response exactly as you would a form submission from a stranger. It is usually well formed and occasionally is not, and the failure should be caught where it arrives rather than three functions later.

from pydantic import BaseModel, Field, ValidationError

class Extraction(BaseModel):
    invoice_number: str
    amount: float = Field(gt=0)
    currency: str = Field(min_length=3, max_length=3)
    due_date: str | None = None        # genuinely optional

def extract(text: str) -> Extraction:
    raw = call_model(text)
    try:
        return Extraction.model_validate_json(raw)
    except ValidationError as e:
        raise ModelOutputError(f"could not parse extraction: {e.errors()}") from e

There are four distinct ways that call can go wrong, and they want different handling. Knowing which one you have is the difference between a retry that works and a retry loop that burns money reaching the same answer.

What went wrong                          What it looks like              What to do
not JSON at all                          json_invalid, "Expecting value"  strip prose, then retry
JSON, wrong shape                        missing / extra field            retry with the error
right shape, impossible value            amount: -5, currency 'POUNDS'    retry with the error
right shape, plausible, wrong            invoice_number: 'INV-0001'       validation cannot help

The first row is common enough to handle rather than retry. A model asked for JSON frequently returns JSON wrapped in a sentence of explanation, or inside a fenced code block, and both parse as invalid. Strip that before validating, and reserve the retry for cases where there is genuinely no JSON in the reply.

import json, re

FENCE = chr(96) * 3          # the triple backtick a model wraps JSON in

def extract_json(text: str) -> str:
    """Return the first complete JSON value in a model reply, as a string."""
    t = text.strip()
    fenced = re.search(FENCE + r"(?:json)?\s*(.*?)" + FENCE, t, re.S)
    if fenced:
        t = fenced.group(1).strip()
    start = min((i for i in (t.find("{"), t.find("[")) if i != -1), default=-1)
    if start == -1:
        raise ValueError("no JSON object or array in model output")
    obj, _end = json.JSONDecoder().raw_decode(t[start:])   # ignores trailing prose
    return json.dumps(obj)

extract_json('Here is the JSON:\n{"amount": 12.5}\nHope that helps.')   # {"amount": 12.5}
extract_json(FENCE + 'json\n{"amount": 12.5}\n' + FENCE)                 # {"amount": 12.5}
extract_json("I am not able to help with that.")
# ValueError: no JSON object or array in model output

One schema, two jobs#

The same model definition can generate the JSON schema you send to the API and validate the response that comes back. That is worth more than it sounds: the contract and the check cannot disagree, because they are the same declaration.

schema = Extraction.model_json_schema()

response = client.generate(prompt, response_schema=schema)
result = Extraction.model_validate_json(response.text)

{"properties": {"invoice_number": {"title": "Invoice Number", "type": "string"},
                "amount":  {"exclusiveMinimum": 0, "type": "number"},
                "currency": {"maxLength": 3, "minLength": 3, "type": "string"},
                "due_date": {"anyOf": [{"type": "string"}, {"type": "null"}],
                             "default": null}},
 "required": ["invoice_number", "amount", "currency"],
 "title": "Extraction", "type": "object"}

Two things follow. Give every field a description, because that text is carried into the schema and is the only instruction the model gets about what the field means; amount: float with no description is guessed at, while a description saying the total including tax, in the invoice's own currency is not. And keep the schema small. A schema with forty fields and deep nesting gets a worse fill rate than three calls with a dozen fields each, and it is harder to tell which part went wrong.

Optional means optional#

The same argument applies to the range checks. Field(gt=0) on an amount is worth having because a negative total is impossible and catching it is free. Field(le=500) on a refund is a different kind of rule: it is a policy limit, and it belongs in the model definition precisely because that is the one place every caller passes through. Where a genuine value might exceed it, a validator that flags for review is better than one that rejects, because a rejection here loses a real refund.

from typing import Literal

class Extraction(BaseModel):
    invoice_number: str = Field(description="As printed on the invoice, e.g. INV-20412")
    amount: float = Field(gt=0, description="Total including tax")
    currency: Literal["GBP", "USD", "EUR"]      # an enum, not a free string
    due_date: str | None = Field(
        default=None,
        description="ISO date. Null if the invoice does not state one.",
    )

Fail with something actionable#

try:
    extraction = Extraction.model_validate_json(response.text)
except ValidationError as e:
    for err in e.errors():
        field = ".".join(str(p) for p in err["loc"])
        log.warning("validation_failed", extra={
            "field": field,
            "problem": err["msg"],
            "run_id": run_id,
        })

# What errors() actually contains, for
# {"invoice_number": "INV-1", "amount": -5, "currency": "POUNDS"}
[
  {"type": "greater_than", "loc": ("amount",),
   "msg": "Input should be greater than 0", "input": -5},
  {"type": "string_too_long", "loc": ("currency",),
   "msg": "String should have at most 3 characters", "input": "POUNDS"},
]

Log the type as well as the message, because type is a stable machine-readable code and msg is prose that changes between versions. Counting failures by field and type is what turns this from an error log into a signal: if ninety percent of your failures are currency, the prompt needs to name the three currencies, and no amount of retrying will discover that for you.

Finally, decide what failure means before it happens, and make it a route rather than an exception someone else catches. Retry once with the specific error fed back, then fail cleanly: store nothing, put the case in a review queue or return an explicit error, and record it. The one option that is never acceptable is continuing with partial data.

Model output arrives as text, and text can be anything: the wrong shape, the wrong type, a refund for fifty times what anyone authorised. This section is about the layer that stops all of that at the edge of your system, by declaring what the data must look like, adding the rules a type cannot express, deciding what happens when validation fails, and letting a type checker read the rest of the code for free. The alternative is a dictionary passed around the codebase that every function downstream has to guess about.

The smallest model#

from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

User(name="Jack", age="31")    # works: "31" is converted to 31
User(name="Jack", age="old")   # raises ValidationError, and says which field

Pydantic checks and converts data when you create the object. After that line, the rest of your code can trust that age is an int.

1 validation error for User
age
  Input should be a valid integer, unable to parse string as an integer
  [type=int_parsing, input_value='old', input_type=str]

Three details about what it does with the data. It coerces where the conversion is unambiguous, so the string '31' becomes 31 and '1.5' becomes a float, but 1.7 into an int field is rejected rather than truncated, because discarding information silently is worse than failing. It ignores unknown fields by default, which is usually right for an API you do not control and wrong for model output, where an unexpected key means the model answered a different question; set extra='forbid' there. And it validates on construction only, so mutating an attribute afterwards skips every check unless you ask for validate_assignment.

class Strict(BaseModel):
    model_config = {"extra": "forbid", "validate_assignment": True}
    a: int

Strict.model_validate({"a": 1, "b": 2})
# ValidationError: b  Extra inputs are not permitted [type=extra_forbidden]

class Loose(BaseModel):
    a: int

Loose.model_validate({"a": 1, "b": 2})     # a=1   the b is silently dropped

class C(BaseModel):
    n: int
C(n="7")      # n=7,  coerced
C(n=1.7)      # ValidationError: Input should be a valid integer,
              #                  got a number with a fractional part

The three entry points do the same job on different input. model_validate takes a dict, model_validate_json takes a JSON string and is the one to use on a model response because it parses and validates in one step with a clearer error when the text is not JSON at all, and constructing the class directly validates keyword arguments. Going the other way, model_dump gives a dict and model_dump_json a string; model_construct skips validation entirely and exists for data you have already checked, not as a way round an error.

Rules that types cannot express#

from pydantic import BaseModel, Field, field_validator

class Refund(BaseModel):
    order_id: str = Field(pattern=r"^A-\d{3,}$")
    amount: float = Field(gt=0, le=500)
    reason: str

    @field_validator("reason")
    @classmethod
    def not_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("reason must say why")
        return v.strip()

Limits like 'at most 500' belong here, next to the data, not scattered through the code that uses it. A model asking for a refund of 5,000 is stopped at the boundary.

Two things about validators are worth knowing precisely. A validator returns the value, which means it can normalise as well as check: the one above strips whitespace, and that stripped value is what the object carries, so every caller downstream sees the tidy version. And it signals failure by raising ValueError, not by returning False and not by raising ValidationError, which pydantic constructs for you by collecting everything that failed.

Refund(order_id="A-102", amount=10, reason="  damaged  ")
# order_id='A-102' amount=10.0 reason='damaged'      note the strip

Refund(order_id="X-1", amount=5000, reason=" ")
# 3 validation errors:
#   order_id  String should match pattern '^A-\d{3,}$'
#   amount    Input should be less than or equal to 500
#   reason    Value error, reason must say why

For a rule that spans several fields, a field validator cannot see the others, so use a model validator with mode='after', which runs once the whole object exists and is fully typed. mode='before' runs on the raw input instead and is the place to reshape something the model tends to get wrong, such as accepting a bare string where an object was asked for.

from pydantic import model_validator

class DateRange(BaseModel):
    start: str
    end: str

    @model_validator(mode="after")
    def ordered(self):
        if self.start > self.end:
            raise ValueError("start must not be after end")
        return self       # an 'after' validator returns self, not a value

DateRange(start="2026-05-01", end="2026-01-01")
# ValidationError: Value error, start must not be after end

One of several shapes#

Agents often return one of several actions. A tag field tells Pydantic which shape to expect, which gives far clearer errors than trying each shape in turn.

from typing import Annotated, Literal, Union
from pydantic import BaseModel, Field, TypeAdapter

class Reply(BaseModel):
    type: Literal["reply"]
    text: str

class Escalate(BaseModel):
    type: Literal["escalate"]
    team: Literal["billing", "tech"]

Action = Annotated[Union[Reply, Escalate], Field(discriminator="type")]
action = TypeAdapter(Action).validate_json(model_output)

TypeAdapter is how you validate anything that is not itself a BaseModel: a union like this one, a list[Extraction], a plain dict[str, float]. Build it once and reuse it, because construction compiles the validator and is the expensive part.

ta = TypeAdapter(Action)

ta.validate_json('{"type": "escalate", "team": "billing"}')
# Escalate(type='escalate', team='billing')

ta.validate_json('{"type": "escalate", "team": "legal"}')
# ValidationError: escalate.team
#   Input should be 'billing' or 'tech'

ta.validate_json('{"type": "unknown"}')
# ValidationError: Input tag 'unknown' found using 'type' does not match any of
#                  the expected tags: 'reply', 'escalate'

The same pattern is what a tool call is: a name that selects a shape, and arguments validated against that shape. Define one model per tool, union them on the name, and the dispatch becomes a match on the validated object rather than a chain of dictionary lookups that each assume a key exists.

Validate, then retry with the error#

from pydantic import ValidationError

def get_refund(messages, attempts=2):
    for _ in range(attempts):
        raw = call_model(messages)
        try:
            return Refund.model_validate_json(raw)
        except ValidationError as e:
            messages = messages + [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"That failed validation: {e}. Fix it."},
            ]
    raise ExtractionFailed(raw)

The error message is specific, so the model usually fixes it in one go. Cap the attempts and count how often the retry happens.

Four things decide whether this helps or just costs twice as much. Feed back the error, not a generic instruction to try again; the errors() output names the field and the constraint, and that is the whole reason the second attempt succeeds. Cap attempts at two, because a third almost never converges and you are now paying three times for a document that needs a human. Do not retry a refusal: if the model declined, the input is the problem and the same prompt will decline again. And record the retry rate per field, because a field that needs a retry on a fifth of documents is a prompt or schema problem that a retry loop is quietly paying to work around for ever.

outcome              share   what it costs        what it means
valid first try      0.91     1 call               working
valid after retry    0.07     2 calls + latency    watch which field
failed after 2       0.02     2 calls, no result   review queue
refused              0.00     1 call               do not retry these

Settings#

The pydantic-settings package reads configuration from environment variables into a typed object and fails at startup if something is missing. A missing API key should stop the service from starting, not surface as an error on the first request an hour later.

from pydantic import Field
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    anthropic_api_key: str                       # no default: required
    model: str = "claude-sonnet-5"
    max_concurrency: int = Field(default=8, ge=1, le=64)
    request_timeout_s: float = 30.0

    model_config = {"env_file": ".env"}

settings = Settings()      # raises here, at import, if the key is not set

Let a type checker read your code#

Pydantic checks data at runtime. A static type checker, such as mypy or pyright, checks your code without running it: passing a str where an int is expected, forgetting that a value can be None. Run one in CI. It catches a class of bugs that tests often miss, especially the None case.

The two are complementary and neither substitutes for the other. A type hint is a claim about your own code that costs nothing at runtime and is not enforced there; pydantic is an actual check on data you did not write. So hints go everywhere inside the system and pydantic goes at the edges, which is the same division the previous section drew between trusted and untrusted.

def cheapest(models: list[str]) -> str | None:
    ...

best = cheapest(available)
print(best.upper())
# mypy: error: Item "None" of "str | None" has no attribute "upper"

# Three annotations worth writing by hand:
from typing import Literal, TypedDict

Role = Literal["user", "assistant"]          # not str
class Message(TypedDict):                     # a dict with known keys
    role: Role
    content: str

def run(messages: list[Message]) -> str: ...