Asking for JSON versus guaranteeing it

If your code has to parse what a model writes, you need the shape of that output to be certain rather than likely. This section is about the gap between requesting a format and enforcing one, and about what schema-constrained output does and does not promise you. It is the difference between a job that runs unattended and one that breaks every few days in a way nobody can reproduce.

The 95 percent problem#

You ask for JSON in the prompt. It works, most of the time. Then occasionally the model wraps it in a code fence, or opens with 'Here is the JSON:', or leaves a trailing comma. Your parser raises, and it happens often enough to matter and rarely enough to be hard to reproduce.

What you asked for:
{"name": "Jack", "role": "engineer"}

What sometimes arrives:
Here is the JSON you requested:
```json
{"name": "Jack", "role": "engineer"}
```

An instruction in a prompt is a request. The model usually honours it, and sampling means usually is not always.

Constrained decoding#

Most providers can take a schema and restrict which tokens are allowed to be generated at each step. If the schema says the next thing must be a closing brace, no other token can be sampled. The output cannot be malformed, because malformed output was never reachable.

schema = {
    "type": "OBJECT",
    "properties": {
        "name":   {"type": "STRING"},
        "amount": {"type": "NUMBER"},
        "overdue": {"type": "BOOLEAN"},
    },
    "required": ["name", "amount", "overdue"],
}

response = client.generate(
    prompt=text,
    response_mime_type="application/json",
    response_schema=schema,
)

data = json.loads(response.text)   # this cannot raise a format error

Types are enforced too. An amount declared as a number comes back as 1240.5, not the string '1,240.50'. That alone removes a layer of cleanup code.

What it does not fix#

Two things follow. Make fields optional when they genuinely might be absent, and say in the prompt that null is the right answer for a missing value. Then verify the values themselves, which is what evaluation is for.

# Forces invention when the field is missing from the source
"required": ["name", "amount", "date"]

# Lets the model tell the truth
"required": ["name"]
# plus, in the prompt:
# "Use null for any field not present in the text. Do not guess."

The failure that remains#

A schema does not stop you running out of tokens. If generation hits the limit halfway through, you get half an object, which is not valid JSON. Check the stop reason before parsing.

if response.stop_reason == "max_tokens":
    raise ValueError("truncated before the JSON was complete")

data = json.loads(response.text)