Reading a model card and technical report

Every model release comes with a report, and they are marketing documents as much as technical ones. This section is about reading one for what it actually establishes: which benchmarks were run under which conditions against which competitors, what a context length claim does and does not promise, and what the licence lets you do with the model and its outputs. What has been left out matters as much as what is there, so it lists the omissions worth noticing.

A checklist#

• Size, architecture, context length and tokenizer.
• Training data: sources, size in tokens, cut-off date, languages.
• Evaluation: which benchmarks, with what prompts and settings, compared against which models under the same conditions.
• Any statement about contamination checks.
• Known limitations and safety evaluations.
• Licence: can you use it commercially, fine-tune it, use its outputs to train other models?

Claims to check#

Comparison tables are often assembled from different sources with different settings. A 'context length' is the maximum the model accepts, not the length at which it uses information reliably; look for long-context retrieval evaluations. And a number without the prompt format and sampling settings cannot be reproduced.

Is the gap real?#

The single most useful habit when reading a results table is to work out the noise floor before reading the numbers. A benchmark score is a proportion measured on a finite sample, so it carries sampling error like any other proportion: about 1.96 times the square root of p times one minus p over n for a ninety-five per cent interval, and about 1.4 times that again for the gap between two models. Most published gaps are smaller than the interval around them.

import math

def half_width(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)          # 95% interval on one score

def gap_needed(p, n):
    return 1.96 * math.sqrt(2 * p * (1 - p) / n)      # smallest gap that is not noise

print("items    one score is +/-   a real gap must exceed")
for n in [100, 250, 500, 1000, 5000, 20000]:
    print(f"{n:5}    {half_width(0.70, n)*100:12.1f} pts   {gap_needed(0.70, n)*100:16.1f} pts")

print()
for n, a, b in [(500, 0.712, 0.704), (500, 0.760, 0.700), (14000, 0.712, 0.704)]:
    gap = (a - b) * 100
    need = gap_needed(0.70, n) * 100
    print(f"{a:.1%} vs {b:.1%} on {n:6} items: gap {gap:.1f} pts, noise floor {need:.1f} pts"
          f"  -> {'real' if gap > need else 'indistinguishable'}")

print()
for gap in [0.008, 0.02, 0.05]:
    n = 1.96 ** 2 * 2 * 0.70 * 0.30 / gap ** 2
    print(f"to call a {gap*100:.1f} point gap real you need about {n:,.0f} items")

# items    one score is +/-   a real gap must exceed
#   100             9.0 pts               12.7 pts
#   250             5.7 pts                8.0 pts
#   500             4.0 pts                5.7 pts
#  1000             2.8 pts                4.0 pts
#  5000             1.3 pts                1.8 pts
# 20000             0.6 pts                0.9 pts
#
# 71.2% vs 70.4% on    500 items: gap 0.8 pts, noise floor 5.7 pts  -> indistinguishable
# 76.0% vs 70.0% on    500 items: gap 6.0 pts, noise floor 5.7 pts  -> real
# 71.2% vs 70.4% on  14000 items: gap 0.8 pts, noise floor 1.1 pts  -> indistinguishable
#
# to call a 0.8 point gap real you need about 25,210 items
# to call a 2.0 point gap real you need about 4,034 items
# to call a 5.0 point gap real you need about 645 items

Two honest qualifications, because someone will raise them. Scoring both models on the same items is a paired comparison and has lower variance than the formula above, so the true threshold is somewhat friendlier than 5.7 points on five hundred items — but you need the per-item results to compute it, and reports almost never publish them. And benchmark items are not independent draws from anything: they were written by people, in groups, about a fixed set of topics. The arithmetic is a floor on the uncertainty, not a complete account of it.

What a context length claim costs#

A long context number is a capability claim and a budget claim at once, and the budget half is arithmetic you can do while reading the announcement. Multiply the tokens you would actually put in the window by the input price.

PRICE_IN = 3.00 / 1e6          # dollars per input token, a typical mid-range rate

print("context used   cost of one call   cost of 1,000 calls a day, per month")
for tokens in [4_000, 32_000, 128_000, 200_000, 1_000_000]:
    c = tokens * PRICE_IN
    print(f"{tokens:12,}   " "$" f"{c:15.2f}   " "$" f"{c*1000*30:26,.0f}")

# context used   cost of one call   cost of 1,000 calls a day, per month
#        4,000   $           0.01   $                       360
#       32,000   $           0.10   $                     2,880
#      128,000   $           0.38   $                    11,520
#      200,000   $           0.60   $                    18,000
#    1,000,000   $           3.00   $                    90,000

Licence#

The licence section decides whether the model is usable at all, and it is the part of a release that most readers skip. Four clauses are worth finding every time. Whether commercial use is permitted at all, and whether it is capped by the licensee's user count or revenue. Whether you may fine-tune and redistribute the result. Whether outputs may be used to train other models, which several licences forbid outright and which quietly rules out distillation. And whether there is an acceptable-use policy attached that binds you downstream, because those are usually a separate document from the licence itself and are updated without a new model release.

What is usually missing#

Detailed data composition, exact compute, failed experiments and many negative results are often left out for competitive reasons. Note the gaps, and treat claims that depend on them with appropriate caution.

One omission is worth naming separately because it changes how you read everything else: per-item results. Without them you cannot compute a paired comparison, you cannot see which subset of the benchmark the gain came from, and you cannot tell a model that is broadly two points better from one that is thirty points better on one category and unchanged elsewhere. Those are completely different products, and the headline number is identical.

What to keep and what to skip#

• Keep: compute the noise floor before reading the table. On a few hundred items it is several points wide.
• Keep: a context length is what the model accepts, not what it uses well, and filling it is a monthly bill.
• Keep: the four licence clauses — commercial use, redistribution of fine-tunes, training on outputs, and the acceptable-use policy.
• Skip: the architecture diagram. It is nearly always the same architecture, and the differences rarely affect how you use it.
• Skip: reproducing their benchmark numbers. Run your own tasks instead; that is the only comparison that decides anything for you.