Asked about Extraction you can measure
How would you run an extraction over 250,000 documents?
An answer that works
Size the worker pool from the rate limit and the call latency rather than by guessing: requests in flight equals rate times latency, so a 4,000 per minute limit with 2.5 second calls justifies about 167 workers. Append each finished row to a file and start every run by reading back what is already there, so a crash costs only the unfinished part. Price it first on the eval set, and use the batch interface if hours are acceptable.
Then they push further
The job is being throttled. What do you change?
Honour the retry-after header, back off exponentially with jitter, and shrink the pool. Retrying harder under throttling turns a slow job into a stalled one. Check which limit is binding, since the token limit usually binds before the request limit on long documents.
How do you watch quality across all 250,000 when you can only label a few hundred?
Run two proxies that need no labels. Agreement: extract twice under settings that should not change the answer and count the fields that differ, since a field the pipeline cannot agree with itself about is one it does not know. Grounding: check the quoted source text really appears in the document. Both are floors rather than estimates, so keep the labelled set as the number you would quote.
What gets marked up
Strong answers derive the concurrency from the limit rather than picking a number, make the job restartable, and separate the unlabelled proxies from real accuracy instead of presenting them as the same thing.
Reading an answer is not being able to give one. The section that teaches this is Extraction you can measure, 4 hours of Structured output.
Learn this properlyAlso asked about Structured output