AI and ML interview questions

A list can be changed after creation, a tuple cannot. Because a tuple cannot change, it can be used as a dictionary key and a list cannot. Use a tuple when the group of values is…

Strings are immutable, so each + builds a whole new string and copies everything so far. Over many iterations that repeated copying makes the loop much slower. join does it in one…

== compares values. is compares whether two names point to the same object in memory. Use is only for None, True and False. Using it on numbers or strings seems to work for small…

It reverses the string. The three parts of a slice are start, stop and step, so a step of -1 walks backwards through the whole thing.

list(set(items)) is the quickest way, but it loses the original order. If order matters, use list(dict.fromkeys(items)), because dictionaries keep insertion order in modern Python.

A list has to compare against each item in turn, so the time grows with its length. A set uses a hash table, so it jumps straight to where the item would be. That takes the same…

It raises a TypeError. Keys must be hashable, which means they cannot change. Lists can change, so they are not hashable. Use a tuple instead.

sorted returns a new sorted list and leaves the original alone. .sort() sorts the list in place and returns None. Assigning the result of .sort() to a variable is a common bug,…

The default list is created once when the function is defined, not on each call, so every call shares the same list. Values accumulate across calls. Use None as the default and…

break exits the loop completely. continue skips the rest of the current iteration and moves to the next one.

None. This is why assigning the result of a method like list.sort() gives you None, because it changes the list in place and returns nothing.

Parameters are the names in the function definition. Arguments are the actual values you pass when calling it. Interviewers ask this to check you can be precise about terminology.

It catches everything, including KeyboardInterrupt and SystemExit, so the program cannot be stopped normally. It also hides real bugs like typos, turning a crash you would have…

load reads from a file object. loads reads from a string. The same distinction applies to dump and dumps.

It closes the file automatically when the block ends, even if an exception is raised inside it. Without it, a failure partway through can leave the file open.

When callers need to handle your failure differently from built-in ones. A ModelOutputError can be caught and retried, whereas catching ValueError would also catch unrelated…

No. Python ignores them while running. They exist for editors and type checkers, which is why data from an API still needs real validation with something like pydantic.

The function returns either a dictionary or None. A type checker will then warn if you use the result without checking for None first. Older syntax writes it as Optional[dict].

It gives you validation, type conversion, clear error messages listing every problem, and a JSON schema, all from one class definition. Hand-written checks drift apart from the…

At the boundary, as soon as untrusted data arrives. After that point everything downstream can assume the data is the right shape, which keeps the checks in one place.

Concurrency means several tasks are in progress and taking turns. Parallelism means they genuinely run at the same instant on different cores. Python's asyncio gives you…

When the work is CPU-bound. Async overlaps waiting, not computing, and there is still one thread. For heavy computation you need multiprocessing.

It stalls the entire event loop, so every other coroutine stops until it returns. The symptom is that concurrent code runs at sequential speed. The fix is an async library or…

Use asyncio.gather with a Semaphore capping how many are in flight, a timeout on each request, return_exceptions=True so one failure does not lose the rest, and retry with…

A list computes and stores every item immediately. A generator produces items one at a time on demand and stores only its current position, so memory stays flat no matter how many…

Loop over the file object rather than calling read or readlines, so it yields one line at a time. Wrap it in a generator if you need to transform the records, and batch it if the…

You get nothing, and no error. It has been consumed. If you need the values again you must recreate the generator or materialise it into a list.

No. A generator does not know how many items it will produce and cannot look ahead. If you need a count, either convert it to a list or count as you iterate.

A unit test checks one piece in isolation with fast, predictable inputs. An integration test checks that several pieces work together, usually including something real such as a…

Tests full of mocks assert that your code called your code, which stays true even after you break the behaviour. They also have to be rewritten on every refactor. Fake at the…

No. Coverage tells you which lines ran, not whether anything was verified. A suite at 90% that asserts almost nothing is worse than an honest 50%, because it creates confidence…

Separate the deterministic parts and test those normally with a fake client returning fixed responses. The quality of the model's output is not a unit test; it needs an eval suite…

It isolates a project's dependencies. Without one, two projects needing different versions of the same library conflict, and installing globally can break system tools that rely…

requirements.txt is a flat list of packages to install. pyproject.toml declares the project itself, including metadata, dependencies and tool configuration, and is the modern…

Keep them in environment variables, loaded from a .env file that is gitignored, and read at runtime. Never commit them. In production they come from the platform's secret store…

Bottom up. The last line names the exception. The frames above show the call path. Find the last frame in your own code, because that is usually where the mistake is.

A file containing one commit hash, plus an entry in HEAD saying which branch you are on. The commits are not stored in the branch, so creating and deleting branches is instant and…

All three move the current branch to the commit you name. Soft leaves your files and the staging area alone, so the changes from the undone commits are staged and ready to…

Merge combines two histories and adds a merge commit, preserving what actually happened. Rebase replays your commits on top of the target, producing a linear history but rewriting…

git revert, which adds a new commit undoing the change. Reset rewrites history and breaks everyone who already pulled it.

git bisect. Mark a known good commit and a known bad one, and it binary searches the range, so a thousand commits takes about ten checks.

Git stores a full copy of every version of a binary, so the repository grows enormously and clones become impractical. Store weights in object storage with a versioned path and…

Revoke the key and issue a new one first, because it may already have been copied by automated scanners within minutes. Then remove it from the code and, if needed, from history,…

Reset moves the branch pointer back, rewriting history, which is fine for local commits. Revert makes a new commit that undoes an earlier one, keeping history intact, which is…

401 means the request was not authenticated, so the credentials are missing or invalid. 403 means authentication succeeded but the caller is not permitted to do this. One is a…

429 after a backoff, and 5xx and network timeouts, which are often transient. Not 4xx generally, because an identical request will fail identically, so retrying wastes time and…

An idempotent request has the same effect whether it runs once or several times. It matters because a timeout does not tell you whether the server acted, so a retry can duplicate…

Without it, all clients that failed at the same moment retry at the same moment, producing a synchronised spike that can knock over the recovering service again. Randomness…

Cap concurrency with a semaphore rather than firing everything at once, read the rate limit headers and slow down before being rejected, back off exponentially with jitter and…

Nothing. The request may have been processed while the connection dropped. That is why any retry of a state-changing call needs an idempotency key, or you risk doing it twice.

Bounded concurrency with a semaphore sized to the rate limit, a timeout on each call, exceptions returned as values so one failure does not lose the rest, results written as they…

From the provider's rate limit, not from your machine. Start conservatively, watch for 429 responses, and raise until you approach the limit. Too high is slower than too low once…

Threads for blocking libraries with no async equivalent, usually via asyncio.to_thread. Processes when the work is CPU-bound, since async and threads both leave you with one…

The calls spend almost all their time waiting for the network. The event loop starts a call, and while it waits, starts others. The waits overlap, so total time is close to the…

A lock in CPython that allows one thread to execute Python code at a time. It matters for CPU-bound Python code, where threads give no speedup and processes are needed. It matters…

At every boundary where data enters from somewhere you do not control, and only there. Past that point everything downstream can assume the shape is correct, which keeps the…

It is generated rather than guaranteed. It is usually the shape you asked for and sometimes is not, and a schema only guarantees structure, never that the values are real. Treat…

The contract you send to the API and the check you run on the response come from one declaration, so they cannot drift apart. Change the shape once and both follow.

At the boundaries: when data enters from users, from other services, and from the model. Validate once at the edge into typed objects, and let the inside of the system trust them.

A union of shapes with a tag field that says which shape an object is. It suits outputs that can be one of several actions, and gives precise errors because the validator knows…

WHERE filters rows before grouping. HAVING filters groups after aggregation. You cannot reference an aggregate in WHERE because it has not been computed yet.

When you need every row from the left side regardless of whether there is a match, and absence is meaningful. Watch the counting trap: COUNT(*) counts the unmatched row,…

EXPLAIN ANALYZE and look at whether it is scanning the table or using an index, and how many rows it reads compared to how many it returns. A sequential scan on a large table…

Computing across a set of related rows while keeping every row, rather than collapsing them as GROUP BY does. Ranking within a group, comparing to the previous row, and taking the…

Rank orders within each user by created_at descending with ROW_NUMBER in a CTE, then keep rows where the rank is 1. Mention that if two orders share a timestamp, you need a…

If the subquery contains a NULL, the comparison with that NULL is unknown for every row, so no row satisfies NOT IN. NOT EXISTS does not have this problem.

COUNT(*) counts rows. COUNT(column) counts rows where that column is not NULL.

The client sends a unique request id. The server stores it with a unique constraint as part of the same transaction as the work, and on conflict returns the original result…

Two transactions read the same value, each modifies it, and one write overwrites the other. Prevent it by doing the update atomically in one statement, locking the row before…

It depends on the company rather than the role title. Large companies and infrastructure-flavoured teams generally still do. Product-focused AI companies where agents and…

From the target list. Check what the specific companies run, through candidate reports or by asking the recruiter directly. Forty hours is either essential or wasted depending…

Whenever the question involves lookup, counting, grouping or duplicate detection. It turns a scan into a constant-time check, which is usually what converts a quadratic solution…

It asks about a contiguous subarray or substring with some property, usually the longest or shortest satisfying a condition. The shape is always expand with the loop, contract…

When the answer is a value in a range and checking whether a candidate works is cheap, you binary search the range of possible answers rather than the input. The input does not…

Next-greater and next-smaller element questions in linear time. You keep the stack ordered, and when a new element breaks the order you have found the answer for everything it…

When the question involves levels, or the shortest path in an unweighted graph, since BFS reaches nodes in order of distance. DFS suits paths, subtree properties and anything…

Keep a heap of size k rather than sorting everything. That is n log k rather than n log n, and it works on a stream where you cannot hold all the data at once.

Ordering with dependencies: course prerequisites, build order, task scheduling. It also detects cycles, since a cycle means some nodes never reach indegree zero and the output is…

Backtracking is recursion that undoes its choice before trying the next one. The undo step, usually a pop, is what makes it backtracking, and forgetting it is the most common bug…

Restate it, clarify input size and edge cases, give the brute force with its complexity, identify the wasteful part, propose the improvement and its complexity, then write it…

Say where I am and what I have eliminated, and propose the next thing to try. A working brute force with clear reasoning scores better than silence, and taking a hint well is a…

Less than candidates assume. A clear approach with correct complexity analysis, communicated well, usually scores better than a silently produced solution the interviewer could…

How aligned two vectors are. It is the sum of elementwise products, and dividing out the magnitudes gives cosine similarity, which compares direction only. It underlies embedding…

Each output entry is a row of the first dotted with a column of the second, and a dot product needs equal-length vectors. So the inner dimensions must match and the result takes…

NumPy stretching a smaller array to match a larger one when shapes are compatible. It is convenient and it can silently produce a valid array of the wrong shape rather than an…

L2 penalises squared weights and shrinks all of them slightly. L1 penalises absolute values and drives many weights to exactly zero, so it performs feature selection. The…

A decomposition of any matrix into rotations and a scaling. Truncating to the largest singular values gives the best low-rank approximation, which underlies compression, PCA and…

PCA finds the eigenvectors of the data's covariance matrix. They are the directions of greatest variance, and the eigenvalues say how much variance each direction captures.

The sum of products of matching entries. Normalised by both lengths it is the cosine of the angle between them, a measure of how aligned they are. It underlies similarity search,…

Stack the inputs into a matrix X with a column of ones, and the targets into a vector y. The best weights minimise the squared length of Xw minus y, the least squares problem,…

Centre the data, then find the orthogonal directions along which it varies most, in order. Projecting onto the first few keeps as much variance as possible in fewer dimensions. In…

It represents the update to a weight matrix as the product of two thin matrices of rank r. For a d by d matrix that is 2 times d times r numbers instead of d squared, a small…

Not on that evidence. The margin of error on 50 samples is around ten points, so the difference is inside the noise. Two identical systems scored on 50 cases each differ by four…

About 9 per cent. Take 100,000 people: 100 have the condition and 99 of them test positive, while 99,900 do not and 1 per cent of those, 999 people, test positive anyway. That is…

Because the base rate dominates. A model that always predicts the majority class scores very well while being useless, and even an accurate classifier for a rare condition…

Latency is long-tailed, so the mean is pulled around by a few slow requests and hides them at the same time. The high percentile is what users actually experience when things go…

Writing the probability of the data you actually observed as a function of the parameters, then choosing the parameters that make it largest. The data is fixed and the parameter…

It is the negative log-likelihood of the correct labels under the model's predicted probabilities. Minimising it is maximum likelihood estimation, and it punishes confident wrong…

Bootstrap: resample the per-example results with replacement many times, recompute the score each time, and take the 2.5th and 97.5th percentiles. It needs no distributional…

Its predicted probabilities match observed frequencies: predictions made at 80 percent are right about 80 percent of the time. Most deep networks are overconfident, which matters…

The exponential of the average cross-entropy per token. It reads as an average branching factor: a perplexity of 8 means the model is as uncertain as if it were choosing uniformly…

The average extra surprise from using distribution q when the truth is p: cross-entropy minus entropy. It is zero when the two match and grows as they diverge, and it is measured…

As the drift penalty in preference training, keeping the tuned model near the one it started from. As the distillation loss against a teacher's full output distribution. And in…

Entropy is a property of the data and cannot be changed by the model. Cross-entropy is what the model's own distribution costs, so it is the only part that training can move, and…

About 17%. Out of 10,000 people, 100 are sick and 99 test positive; of 9,900 healthy people, 495 test positive. So 99 of 594 positives are real.

The average of many independent draws is approximately normal whatever the shape of the individual draws, centred on their mean, with a standard deviation equal to theirs divided…

As many as the Monte Carlo margin allows, and no more. The estimate is a proportion over n trials, so the margin is 1.96 times the square root of p times one minus p over n. At…

Latency is heavy-tailed. The mean is pulled up by a few very slow requests while describing few real ones, and it hides the tail entirely. High percentiles describe the experience…

By what generated it rather than by looking at a histogram. One trial with two outcomes is Bernoulli, and a count of successes out of a fixed n is binomial. A count of events in a…

If there were no real effect, data at least this extreme would occur about 3% of the time. It does not mean there is a 97% chance the effect is real, and it says nothing about the…

It depends on four things and nothing else: the baseline rate, the smallest lift worth shipping, the false positive rate you accept and the power you want. For a 3.0 per cent…

Treat it as nothing. With twenty independent metrics at a 5 per cent threshold, the chance at least one fires when nothing has changed is one minus 0.95 to the twentieth, about 64…

The gradient gives a direction; the learning rate is how far you step along it. There is a threshold set by the curvature of the loss: on a loss of 5 w squared the update…

The vector of partial derivatives of the loss with respect to every parameter. It points in the direction of steepest increase, so training steps against it to reduce the loss.

The chain rule multiplies derivatives along the path, so many factors below one shrink towards zero and early layers receive almost no signal. Residual connections and activations…

Exploding gradients first: clip them and lower the learning rate. Then check for division by zero or a log of zero in the loss, and for bad input data such as NaNs already present…

Mostly not. In high dimensions, most flat points are saddle points, where some directions still go downhill. Saddle points and plateaus slow training far more often than true…

It accumulates a running average of past gradients, so consistent directions gain speed and oscillations cancel. That carries the optimiser across saddle points and plateaus and…

The predicted probability vector minus the one-hot target vector. It is simple and well-behaved, which is one reason softmax and cross-entropy are always used together.

It keeps a running average of gradients, which acts as momentum, and a running average of squared gradients, which estimates each parameter's gradient scale. Both are…

The loop executes in the Python interpreter with per-element overhead, while the vectorised operation runs in compiled code over contiguous memory, often using multiple values per…

You may be assigning to a copy rather than the original DataFrame, so the change silently does nothing. It usually follows a filter, and using .loc for conditional assignment…

First find out whether missingness is informative, and if so record it as a feature before imputing. Then drop rows missing something essential, and impute the rest with a median…

A view shares memory with the original array, so modifying it changes the original; slicing usually returns a view. A copy is independent. Confusing them causes silent…

Choose smaller numeric types where the range allows, convert repeated strings to categorical, and read only the columns you need from a columnar format.

The right-hand table probably had duplicate keys, so each left row matched several right rows and was repeated. Check key uniqueness before joining, use validate to enforce the…

Read only needed columns with compact dtypes, process in chunks, convert to Parquet, or use a tool that streams and parallelises such as Polars in lazy mode or DuckDB, which can…

Idempotency: write to a partition keyed by the period being processed and replace it rather than appending, so a rerun produces the same result. Parameterise by date range so…

Validate incoming data against an expected schema and fail loudly rather than writing whatever arrived. Adding nullable columns is safe; removals and type changes need a versioned…

Row count against expectation, uniqueness of keys, null checks on required fields, range checks on numerics, and a volume check. The volume check is what catches an upstream…

Only when freshness changes a decision, such as fraud detection or alerting. Streaming is much harder to operate, so the test is whether anyone would act differently if the data…

Decide explicitly how late an event may be and still be included, using a watermark, and define what happens to events after that. Assuming ordered arrival produces subtly wrong…

Make each run idempotent: process a well-defined partition such as one day, write to a temporary location, validate, then replace the whole partition atomically. Reruns then…

Run an incremental sync: hash each document's content, and permissions, to skip unchanged ones, re-chunk and re-embed changed ones by replacing all their chunks, delete chunks for…

That the task definition is ambiguous, not that someone is careless. It also caps achievable model performance at around that agreement rate, since there is no stable target above…

Define each label with concrete positive and negative examples rather than adjectives, state explicitly how to handle ambiguity, include a not-applicable option, then have someone…

Get out-of-fold predictions for every example, rank by how confidently the model disagrees with the given label, and review the top of that list by hand. Estimate the error rate…

For each test example, find its nearest training examples by exact hash, fuzzy text matching or embedding similarity, and read the closest pairs. Remove or reassign duplicates,…

A piece of text that the model treats as one unit, usually a subword. Text is converted into a list of token ids before the model sees it, and the model only ever works with those…

Characters make sequences too long, and sequence length drives cost and compute. Whole words need an impossibly large vocabulary and cannot handle unseen words. Subwords give a…

About 1,300, using the rule that a token is around three quarters of a word. For anything that matters, count with the tokenizer rather than estimating.

Tokenizers are trained mainly on English text, so English gets efficient single tokens for common words. Other scripts fall back to smaller fragments, often two to three times as…

Generating tokens is sequential work: the model produces one token at a time, each conditioned on all the previous ones. Reading input is done in one parallel pass. The pricing…

Input tokens across the whole conversation, because history is resent every turn and grows quadratically. Prompt caching on the stable prefix, trimming or summarising old turns,…

Take a representative prompt and response, count both, apply the per-million rates, and multiply by expected requests. Add a margin for retries and for conversations being longer…

The maximum number of tokens a model can process in one request. It includes the system prompt, the full conversation history, any retrieved content, tool definitions and the…

The request is rejected rather than truncated. Anything building a prompt from variable-length pieces has to check the size before sending.

Cost scales with every token on every call, latency grows, and recall across a long window is uneven, with material in the middle attended to least reliably. Retrieving the…

Do not send all of it. Chunk the document, index it, retrieve the sections relevant to the question, and send only those with citations back to the source. That keeps cost flat as…

Training: split the corpus into words, represent each as a tuple of bytes or characters, then repeatedly count every adjacent pair weighted by word frequency, merge the most…

Because which sequences deserve to be one token is a property of the corpus, not of the language. Frequent sequences get merged early and become single tokens; rare ones stay…

Both follow from the segmentation. A number may be one token or three depending on whether that digit sequence was frequent enough to merge, so digits that should be treated…

system carries your instructions and is not shown to the user. user is what the person said. assistant, called model by some providers, is what the model said on previous turns.…

It says why generation ended: finished naturally, hit the token limit, or requested a tool. Hitting the limit truncates the output without raising an exception, so without…

No. It is a hard cut-off, not an instruction. The model writes as though unconstrained and gets chopped mid-word. To get a shorter answer, ask for one in the prompt.

Stateless. The model retains nothing between calls. Conversation is created by the caller resending the full history each turn. Some products offer a session id, but underneath…

Every turn resends all previous turns, so total tokens grow with the square of the number of turns rather than linearly. Most of that resent text is the model's own previous…

Prompt caching on the stable prefix first, since it is free quality-wise. Then summarise or drop older turns, keeping a short running summary. Then retrieve only relevant history…

It scales the probability distribution before a token is sampled. Low values concentrate probability on the most likely token, high values spread it out. It can only produce…

Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. This is why exact-match…

Ask for it in the prompt. max_tokens is a hard cut-off that truncates mid-sentence rather than an instruction to be concise. Keep max_tokens as a safety limit and check…

Anything constant across the conversation: role, output format, tone, and the rule for what to do when information is missing. Keeping it stable also lets prompt caching apply to…

First check whether the task and output format are fully specified, including what to do in the empty or unknown case. Then lower the temperature, since format drift is sampling…

Explicitly permit null for missing fields and state that guessing is not acceptable. Models fill gaps because the prompt demanded a value; giving them a legitimate alternative…

Only when the role genuinely narrows the domain, vocabulary or conventions. Piling on superlatives does nothing measurable. The gain comes from specificity, not flattery.

When the task is easier to demonstrate than describe: an unusual format, a subtle judgement, a tone. Start zero-shot, because examples cost tokens on every call, and add them when…

Usually three to five, chosen so each teaches something different. More similar examples add cost without adding information. Always include the empty or unknown case, or the…

The model imitates the examples closely, including any inconsistency or mistake in them, and it can over-fit to their surface pattern. It can also bias the output distribution if…

It depends on the model. On older and smaller models it gives a real improvement on multi-step tasks. Newer reasoning models do it internally, and explicit instructions can…

Score it against a fixed set of inputs with known expected outcomes, before and after. Without a baseline you are comparing against your memory of previous outputs, which is…

When it is doing several distinct jobs. Quality on each drops when they compete, and a single failure gives you no information about which step went wrong. Separate calls cost…

Temperature rescales the whole probability distribution, making peaks sharper or flatter. top-p truncates it, keeping the smallest set of tokens whose probabilities sum to p and…

Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. Locally, with a fixed…

Low temperature, around 0 to 0.2, and schema-constrained decoding if available, which makes structure a guarantee rather than a preference. Then check the stop reason, because…

Sampling several answers to the same question and taking the most common one. Independent errors rarely agree, so the majority answer is more accurate than one sample. It costs…

The response hit max_tokens. The model does not plan around that limit, it is simply stopped. Check the stop reason, raise the limit or ask for shorter output in the prompt.

Divide the logits by the temperature, sort descending, softmax, take the cumulative sum, and find the first index where it reaches p. Keep everything up to and including that…

Dividing the logits by a temperature approaching zero sends every gap between logits to infinity, so the softmax converges to one on the largest logit and zero everywhere else.…

Top-k needs the identity of the k largest, not their order, and argpartition finds them in time linear in the vocabulary rather than the n log n a full sort costs. On a vocabulary…

Two. Format drift from sampling, which shows as code fences or preamble, fixed by constrained decoding or a lower temperature. And truncation from hitting the token limit, which…

That the output is structurally valid and matches the declared types. It guarantees nothing about whether the values are correct. A schema will happily return well-formed JSON…

At each generation step the sampler is restricted to tokens that keep the output valid against the schema. Invalid continuations are given zero probability, so malformed output is…

No. Marking a field required when the source may not contain it forces the model to invent a value. Make genuinely optional fields optional and instruct the model to return null…

The model generates the output in order. If the conclusion comes first it is chosen before any supporting text is written. Putting evidence or reasoning fields before the…

Allow null for anything that may be absent and say so in the field description, include an escape value in every enum, and ask for the source text of each value so code can check…

That the categories no longer fit the traffic. Read a sample of the 'other' cases, group them, and either add categories or route them differently. It is a useful signal, which is…

Hand-label fifty or more real invoices, normalise values before comparing, and report accuracy per field. For lists such as line items report precision and recall separately. Then…

For work that can wait hours, such as processing a backlog or nightly jobs. It is much cheaper per request. It is wrong for anything a user is waiting on.

Size the worker pool from the rate limit and the call latency rather than by guessing: requests in flight equals rate times latency, so a 4,000 per minute limit with 2.5 second…

It reduces perceived latency by showing tokens as they are generated. Total time to the complete answer and total cost are unchanged.

Validation, since you cannot check a schema until the stream ends. Error handling, since a failure can arrive after the user has seen half an answer. Retries and output…

There is little point. You cannot use it until it is complete, so streaming only gives you a progress indicator. Stream prose to users, and buffer anything you intend to parse.

Something between the server and the browser is buffering the response, usually a reverse proxy or load balancer. Turn off buffering for that route, check compression settings,…

The server should notice the client disconnect and cancel the upstream request, so generation stops. Otherwise the model keeps producing tokens you pay for and nobody reads.

Time to first token, which is what users perceive as speed. Total generation time is unchanged.

The arguments come as fragments of a JSON string, split at points that respect nothing about JSON, so you accumulate them keyed by the call index and parse once the block is…

Everything sent plus everything generated: system prompt, tool definitions, all messages, all tool results, retrieved documents and the reply itself. Forgetting to reserve room…

No. Content near the start and the end is recalled more reliably than content in the middle. So position matters, and filling a large window because it exists is worse than…

Because every turn resends everything before it. The conversation grows linearly and the tokens billed grow with the square of the turn count: forty turns of a conversation that…

Old tool results, then middle conversation turns summarised, then low-scoring retrieved documents, then untested instructions. Never the original task, user corrections or agreed…

Cost and latency grow with every token, recall is weaker for material in the middle of a long context, and irrelevant text can distract the model. Sending only what is relevant is…

The goal, ids and numbers, constraints the user stated, decisions made and attempts that failed, ideally in a structured record, plus the most recent turns word for word.

Providers cache the processed form of a stable prompt prefix and charge much less to reuse it. It helps whenever calls share a long fixed preamble: a large system prompt, tool…

Any change in the prefix, since matching runs from the start and stops at the first difference. Timestamps, request ids, a user name near the top, or JSON serialised with unstable…

Check the cached token count in usage data on repeated similar requests. Consistently zero means something volatile is in the prefix. It is worth asserting on in a test, because…

When requests are too infrequent for the cache to stay warm, so most requests pay the write price, or when the prompt changes near the start so the prefix rarely matches. Work out…

Returning an answer to a question that is similar but means something different, and leaking answers between users when responses are personal. Limit it to non-personal questions,…

Break-even is a hit rate, and it comes from the multipliers: a write costs about 25% more than plain input and a read a tenth of it, so a hit has to repay the extra quarter, which…

You send messages plus tool definitions. The model responds with a structured request naming a tool and arguments rather than a final answer. Your code executes that function,…

It says when to use the tool, not only what it does, and it says what the tool returns. The model uses the description to choose between tools, so vague or overlapping…

At least two. One where the model requests the tool, one where it sees the result and answers. More if it chains several tools. This is why agents are slower and more expensive…

It constrains the argument to valid values at generation time, so an invalid status or category cannot be produced. It also removes a whole class of validation code from your…

In most cases return the error to the model as the tool result, so it can recover, retry with different arguments or explain the problem. Terminal failures such as authentication…

Execute them concurrently since they are usually network-bound, then return all results together in a single message. Splitting them across messages trains the model out of making…

It enters the context and is resent on every subsequent turn, so a large result raises the cost of the whole remaining conversation. Return only what is needed and cap anything…

Architecturally, not by instruction. Scope the tool's permissions, require an explicit confirmation argument, add a human approval gate, and make the operation idempotent so a…

A loop that calls a model, executes any tools the model requests, feeds the results back, and repeats until the model stops asking or a budget runs out.

When the steps are known in advance. A fixed pipeline is cheaper, faster and far easier to debug. Agents earn their cost only when the path depends on what is discovered along the…

In your code. The model cannot execute anything or call itself. It only emits a structured request, and your program decides whether to act on it and what to send back.

Three independent budgets: turns, wall-clock time and cost. Any one being exceeded stops the run. Turns alone are not enough because a single turn can be slow or expensive.

Detect the repeat by hashing the tool name and arguments, and when it repeats, return a result telling the model it is stuck and should try something else. Killing the run is the…

A structured partial result: what it achieved, what it could not, why it stopped, and what it cost. Raising an exception throws away work the caller could still use.

In order: make tools return less, drop stale tool results, summarise older turns while keeping recent ones verbatim, and offload large payloads to storage with an id the model can…

The original task, anything the user said including corrections and constraints, decisions already taken, and failures already hit. Losing those causes the agent to wander or…

Caching matches a stable prefix, so anything volatile must go last. If a summary is inserted near the front, it invalidates the cache every time it updates. Order from most stable…

Check the end state of the world rather than the text, because the agent's job is to do something. Then track efficiency alongside correctness: turns and cost per task. A run that…

Not necessarily. Check turns and cost as well. A change can hold the pass rate while doubling spend, and nothing in a pass-rate-only eval will tell you.

Twenty real ones beats two hundred synthetic ones. Draw them from actual usage, include every failure you have already seen in production, and version the set so a score from last…

A standard protocol for exposing tools, resources and prompt templates to model clients. It removes the repetition of rewriting the same integration per application and per…

A tool is an action the model can invoke and which usually changes something. A resource is read-only data the client can fetch, addressed by URI. Most servers implement only…

A single application with a few tools that nobody else needs. Plain function calling is simpler and has fewer moving parts. The protocol pays off with multiple clients or when you…

Answer with the specific thing you built, what it wrapped, and one design decision you had to make, such as how you capped result size or handled auth. This is asked to find out…

One clear job, a description that says when to use it rather than only what it does, bounded results, readable errors, and argument names a person would use. The description is…

The server reads them from the environment and the model never sees them. Never accept a key as a tool argument, because arguments are generated by the model and end up in…

Selection accuracy degrades, because the model is choosing between too many similar descriptions. Group related operations behind fewer tools with a mode argument, or split by…

Mainly to keep each context small and focused. One agent doing everything ends up with the relevant material buried among everything else it has accumulated. It is a context…

More calls and more money, more latency unless workers run concurrently, more failure modes, and harder debugging since a wrong answer can come from any worker or from the…

Measure. Run the same task set both ways and compare success rate, cost and latency. Start with a single loop, because it is simpler and often sufficient, and split only when the…

Orchestrator and workers, fixed pipelines, critic or reviewer loops, and parallel exploration where several attempts compete. Orchestrator and workers is the most common and the…

Without a stopping rule they never end, because a reviewer can always find something. Cap the rounds and define what acceptable means, since quality usually plateaus after the…

The safety net stops the bill, but the bug is that there is no termination condition, only a limit. Define what acceptable output is, have the critic return a structured verdict…

Trim what is no longer needed, especially old tool results. Summarise older turns while keeping recent ones verbatim. Offload large content to storage with an id. Order everything…

The risk is losing the original task, user corrections, decisions already made and failures already hit. The agent then continues confidently while solving a slightly different…

Cost scales with every token on every call, latency grows, and recall across a long window is uneven with material in the middle attended to least. A bigger window postpones the…

Memory is three parts, not a model feature: durable storage, a write policy and a retrieval policy. Store stable facts and decisions rather than transcripts, keyed by what they…

It recreates the context problem at larger scale: expensive to include and mostly irrelevant. Distilled facts and decisions are what is worth carrying between sessions.

Look at the retrieved set for that request before looking at the model. Usually one of three things: a stale memory that was never superseded, a memory retrieved above the floor…

Partial completion reported as success, confidently taking the wrong action, and compounding error where a small mistake early becomes an assumption later. None of these raise, so…

Do not trust the agent's summary. Check the world: whether each intended side effect actually happened. This is why agent evals assert on end state rather than on the text of the…

Every turn is conditioned on all previous turns, so a wrong intermediate result becomes an assumption and then a conclusion. Without a human watching, nothing interrupts the…

Idempotent tools keyed on run id and intent, so a repeated call returns the previous result instead of acting twice. Checkpoint after each turn so a resumed run continues rather…

Anything irreversible or externally visible: money, deletion, anything reaching a customer. Threshold-gate the rest by amount or volume. Gating everything defeats the purpose and…

Verify against the world in code, not by asking the model. Check that each required side effect happened. Asking the agent uses the same process that produced any mistake, so it…

A vector representing a piece of text, positioned so that text with similar meaning lands nearby. It converts semantic similarity into geometric distance, which makes similarity…

Cosine measures the angle and ignores magnitude, so it compares direction of meaning rather than length of text. Euclidean distance would let a long document score differently…

No. Each model has its own space, so the numbers are not comparable. Changing embedding model means re-embedding the entire corpus, which is why the choice is worth a little…

It measures topical similarity, not whether a passage answers the question. Text about the right subject scores highly even when it contains nothing useful. It also loses exact…

Running keyword and vector search together and fusing the results, usually by rank rather than raw score since the two scoring scales are incomparable. Keyword catches exact…

It scores each candidate against the query directly rather than comparing pre-computed vectors, which is more accurate and much slower. You use it to reorder a shortlist, never to…

Check recall first: was the correct document in the retrieved set at all? If not, the problem is chunking or embedding. If it was retrieved but ranked low, the problem is ranking,…

When exact search stops being fast enough, which is usually well past a hundred thousand chunks, or when you need operational features like filtering, replication and incremental…

Recall. It searches a promising subset rather than everything, so it can miss a vector exact search would have found. Both HNSW and IVF expose a knob trading accuracy against…

How well it filters alongside similarity search, because real queries are scoped by tenant, date and permissions. That, plus incremental update and delete, matters far more in…

Measure recall against exact search on a sample of real queries: run each with and without the index and see what fraction of the true top results the index returns. Then tune the…

The approximate index picks candidates before the filter is applied, and most of them belong to other tenants. Use iterative scanning if your version supports it, partial indexes…

Treat it as a migration. Store new vectors alongside the old, backfill, compare on the retrieval eval, switch reads if the new model wins, and keep the old vectors until you are…

Models only know their training data, so they cannot answer about your private or recent documents, and they tend to invent an answer rather than decline. RAG retrieves the…

On structure rather than a fixed character count: paragraphs and sections, keeping headings attached to their content, with slight overlap between chunks. Store the document title…

It lets users verify, it lets you debug which chunk caused a wrong answer, and it reduces invention because the model is less willing to assert something it must attribute.

Fine-tuning teaches form: a consistent output format, a domain vocabulary, a house style. RAG supplies facts. If the problem is that the model does not know something, RAG is the…

Separate the stages. Check whether the correct chunk was retrieved. If it was not, the problem is chunking, embedding or ranking. If it was, feed it deliberately and check the…

Threshold on retrieval score so a weak best match does not proceed, instruct the model that declining is acceptable, and include no-answer cases in the eval set. A confident wrong…

Rewriting the user's question into a standalone search query using the conversation context. Follow-up questions like 'and for international?' are meaningless to a search index on…

Re-index on change rather than rebuilding, keyed by content hash. Delete chunks for removed documents. Store and display a last-updated date, because a stale answer is…

Real documents are PDFs with multi-column layouts and tables, HTML full of navigation, and scans with no text layer. Extraction frequently produces the wrong reading order or…

Extract them separately rather than letting them flatten into a run of numbers, or send the page to a multimodal model as an image. A flattened table is worse than useless because…

Navigation, footers, cookie banners and sidebars, because repeated boilerplate appears in every chunk and destroys the discriminating power of retrieval. Keep the heading…

That it depends on how the documents are structured and has to be measured, and that it is usually not the biggest lever available. In a comparison across four splitters on two…

Split on structure rather than a fixed character count: sections, then paragraphs, then sentences, never mid-sentence. Around 200 to 500 tokens with a sentence or two of overlap,…

Small chunks retrieve precisely but often lack the context to be useful on their own. Large chunks carry context but dilute the embedding so they match many things weakly. The…

So an answer that straddles a boundary is not lost from both chunks. It costs a little storage and duplication and prevents a whole class of silent retrieval failure.

Recall and precision want different things. The first stage is cheap and broad so the answer is definitely in the candidate set. The second is expensive and accurate and orders a…

A BM25 score and a cosine similarity are on different scales with no meaningful conversion, so combining raw scores is arbitrary. Reciprocal rank fusion uses only position, which…

It reads query and candidate together and scores the pair, which is much more accurate than comparing independently computed vectors. It also means one forward pass per candidate,…

Follow-ups depend on earlier turns, so embedding the raw message retrieves the wrong things. Rewrite each turn into a standalone question using the history before retrieving, and…

Mean reciprocal rank averages one over the position of the first relevant result. It rewards getting a relevant result to the very top, which matters when only the first few…

When questions and passages are written differently enough that a question embedding sits far from the passage that answers it. You ask a model for the answer it would expect and…

Build a small labelled set: twenty or so real turns from a chat log with the chunks that answer them. Measure recall@k and MRR for each technique added one at a time, and record…

A trivial baseline, then logistic regression for an interpretable reference, then gradient boosted trees, which are the strongest general option on tabular data. A neural network…

Tabular features are heterogeneous and many are uninformative, and trees select among them rather than having to learn to ignore. Trees are also invariant to feature scaling,…

Bagging, as in random forests, trains many independent models on resampled data and averages them, which reduces variance. Boosting trains models sequentially with each correcting…

It evaluates every feature and threshold and picks the split that most reduces impurity in the child nodes, measured by Gini impurity or entropy for classification and variance…

Each tree is fitted to the gradient of the loss with respect to the current ensemble's predictions, so adding it moves the ensemble towards lower loss. It is gradient descent over…

Not when features are correlated. The coefficients become unstable and can flip sign between runs while predictions barely change, so they stop meaning what they appear to.

Look at the data and target balance, hold out a test set, get a dummy baseline, build a preprocessing pipeline so cross-validation is leak-free, try a regularised linear model…

Error from a model has a part from wrong assumptions, bias, and a part from sensitivity to the particular training sample, variance. Simple models have high bias and low variance;…

First find out why it is missing, since missingness can itself be informative. Options are imputation with a missing indicator, models that handle missing values natively such as…

When interpretability or calibrated, stable probabilities matter, when data is small, when features are already well engineered and roughly linear in effect, or when the model…

As the number of features grows, data becomes sparse in the space and distances between points become similar, so methods that rely on neighbourhoods, like k-nearest neighbours,…

It adds a cost for model complexity, such as large weights, so the model prefers simpler explanations that generalise better. L2 shrinks weights smoothly; L1 drives many to zero.…

When a miss costs more than a false alarm: disease screening, fraud, safety. When a false alarm is expensive, such as a spam filter deleting real mail, precision matters more. The…

It includes the true negative rate, and with overwhelming negatives that stays high regardless, flattering the score. Precision-recall AUC ignores true negatives and is the honest…

Information from the test set influencing training. Common routes are fitting a scaler or imputer before splitting, features built from future information, duplicate rows across…

From the business requirement rather than the default of 0.5. Plot the precision-recall curve and pick the point meeting the constraint, such as maximum recall while precision…

When large errors are disproportionately costly, since RMSE squares errors before averaging and so penalises big misses heavily. MAE treats all errors equally and is easier to…

It divides by the actual value, so it becomes huge or undefined when actual values are near zero, and it treats over- and under-prediction asymmetrically.

When classes are imbalanced and each one matters. Accuracy is dominated by the common classes; macro F1 averages per-class F1 equally, so a model that ignores a rare class is…

Start from the process: investigators can review a fixed number of claims a week, and a missed fraud costs far more than a review. So the question is how much fraud is caught in…

Training performance far exceeds held-out performance. If both are poor it is underfitting, which is a different problem needing more capacity rather than less.

More data will not help, because the model has already extracted what it can. The answer is more capacity, better features, or a different model class.

Randomly ignoring a fraction of units during training, so the network cannot depend on any single one and must learn redundant representations. It behaves like training an…

Most often leakage in the validation setup: the same entity on both sides of the split, preprocessing fitted on all data, a feature that is only known after the outcome, or…

L2 penalises squared weights and shrinks them all smoothly. L1 penalises absolute weights and drives many to exactly zero, which also performs feature selection.

For linear models, neural networks and distance-based methods, where a feature on a larger numeric scale otherwise dominates. Tree models are invariant to monotonic…

Replacing a category with the mean target value for that category. It is powerful and leaks the label directly unless computed within each training fold, in which case the offline…

Whether it leaks. Is it populated only after the outcome is known, does it correlate with the label through how the data was ordered, is it an aggregate computed over the whole…

Behaviour aggregates over several windows, such as usage, logins and support tickets in the last 7, 30 and 90 days, trends between windows, tenure, plan and billing features, all…

Replacing a category with the average outcome for that category. The risk is leakage: each row's own label influences its encoding. Compute it out of fold, and smooth rare…

I would generate random data with the same shape and range and cluster it the same way. k-means returns k tidy groups from uniform noise, and both the elbow and the silhouette…

Where adding another cluster stops buying much reduction in within-cluster variance. That is a hint, not an answer: it should be checked against silhouette score and against…

Finding the orthogonal directions of greatest variance and projecting the data onto the first few. It compresses while keeping as much variance as possible, and the components are…

It optimises for preserving local neighbourhoods, so global structure is distorted. Cluster sizes and between-cluster distances in the visualisation carry no reliable meaning.

Learn what normal looks like rather than training on the rare anomalies. Isolation forests work well on tabular data because anomalies isolate in few random splits; distance to…

By inspection and stability. Read samples from each cluster to judge coherence, and rerun with different seeds to see whether the grouping holds.

Try a range and compare internal scores such as silhouette and the elbow in within-cluster variance, then check stability across seeds and subsamples, and above all whether the…

Embed the questions, cluster them with a method that tolerates noise, rank clusters by size and growth, name them from samples with a model, and verify a few by reading. Then…

Without them, stacked linear layers collapse into a single linear transformation, since a matrix multiply of matrix multiplies is one matrix multiply. The non-linearity is what…

Sigmoid saturates at both ends, so gradients approach zero and early layers stop learning. ReLU does not saturate for positive inputs, keeping gradients alive, and it is cheaper…

A unit whose input is always negative outputs zero and has zero gradient, so it never updates and is permanently dead. Leaky ReLU and GELU avoid it by keeping a small gradient for…

Cross-entropy, usually with softmax at the output. It penalises confident wrong predictions heavily, which is the behaviour you want, whereas squared error treats a confidently…

They add each layer's input to its output, giving gradients a direct path back through the network. Without them gradients shrink through each layer's transformation; with them…

Layer norm normalises across each example's features independently, so it does not depend on batch composition or size, which suits variable-length sequences. Batch norm's…

Run the forward pass, keeping intermediate values. The gradient of the loss with respect to the logits is the probabilities minus the targets. From there, the output weight…

The model is overfitting: memorising training examples rather than learning patterns that generalise. Stop at the best validation point, then add regularisation such as weight…

First check the model can overfit a single batch, which isolates whether the setup works at all. Then the learning rate, which dominates everything. Then the data: labels correct,…

It adapts the step size per parameter using running estimates of the gradient and its square, so parameters with consistently small gradients still move. It is less sensitive to…

Early in training the weights are random and gradients can be large, so a full-size step can destabilise the run. Warming up from a small rate avoids that, and it matters most for…

Gradients accumulate across batches instead of being replaced, so each update is the sum of all previous gradients. Updates grow and training diverges, with no error raised.

Larger batches give less noisy gradients, which tolerate a larger learning rate, so the two are often scaled together. Changing one alone can destabilise or slow training.

Adam rescales gradients per parameter, so an L2 term in the loss does not act like true weight decay. AdamW applies decay directly to the weights, decoupled from the gradient…

A gradient check. Compute the analytic gradient, then for each parameter nudge it by about 1e-5 in both directions, recompute the loss, and divide the difference by twice the…

During the forward pass each operation records its inputs and how to compute its local derivative. Backward starts with a gradient of 1 at the output and visits nodes in reverse…

Backpropagation multiplies local derivatives layer by layer; if they are consistently below one, as with saturating activations or poorly scaled weights, gradients shrink…

Switches layers that behave differently at inference, notably dropout and batch normalisation, into evaluation mode. Forgetting it makes validation numbers noisy and pessimistic,…

It stops the framework recording operations for gradient computation, which is unnecessary during evaluation. It saves memory and time, and without it a large validation pass can…

Almost certainly data loading, not the model. Raise num_workers, enable pin_memory, and check for expensive per-item preprocessing that should be precomputed or moved onto the GPU.

The operation that produced it was not recorded in the autograd graph, usually because it was detached, created from a Python number, or computed inside no_grad. Autograd can only…

Profile first, because the bottleneck is usually data loading or synchronisation rather than arithmetic. Then mixed precision, more data loader workers, removing per-step .item()…

Shapes. A prediction of shape [N, 1] subtracted from targets of shape [N] broadcasts to [N, N] and computes a meaningless loss without error. Assert shapes where the loss is…

Token ids go through token and position embeddings and are added. A stack of blocks follows, each with layer norm, causal multi-head self-attention added back to the residual,…

Check the initial loss against the expected value for uniform guessing, try to overfit one small batch, decode and inspect a batch with its labels, sweep the learning rate by…

Model weights, optimiser state, scheduler state, the step or epoch counter, random number generator states, and the position in the data. Missing the optimiser state is the common…

Each position produces a query, a key and a value. The query is compared against every key to produce weights, and the output is a weighted mixture of the values. It lets any…

Recurrence processes sequentially, so distant information degrades through a carried summary and nothing parallelises across the sequence. Attention connects any two positions…

Attention compares every position with every other, so cost grows with the square of sequence length. Doubling the input quadruples the work, which is why context limits exist and…

Each head learns a different kind of relationship, such as grammatical agreement in one and topical relevance in another, and their outputs are combined. One set of weights can…

It prevents each position attending to later positions, so the model learns left-to-right prediction. It also lets the full sequence be trained in parallel, since every position…

Attention moves information between positions. The feed-forward layer transforms each position independently. A block alternates the two, with residual connections around each.

Attention compares every position with every other and blends the results, and none of those operations knows where anything is. With no position signal the model is exactly…

Two different jobs. The residual makes each block add to a running vector rather than replace it, so information survives dozens of layers and gradients have a short path back;…

Stored keys and values for tokens already processed, so each generation step computes only the new position rather than recomputing the whole prefix. It makes generation practical…

Several query heads share one key and value head instead of each having its own. The KV cache shrinks by exactly that ratio, so a model with 32 query heads and 8 key and value…

A learned table has a fixed number of rows, so there is nothing for a position beyond the training length, and it encodes absolute position when what attention needs is relative…

It is a mixture of experts: a router picks a couple of experts per token, so compute per token is that of a 5 billion parameter model while all 120 billion must sit in memory. It…

No. It computes exactly the same attention, tiling the work so blocks fit in on-chip memory and using a running softmax so the full scores matrix is never written to GPU memory.…

Project the input three times to get q, k and v of shape (T, d_k). Compute scores as q @ k.T divided by the square root of d_k, giving (T, T). Set the strictly upper triangle to…

Input (2, 6, 16). Each projection keeps (2, 6, 16). Reshape to (2, 6, 4, 4) and transpose to (2, 4, 6, 4), so heads sit in front of positions. Scores are (2, 4, 6, 6), one square…

Purely for numerical stability. Exponentiating a large score overflows to infinity in floating point and the division then gives NaN. Subtracting the row maximum is algebraically…

The mask is being applied after the softmax rather than before. The first position can see one entry out of many, so once the remaining entries are zeroed most of its normalised…

By predicting the next token across an enormous corpus. General capability is a side effect: predicting well requires modelling whatever produced the text, so reasoning, world…

Loss falls predictably as a power law in compute, data and parameters, across many orders of magnitude. They let a lab decide how to spend a compute budget before committing, and…

The objective rewards plausible continuations, not calibrated uncertainty. Fluency and correctness were never separated during pretraining, so a wrong answer is produced with the…

Compute is about six FLOPs per parameter per token, so seven billion parameters on a compute-optimal 140 billion tokens is about 5.9e21 FLOPs. An H100 does 989 TFLOP/s of bf16…

C equals 6ND fixes the trade, so choosing a size chooses the token count. The compute-optimal point is around twenty tokens per parameter, which for a 1e22 FLOP budget lands near…

Increasingly the data: deduplication, quality filtering and the mixture of sources. Architectures have largely converged, and much of the remaining gap between models comes from…

Repeated text is memorised rather than generalised from, and it wastes compute on seeing the same thing many times. Near-duplicates matter as much as exact copies.

Start from web crawls, extract main text, identify language, apply heuristic filters, use a quality classifier, deduplicate exactly and approximately, remove personal and unsafe…

Check whether the comparison used the same prompts, settings and number of shots, whether contamination was addressed, whether the benchmarks match my use case, and then run my…

When the need is consistent form rather than facts: a fixed output format, a domain style, or making a small model good enough at one narrow task to cut cost and latency. If the…

Parameter-efficient fine-tuning. The original weights are frozen and a small number of added parameters are trained, reaching most of the benefit at a fraction of the cost. The…

A dataset you must build and keep current, redoing the work when you change base model, and a maintenance obligation a prompt does not have. That is why it sits last in the order…

Training a smaller model to imitate a larger one's outputs. For LLM products it is frequently the largest cost reduction available, because a distilled small model can match the…

Depends on the goal: a few hundred clean examples for format or style, more for new behaviour. Quality and consistency matter more than volume, since the model learns exactly what…

Rarely from writing them by hand, which stalls at a few hundred. The two that work are a larger model's output filtered through your own evaluation, which is cheap and bootstraps…

Fine-tuning on a narrow task degrading the model's general abilities. LoRA reduces it by leaving the original weights untouched, and mixing general examples into training helps…

Score both on the identical held-out set with the identical grader in the same run, and keep the pairing. Report the disagreement counts rather than just the two accuracies: 78…

Separate. Merging gives five full copies of the base weights, which for an 8B model at fp16 is about fifteen gigabytes each, against one shared base plus five adapters of a few…

Keep the base weight frozen and add a parallel branch: x through A, which is (r, d_in), then through B, which is (d_out, r), scaled by alpha over r, added to the base output. A is…

Because LoRA is a parallel branch that is added to the output, and addition of a low-rank product into the weight matrix is exact: W plus alpha over r times B times A has the same…

Unmerged. Fifty merged models is fifty copies of the base weights in memory, where fifty adapters at r=16 are a few tens of megabytes each on top of one shared base, and the…

The attention projections first — query and value are the usual default, and they were what the original work found sufficient. Adding key and output projections, and then the…

It continues text rather than answering. A question may be continued with more questions, because that is a plausible continuation. Instruction tuning teaches the question-answer…

RLHF trains a separate reward model on human preferences and then optimises the policy against it with reinforcement learning. DPO optimises directly against the preference pairs…

Preference data rewards answers people like, and people tend to prefer agreement. The training signal therefore encourages agreeing, and separating being helpful from being…

People rank far more consistently than they assign absolute scores, so comparisons produce cleaner training signal. The quality of that data bounds the whole post-training…

It penalises the policy for moving too far from the original model. Without it, optimising against the reward model finds exploits: odd or sycophantic outputs that score well and…

A model critiques and revises its own outputs against written principles, producing preference data without human annotators for every comparison. It scales feedback and moves…

The goal is to learn how to respond. Including prompt tokens spends capacity learning to generate user messages and system prompts, and can shift the model's behaviour in unwanted…

How far the trained model may move from the reference model. Low beta allows the policy to deviate a lot to fit the preferences; high beta keeps it close to the reference, like a…

Take questions with known answers, keep the ones it answers correctly, push back mildly without new evidence, and measure how often correct answers change to wrong. Also test…

Memory bandwidth. Generating each token requires reading all the model weights from memory, so the GPU is mostly waiting on memory rather than computing. That is why quantization…

Accuracy for memory and speed. 8 bit usually costs little, 4 bit costs something real and task-dependent. The loss is not uniform across tasks, so it has to be measured on your…

The weights are read once and applied to every request in the batch, and reading them is the bottleneck. This is why hosted APIs are cheap and why self-hosting for low traffic is…

When data cannot leave your infrastructure, when volume is high and steady enough to keep hardware busy, when you need a fine-tuned model nobody hosts, or when latency rules out a…

Work it out. At 16-bit the weights alone are 16 GB. The KV cache for a typical 8B design is about 128 KiB per token, so 16 users at 8k tokens need about 16 GiB. That does not fit.…

Generating each token requires reading all the weights from memory, and that reading is the bottleneck. Smaller weights mean less to read per token, so each step is faster even if…

They are different workloads. Prefill processes the whole prompt in parallel, so it is compute-bound and scales with prompt length, and it is what time to first token measures.…

At batch size one, each token requires reading every weight, so divide the weight bytes by the memory bandwidth. An 8B model at 16-bit is 16 GB, and on a 2 TB/s card that is 8 ms…

Decode is memory-bound, so one step reads the weights once regardless of how many sequences are in the batch, and extra sequences use arithmetic that would otherwise be idle.…

A small draft model proposes several tokens and the large model verifies them in one pass, accepting the longest prefix it agrees with. It is exact: the output distribution is the…

Prefill runs the prompt through in one batched pass, with the causal mask, and keeps the keys and values. Each decode step then embeds the single new token, computes one query,…

Because it is used exactly once. The query for position t is compared against every key up to t to produce that position's output, and no later position ever refers to it — later…

Each step re-projects the whole prefix, so step t does work proportional to t rather than to one. Summed over 500 steps from a base of 2,000 that is roughly 1.1 million…

It is the same data, kept between requests rather than within one. A shared prefix produces identical keys and values every time it is processed, because nothing after it can…

Inference needs the weights. Training also needs gradients, optimiser state which for Adam is several times the model size, and the activations retained for the backward pass.

Data parallelism replicates the whole model on each device and splits the batch, requiring the model to fit on one device. Tensor parallelism splits individual layers across…

Communication bandwidth. Synchronising gradients every step moves a large amount of data, and across machines that frequently dominates compute time, which is why interconnect…

Stage 1 shards optimiser state, stage 2 adds gradients, stage 3 adds the weights. Each saves more memory and adds more communication, with weights being the costliest because they…

Storing only some forward-pass activations and recomputing the rest during the backward pass. It trades roughly a third more compute for a large reduction in memory, and is often…

Each process runs forward and backward on its own slice of the batch. During the backward pass, gradients are averaged across processes with an all-reduce, overlapped with the…

When the model's weights, gradients and optimiser state do not fit on one GPU with plain replication. FSDP shards all three and gathers each layer's weights just in time, at the…

About 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients, 4 for 32-bit master weights and 8 for Adam's two moments. That is about 112 GB before activations, so it needs…

The core operation is matrix multiplication, which is thousands of independent multiply-accumulate operations. A GPU has thousands of simple cores suited to exactly that, while a…

Using 16-bit for most operations and 32-bit where precision matters, with loss scaling to stop small gradients underflowing. It roughly halves memory and can double throughput at…

The GPU is waiting, almost always for data. Raise data loader workers, enable pinned memory, and check for per-item preprocessing that should be precomputed or moved onto the…

Generating one token at a time is typically memory-bound, since each step reads all the weights to do comparatively little arithmetic. Batching many requests together is the main…

Combines several operations into one kernel so intermediate results stay on-chip instead of being written to and read from memory between steps. The saving is in memory traffic…

Estimate its arithmetic intensity, FLOPs per byte moved, and compare with the hardware's ratio of peak compute to bandwidth. Then confirm with a profiler: memory-bound kernels…

Check it fits with room for the KV cache at the required context and concurrency, likely 16-bit or quantized weights on a 24 GB or larger card. Then benchmark candidate GPUs with…

A fixed set of inputs with a defined way of scoring the output, run on every change, producing a number you can compare between versions. It is the equivalent of a test suite for…

Model output varies between runs and there is usually no single correct string. Exact-match assertions flake and get deleted. Evals score properties and use graded judgement…

Read real traces first and write down how it fails, so the eval measures actual problems. Then take twenty real inputs, including ones that have failed, write down what a good…

Get full traces logged if they are not already. Read about a hundred real ones, write a note on the first failure in each, group the notes into failure modes and count them. Fix…

Later failures are often consequences of the first. If retrieval fetched the wrong document, the wrong answer that follows is not a separate problem. Counting both inflates the…

A binary question about one failure mode can be checked against human labels and has a clear meaning. A five point scale has no agreed meaning between adjacent points, drifts…

Real traffic, not imagination. Common cases in realistic proportions, every production failure you have seen, edge cases, and cases where the right answer is to refuse or return…

Tuning against every case overfits the prompt to that set, so the score improves while real performance does not. A held-out split you never inspect gives a number that still…

Twenty real cases is enough to start and already catches regressions. A few hundred gives tighter confidence. Beyond that, variety matters far more than volume.

Not with the general eval set. At 6% prevalence, a hundred random cases contain about six that exercise the bug, and a paired test on that few detects a working fix less than a…

Use deterministic checks for everything checkable first. For what remains, break 'quality' into specific failure modes you have seen, such as unsupported claims or missing the…

Preference for longer answers, preference for its own writing style, sensitivity to the order of compared options, and general leniency. Each has a mitigation: rule length out in…

Not much, for two reasons. Without a measured comparison against human grading the number is unanchored. And an average on a five point scale does not say what is wrong. I would…

That number on its own cannot answer it, because it depends entirely on how often failures occur. If 12% of outputs are bad, a judge that simply passes everything already agrees…

Run every comparison twice with the answers in both orders, and average. The gap between the win rate in one order and the other is the size of the bias, and it is worth reporting…

Run a small fast subset on every push and the full set nightly or before release. Cache results for unchanged inputs, and report the cost of the eval run itself so the trade-off…

Cost and latency, because changes routinely trade quality against them, and a quality-only report hides that. Also which specific cases changed state, since that is what you…

Sample from real traffic periodically and add every production failure as a case. Version the set, and re-baseline when it changes so scores stay comparable.

By measuring, not by taste. Run the suite ten times against an unchanged commit and see how far the score moves on its own; that spread is the floor. Then simulate the gate…

Build tasks in a sandboxed copy of the systems it touches, and grade the end state: the right order changed, nothing else changed, the refund amount is correct. Separately check…

It does not tell you where failures start, it ignores side effects on other data, and it ignores cost and step count. Two runs can reach the same answer, one cleanly and one after…

Walk each failed trace and mark the earliest step that went wrong. Tallied across runs, it shows which kind of step fails most, such as filling tool arguments. That points…

Not on that evidence. The interval on 60 cases is roughly plus or minus ten points, so the difference is well inside noise. I would look at the cases where the versions disagree,…

Most cases pass or fail for both versions, and those add noise without information. Looking only at disagreements removes that shared noise, so a real difference shows up with far…

Grade a sample by hand, making sure it contains real failures, and measure on labels the judge was not tuned on how often it catches the failures and how often it accepts the…

Preference for length, preference for its own style, sensitivity to the order of compared options, and general leniency. Mitigate by ruling length out in the rubric, using a…

When comparing two versions of a system on open-ended quality, such as which reply is more helpful. Asking which is better is more reliable than asking for an absolute score,…

Usually yes, briefly, before the verdict. A short critique first tends to make the verdict more accurate, and when the judge disagrees with you it shows whether the rubric or the…

Run validated pass or fail judges for the main failure modes on a random daily sample, plus more of the high-stakes traffic. Track the failure rate over time with an error bar,…

Not exactly. The judge misses some failures and wrongly fails some passes. With its catch and accept rates measured on labelled data you can correct the estimate, and you should…

Score a baseline, make one change, score again, keep or revert, repeat. Hold out a test split you never inspect during iteration and report that number, and track cost alongside…

Because prompt tuning overfits exactly as model training does. Iterating against every case drives the score up while real performance stays flat. The held-out number is the one…

When several consecutive changes produce nothing, when the remaining failures are cases humans also disagree on, when further improvement costs more than it is worth, or when the…

Read a hundred or so traces and categorise the failures, count them, and build an eval around the top modes with a held-out split. Then make one change at a time, starting with…

Pull the trace for that run. Read what was actually sent at each turn, which tools were called with which arguments, what came back, and where it diverged. Most surprising…

A run id to tie the run together, the step and turn, model, input and output tokens, cached tokens, cost, latency and stop reason. For tools, the name, arguments, success, error…

It is the only way to tell whether prompt caching is working. If cached tokens are consistently zero, something volatile has crept into the prompt prefix and your largest cost…

Keep failures, slow and expensive runs, and a sample of normal ones. Categorise failures and fix the largest category first, and add each real failure to the eval set so it cannot…

Useful as a pointer to where to look, and noisy and biased as a measure, since only some users click. Behavioural signals such as retries and abandonment are often more…

Cost per completed task rather than per call, including retries and failed runs. Tag every call with a task id so a whole job can be summed, and record tokens, cost and latency…

Two different things. Time to first token is prefill over the prompt, which is compute-bound and superlinear in prompt length because attention is quadratic, so a long prompt…

It makes it feel faster by showing tokens as they arrive. Total time to the complete answer and total cost are unchanged.

Measure first and find which step dominates. Then in order: prompt caching on the stable prefix, trim what is sent including history and tool results, route easy cases to a…

The provider caches the processed form of a stable prompt prefix and charges much less to reuse it. It is prefix-matched, so any change early in the prompt invalidates the rest. A…

When you cannot measure which requests are hard, or when a failure is expensive. Routing adds a decision that can be wrong, so it needs an eval showing quality holds and a…

Untrusted content containing instructions that the model follows. It works because a model has no separation between instructions and data: the system prompt, user input,…

The instruction can be rephrased, translated, encoded or split, legitimate content triggers false positives, and the filter itself has the same weakness. Filtering raises the cost…

Direct is the user typing overriding instructions, which mainly affects their own session. Indirect is an instruction hidden in content the agent reads while working, such as a…

Architecturally, not textually. Scope tool permissions to the minimum, require human approval on irreversible or externally visible actions, separate the component that reads…

An email can contain instructions the agent follows, so refunds become attacker-controllable. Split it: an extractor with no tools reads the email and emits structured fields; a…

Through any outbound request it can influence. A fetched URL containing the data in its path or query, or a rendered markdown image, both make a request without looking like…

Requests take seconds not milliseconds, concurrency is governed by the provider's rate limit rather than your CPU, streaming means long-lived connections, cost varies per request…

Retry briefly with backoff, then fall back to an alternative model, then serve cached results if reasonable, then return an honest message. Decide this before the outage, because…

Latency at p50 and p95 split by step, error rate broken down by type, cost per request and per hour with an alert, token usage including cached tokens to confirm caching works,…

By writing the latency budget: every component on the path with its p50 and p99, measured rather than guessed. On a typical path with two feature lookups, the model forward pass…

Like code: versioned, tested against the eval suite before release, and rolled out gradually with the ability to revert. A prompt edit can change behaviour as much as a code…

Check whether the provider is having issues, then what recently changed, including prompt edits and model versions. Mitigate first by rolling back or switching to a fallback, then…

Long enough to collect the sample the metric needs, which is a calculation rather than a judgement. A five per cent canary on a fifteen-requests-a-second service needs about…

Three things, and they need different defences. Feature skew, where serving computes a feature differently from training, fixed by one implementation and preprocessing saved…

By Little's law about 400 concurrent requests. Every layer, from server workers and connection pools to the provider's concurrency, requests per minute and tokens per minute…

Continuous batching keeps the GPU full by adding and removing sequences at every step, and paged attention allocates KV cache memory in blocks so many more sequences fit. Together…

Data drift is the inputs changing; concept drift is the relationship between inputs and outputs changing. Detect it by monitoring input and prediction distributions against a…

Code, data, hyperparameters and environment. Data is the one people omit and the one that makes reproduction impossible, so datasets need a content hash or an immutable versioned…

Shadow deployment first, running it on live traffic without serving its output so you can compare per request rather than by distribution. Then a canary on a small share with…

Features computed differently at training and serving time, so the deployed model sees inputs unlike its training data. It degrades accuracy with no error, and sharing one…

One computation of each feature served to both training and inference, removing skew by construction, plus point-in-time correct training sets so features reflect only what was…

Monitor inputs and outputs daily against a reference: feature drift with PSI or similar, the score distribution, the alert rate and data quality. Use proxies such as analyst…

Validate offline against the current model on the same held-out data, register it, run it in shadow on live traffic and compare outputs, then canary a small share with business…

Clarify first: corpus size, freshness, latency, whether citations are required, budget. Then ingestion and structural chunking, hybrid retrieval with reranking, generation with…

Assume it will be. Show sources so users can verify, make correction cheap, keep a human in the loop where being wrong is expensive, gate destructive actions behind confirmation,…

A fixed eval set run on every change, reporting quality, cost and latency together, with a held-out split. Plus production monitoring on the same signals, and sampling real…

Store permission groups as chunk metadata and filter at retrieval time, so the model never receives text the user may not see. Instructing the model to withhold restricted content…

Ingestion, retrieval with permission filtering, and a retrieval eval, because retrieval quality and access control decide whether the system works. Generation is comparatively…

Architecturally: a narrow refund tool with a threshold above which it returns a proposal requiring human approval, idempotency keys so retries cannot double-refund, and never…

Against the end state of test accounts rather than the agent's words: whether the correct action happened, for the right amount, exactly once. Include adversarial cases where the…

Cache first, since it is the cheapest win, with an expiry chosen from how stale the data may acceptably be. Then read replicas, accepting replication lag. Sharding only when…

Many requests miss the same key simultaneously and all recompute the expensive value at once. Prevent it with a lock so only one recomputes while the others wait, or by staggering…

Queues generally guarantee at-least-once delivery, so a message can be delivered twice after a failure or a timeout. If the handler is not idempotent, that duplicate does the work…

Replicas converge over time rather than immediately, so a read straight after a write may return the old value. Most systems accept this in exchange for availability under…

Token bucket: tokens refill at a fixed rate, each request spends one, and bursts are allowed up to the bucket size. Distributed, it needs shared state in a fast store with atomic…

Both. Fan-out on write makes reads fast at the cost of expensive writes, which fails for accounts with millions of followers. Those use fan-out on read. The hybrid is the standard…

Assume about 20 messages per user per day at roughly 1 KB each including metadata: 200 million messages a day, about 73 billion a year, around 73 TB, roughly 220 TB with three…

Not 99.9. Availability multiplies, so 0.999 to the eighth is about 99.2 per cent, which is around 4,200 minutes of downtime a year. To promise more than my dependencies I have to…

Use an idempotency key generated by the client per payment attempt. The server stores the key and the outcome atomically with the charge, and returns the stored outcome for a…

For scoped, well-understood pieces where I can review the result quickly: boilerplate, test scaffolding, a function with a clear contract. I keep the design and the structure…

Scope the request narrowly enough that the output is reviewable, read the diff rather than the summary, check it against the actual requirement rather than plausibility, and run…

When I do not yet understand the problem, because I would be delegating the thinking rather than the typing. Also on anything security-sensitive or subtly stateful, where a…

Clarify the requirements and state my assumptions. Then get a plan and revise it before any code is generated. Then work in small pieces I can review properly. Generating first…

Read the diff rather than the summary, check it against the requirement rather than plausibility, and pay particular attention to error handling and boundary cases, which is where…

Stop and understand it, or replace it with something I do. Shipping code nobody on the team understands is a liability, and in an interview it guarantees failing the part where…

Triage rather than volume. Abstract, figures and conclusion for most, method and results for the promising ones, full depth only for what I will build on. Breadth comes from…

A weak baseline first, since most disputes are really about baselines. Then single-seed results with no variance reported, small effects framed as large, and missing ablations,…

Compare the setting rather than the number. Is the baseline the thing I actually run, is the data distribution like mine, and is the effect larger than the seed spread they…

When I am about to build on it, and then scoped to one figure at the smallest scale where the effect should still appear, with my own baseline run first. A handful a year. What it…

The absolute size of the improvement, whether there are error bars or multiple seeds, and whether the baselines were run under the same conditions. Then whether the claimed result…

Single-seed results, a large gain from a small change with no mechanism, an untuned baseline, no released code, and improvements that only appear on the authors' own benchmark.

Identify every symbol and its shape, rewrite it in words, substitute small numbers and check extreme cases, then write it as code with shape comments and test it on a case where I…

Check the evaluation setup matches exactly, check whether the gap is within seed variance, compare against the official code for undocumented details, verify the baseline matches…

A baseline tuned as hard as the method, one variable changed at a time, ablations showing each component earns its place, and several seeds with variance reported. A difference…

Removing one component at a time and measuring the loss, to show each part contributes. It often reveals that something assumed important does not help, which is better to find…

Enough to know the variance, usually three to five for expensive runs and more when cheap. Without knowing the spread you cannot say whether a difference means anything.

Something that changed alongside the variable you meant to test, such as extra compute or longer training. Control it by matching the baseline on everything except the one change,…

With many metrics available, one will improve by chance. Committing beforehand turns the result into a genuine test rather than a search for something that looks good.

Run both with several seeds to measure training variance, compute a paired bootstrap interval over the test examples, and check the gap is clearly larger than both sources of…

Run the baseline plus each component alone, and the full method minus each component, with several seeds each. Compare effects against seed variation, and check interactions for…

Problem and why it matters, what is missing from existing approaches, what you did, what happened with numbers, and what it means and does not mean. Written so the abstract and…

An unacknowledged limitation found by a reviewer reads as a flaw. The same limitation stated by the author reads as scope and rigour. It also stops readers drawing conclusions the…

At the highest rung the evidence reaches, which is the specific one: accuracy improved by 3.2 points on GSM8K with Llama-3-8B, averaged over five seeds with a standard deviation…

Ask a specific question, and ask the reader to state the main contribution in one sentence. If they cannot, the draft is not communicating it, whatever the quality of the prose.

Treat it as evidence the draft allowed the misunderstanding and revise the text so the next reader does not repeat it. Arguing the reviewer was wrong rarely helps.

Accept the observation behind the request before addressing the request, because the observation is usually correct even when the proposed fix is not. Then narrow whatever claim…

Work in one pass over mechanical things before touching anything else: delete the throat-clearing openers, replace quantity words with quantities, cut stacked hedges down to one…

Around the claims in the introduction. Each claim gets the experiment that supports it, with fair baselines and variance reported, followed by ablations that attribute the gain…

Describe the post, the one idea it made, the evidence behind it, what readers asked or pushed back on, and what you changed as a result. Having a link ready is better than any…

Name something specific, say why it matters, why it is tractable now rather than five years ago, and what your first experiment would be. A vague area name reveals that you have…

Answer with a real opinion and a reason, and be fair to the other side. The question assesses independent judgement, and refusing to have a view scores worse than having a…

Something specific you expected to go one way and did not, and what you concluded. Following up on your own surprises is how directions are found, and the answer shows whether you…

Read for gaps rather than results, especially limitations sections, which are authors honestly saying where the work stops. Keep a running list of questions you cannot answer,…

Follow it, because a surprise is a gap in my model of the field and those are where directions come from. I write it down rather than shrugging it off, which is what most people…

Design the cheapest experiment that could falsify it and run that first, and write down in advance what result would make me stop. Then judge whether the next month is the best…

When the failure is mechanical rather than conceptual and I can name exactly what went wrong, or when strong people keep independently suggesting the idea. Not when I am…

The quantity, the comparison, the noise floor measured on the baseline first, and the threshold that makes me stop — plus an explicit list of the moves I am not allowed to make…

State the question in one sentence, why it matters and to whom, why it is tractable now, the first cheap experiment and what result would make you stop, and what you would do if…

Describe what was believed before, what had changed to make it tractable, why it was a natural next step others missed, and why the answer would have been informative either way.…

Pick one and be fair to it, since the useful cases are the ones where the argument was good. Then name the signals: results demonstrated only at small scale, a cost that grows…

lsof -i :PORT, or ss -tlnp on Linux. It gives you the process id, which you can then inspect or kill.

Whether the process is running, the recent logs filtered for errors, disk and memory, and whether it can reach its dependencies. Then follow a single request id through the log to…

Environment variables injected by the platform or a secret manager, never committed files. A gitignored .env with restrictive permissions is acceptable locally and not in…

Check the job's log for a traceback, then system signs: whether the disk filled, whether the process was killed for running out of memory (the kernel log shows this), and whether…

Each instruction is a cached layer, and a change invalidates every layer after it. Copying dependency files and installing before copying source means a code change does not…

Liveness asks whether the process is functioning at all; failing it triggers a restart. Readiness asks whether it can serve traffic yet; failing it removes it from the load…

Environment variables at runtime, with secrets from the platform's secret store. Required secrets get no default so a misconfigured deploy fails immediately. Baking configuration…

The code is probably copied before dependencies are installed, so every change invalidates the install layer. Copy the requirements file and install first, then copy the code. Add…

If the application is compromised, root inside the container makes it easier to damage mounted data or exploit a container escape. A normal user limits what an attacker gets.

Fake the model at the boundary with scripted replies, which makes everything around it deterministic: parsing, retries, budgets, tool dispatch, prompt assembly. The quality of the…

Prompt assembly. What actually ends up in the prompt after retrieval, history management and compaction is often not what was intended, and the failure is silent. Asserting that…

A test asserts something that must always hold and is deterministic. An eval measures a rate over a task set and is expected to be imperfect. Mixing them produces flaky tests that…

Replace the model with a scripted fake that returns a planned sequence of tool calls and answers. Then test the loop's behaviour: tools run with the right arguments, results are…

Generating many random inputs and checking a property holds for all of them. It suits parsers and validators of model output, which must never crash on strange input.

Reproduce it reliably first. Then one hypothesis at a time, choosing the cheapest experiment that could falsify it rather than confirm it. Bisect the pipeline to eliminate half…

The rendered prompt, exactly as sent, and the retrieved context. Most of the time the answer was not in what the model was given, which makes it a retrieval problem rather than a…

Measure the rate before changing anything, because a clean run proves almost nothing: at one in twenty, three consecutive passes happen about 85 per cent of the time, and you need…

Find what varies between runs: concurrency, timing, external dependencies, or sampling. For AI systems, non-zero temperature means identical input genuinely produces different…

Check what changed: code, data volume, dependencies, the external APIs it calls. Profile a run on a sample to see where time goes, and look at logs for per-step timings. If a code…

Binary search through commit history to find the one that introduced a problem. You mark a good and a bad commit, and git checks out midpoints for you to test, or runs a test…

Use benchmarks only to shortlist, then run my own eval on real tasks and compare pass rate, cost and p95 latency together. Keep the prompt identical for the first comparison, then…

The identifier should already be in configuration in one place, so the change is one line. Then re-run the eval before switching, because a newer model is not automatically better…

Cost and latency, and often neither is justified. Most workloads are mostly easy, so routing the easy majority to a smaller model and escalating the rest usually holds quality at…

Pick candidates, run the eval set on each, adapt the prompt for the best one, shadow real traffic, then roll out gradually with the old model still available. Start early, because…

So behaviour does not change without a deliberate decision. An alias that moves to a new version can alter output format or quality overnight, and you learn about it from users.

When layout carries meaning: tables, forms, charts, and any scanned or photographed document. It replaces OCR plus layout reconstruction plus extraction with one call, and handles…

Dense tables where values drift between columns, small or low-contrast text where it guesses, counting, and precise spatial relationships. All of them produce confident wrong…

It is cut into a grid of patches, typically 14 or 16 pixels square, and each patch is projected through one linear layer into a vector the same width as a text embedding. The…

Resize before sending, since most documents are readable well below original resolution and image tokens are expensive. Send one page per call rather than batching many, which…

Streaming speech to text, a model with tools for menu, hours and bookings, streaming text to speech, with a latency target of about a second per turn. Handle interruptions,…

Assume mistakes rather than trying to eliminate them. Show sources so users can verify cheaply, make correction a single gesture, set expectations honestly in the copy, and match…

Ordinary software fails visibly. An AI feature fails by producing something confident and well written that happens to be wrong, so the user has no signal. Design has to supply…

By comparing time saved against time spent, not by picking a threshold. The cost has three parts: checking every answer, redoing the ones the check catches, and the incidents…

Only if it is calibrated, meaning that things marked 80 percent are right about 80 percent of the time. An uncalibrated number is worse than none, because it looks like…

Look at behaviour, not just ratings: how much of each draft survives editing, how often users regenerate, accept rates, and whether the time to finish the task dropped. Tie that…

By how often it happens and how costly a mistake is. Frequent, low-cost tasks can be automated. Costly ones get a draft that a person approves. Rare, costly ones are usually best…

It means the output moved in the direction the eval defines as better. Whether users are better off is a separate question that needs a live experiment with a metric chosen…

Behind a flag, internal first, then one percent watching errors and latency, then ten and fifty percent for a real comparison, then full. Instant rollback at every stage, and the…

Output varies per user so the treatment is not identical, quality effects are slow because trust builds and erodes gradually, cost is a genuine variable that belongs in the…

Not yet. Check whether the planned sample size was reached, since stopping at the first significant reading inflates false positives. Check the sample ratio, since a broken split…

Users who see both versions blur the comparison and may change behaviour because of the inconsistency. Per-user assignment gives each person one consistent experience, which is…

A dedicated identity per service with least privilege: exactly the resources it needs and nothing more. Broad wildcard grants are convenient and turn a small compromise into a…

A managed secret store, injected at runtime as environment variables, never baked into an image or committed. Scoped per environment and rotatable without a code change.

Budget alerts from the first day, resource tagging so the bill can be attributed, shutting down idle GPU instances which cost the same as busy ones, and deliberate log retention…

Build the image in CI and push it to a registry. Deploy to a managed container service with its own least-privilege identity, secrets read from the secret manager, a health check,…

Lead with the problem and why it mattered, not the stack. Name one hard decision, the alternative you rejected and why. Include a number that shows you measured. Finish with what…

Pick something genuinely wrong with a real consequence, say how you found out, what you did immediately, and what changed afterwards so it could not recur. Avoid the disguised…

Say so, state what has been ruled out and why, and propose the next thing to try. Interviewers expect candidates to get stuck; they are assessing how you behave when it happens.

Pick a real one where the decision mattered. Say what you believed and why it was reasonable at the time, what evidence showed you were wrong, what you did about it, and what you…

Two sentences of context, then the hardest problem and how you solved it, your decisions and the alternatives you rejected, and the result with numbers. Be ready to go deep on any…

Spend only on extensibility that costs a line rather than a structure: inject the clock or the store, key the state even when there is one key, and keep the decision in one…

Keep one allow(key) entry point and key the state by key and rule. Check every rule before consuming any budget, so a call rejected by the hourly rule does not spend the…

Say so, immediately, with the reason and a time estimate: what the requirement contradicts, what the restructure is, and roughly how long it takes. Then do it. Recognising it…

Two sentences of context, then the decision you owned, the alternative you rejected and the evidence that settled it, then a number, then what went wrong and what you would…

State what happened, what it cost, how you found out, what you did immediately, and the change that means it cannot recur. Keep the tone flat. Minimising and performing remorse…

The one where you made real decisions, not the one with the most recognisable name attached. The questions are about why you chose an approach and what you rejected, and you only…

Give a specific position with the reasons you hold it, then answer the second half honestly, because it is the half most candidates have nothing for. Name the evidence that would…

Ask this of the recruiter before the loop, in writing, for each round, because the rules differ by company and by round within one company. Asking on the day is fine but late;…

You are not meant to. Read what the organisation has published, form an actual response including the parts you disagree with, and work out what would change your mind. Then say…

They exploit locality, since nearby pixels are related, and translation invariance, since a feature means the same thing wherever it appears. Sharing the same filter weights…

Almost never. Start from a pretrained backbone, replace the head for your classes, and fine-tune. It needs far less data and compute and usually outperforms training from nothing…

At large scale with large datasets they match or beat them. Convolutional networks remain cheaper, faster and better with limited data, so for most production problems they are…

Detection locates objects with bounding boxes and labels. Segmentation labels every pixel, either by class in semantic segmentation or separating individual objects in instance…

Only transformations that preserve the label. Flips suit natural photographs but change a 6 into a 9 and make text unreadable, so the choice depends on the domain.

Spatially floor((64 - 3 + 2) / 2) + 1 = 32, so 32x32x64. Parameters: 3 x 3 x 32 x 64 = 18,432 weights plus 64 biases, 18,496.

It learns a shared embedding space for images and text by training on image and caption pairs with a contrastive objective. To classify, embed the image and a text prompt for each…

For each class, rank predictions by confidence, match them to ground truth boxes by IoU above a threshold, and compute the area under the precision-recall curve: that is average…

Long-range dependencies degraded, because information had to survive repeated compression into a fixed-size hidden state and gradients vanished across many steps. And they could…

Gates that let information pass through the cell state largely unchanged, which helped gradients survive across many steps. It addressed long-range dependencies and did nothing…

When inputs stream indefinitely and memory or compute per step must stay fixed, such as on-device audio or sensors. Their cost per step does not grow with history, while a…

Early layers learn general features of the medium rather than the task, so they transfer. Only the later, specialised layers need replacing, which is why a new head plus a small…

The randomly initialised head produces large early gradients, and at a normal learning rate those destroy the pretrained features within a few batches. Train the head first with…

When your data is far from the pretraining distribution, such as medical scans or spectrograms against a model trained on everyday photographs. The test is empirical: if the…

Start from a pretrained model with the matching preprocessing. First a linear probe on frozen features as a baseline. Then train a new head, and if needed unfreeze the last blocks…

You give each layer its own rate, smallest at the input and rising towards the head, usually by a fixed multiplier per block. Early layers hold general features that should barely…

Learning rate. It has more effect than the rest combined. Search it by orders of magnitude first, then refine, and only then move to schedule, batch size and regularisation.

Most hyperparameters barely matter. A grid spends its budget evenly, so it tries only a few distinct values of the one that does. Random search tries a different value of every…

Because tuning decisions leak the score they are based on into the model. With enough trials you find a configuration suited to that specific set. Validation drives decisions;…

One or two runs to confirm the baseline trains, a learning rate range test, then a random or Bayesian search over learning rate and regularisation on log scales with early…

Early in training the weights and the optimiser's running statistics are poorly set, so a full learning rate can cause divergence. Ramping up over the first steps avoids that,…

Hyperparameters, metrics per epoch, the git commit with a dirty flag, the data version as a content hash, the environment and package versions, the seed, and the resulting model…

GPU operations are not always deterministic: parallel reductions can vary in order and some algorithms are selected at runtime. You can force determinism at a speed cost, but…

Look up the model's run record: code commit, data version, config and metrics. If those were not logged, that is the finding and the fix. Then compare that run's evaluation with…

Stop using accuracy, and use precision, recall and PR AUC. Try class weights first, tune the decision threshold from the precision-recall curve rather than leaving it at 0.5, and…

After, and only the training fold. Oversampling before splitting puts copies of the same example in both train and test, which inflates the score. The validation set must keep the…

From the cost of each error type. Compute expected cost across thresholds using the real cost of a miss and of a false positive, and pick the minimum. That turns a metric argument…

Possibly both. With fraud at 0.2%, even a low false positive rate produces far more false alerts than real fraud, so precision at the operating threshold can be very low. I would…

Its scores match observed frequencies: cases scored 0.3 are positive about 30% of the time. You need it whenever the score is used as a probability, such as in expected cost…

It shuffles, so future data ends up in training and past data in validation. Because consecutive points are correlated, the model effectively sees the answer and the score is…

Compute the naive baseline: tomorrow equals today, or equals the same day last week. It is often hard to beat, and many forecasting projects never do without realising.

Shift before rolling, so the window covers only past values. A rolling mean that includes the current row leaks the target and produces a model that cannot work in production.

Start with classical baselines and a gradient-boosted model with lag features, which are fast, strong and interpretable on most business series. A neural network needs to beat…

Error grows with the horizon, and an averaged single number hides how quickly accuracy decays. Evaluate at each horizon you will actually use.

Rolling-origin backtesting that trains only on the past and forecasts the real horizon, a seasonal naive baseline, a scale-free metric such as MASE so products of different volume…

A global model trained across all products with product, calendar and lag features, so short-history products borrow strength from similar ones. Reconcile across the product and…

Two stages: cheap candidate generation narrowing millions to hundreds, then expensive ranking of those. Combine collaborative and content signals, handle cold start with content…

New users have no history and new items have no interactions, so collaborative filtering has nothing to work from. Content features, popularity defaults and exploration cover the…

You only observe interactions with what was shown, under the previous system. A better recommender surfaces different items, which offline scoring counts as wrong. Live…

Separate networks embed users and items, and relevance is their similarity. Item embeddings are precomputed, so retrieval is a fast nearest-neighbour search. It is the same design…

NDCG rewards placing the most relevant items at the top, since users mostly see the first few results. Precision at k treats every position in the top k equally.

A time-based split so the model never sees the future, ranking metrics such as recall at k and NDCG against each user's later interactions, and comparison against a tuned…

Clarify the objective and scale. Then describe multi-source candidate generation, including two-tower retrieval with approximate nearest neighbour search, a ranking model…

Training a small student model to reproduce a large teacher's outputs. Learning from the teacher's full probability distribution carries more information than hard labels, so the…

It zeroes individual weights but leaves the tensor shape unchanged, so the hardware still processes the zeros. Structured pruning removes whole units or layers, which is a genuine…

Measure on your own eval rather than trusting published aggregates, because degradation is uneven across capabilities. Then compare size, latency and quality together, which makes…

Perhaps, after trying cheaper options. First check whether a smaller model with a better prompt is good enough. If not, and the task is narrow and high volume, generate outputs…

At normal temperature the teacher's distribution is very peaked, so it carries little information beyond the top label. Higher temperature spreads it out so the student can learn…

A reward arrives after a long sequence of actions and it is unclear which actions caused it. It is the central difficulty in reinforcement learning and why it needs far more…

Mainly in post-training. A pretrained model predicts plausible text; preference-based methods shape it towards being helpful and harmless, using human judgement as the reward…

Optimising a proxy hard enough that the model finds unintended ways to score well. With human preference data that shows up as confident, agreeable, well-formatted answers being…

Value methods learn how good states or actions are and act greedily with respect to that, suiting discrete actions. Policy methods adjust behaviour directly, handling continuous…

Because the policy determines the data you collect next. A bad update in supervised learning is corrected by the next batch; in RL it changes the distribution and can make…

The data distribution depends on the current policy so nothing is stationary, rewards are sparse and delayed making credit assignment ambiguous, exploration is unsolved, and…

Choosing between the action that looks best so far and trying others that might be better. Exploit too early and you may lock in a worse option; explore too much and you waste…

New items have no measurements, so they are never shown, so they never acquire any. The ranker is a greedy agent locked onto whatever was popular when it was first switched on,…

Q of a state and action moves towards the reward received plus the discounted best Q value of the next state. The gap between that target and the current estimate is the temporal…

You do not, from that alone. On a small environment, compute the exact answer with value iteration and compare the learned policy's true value against the optimum. On a large one,…

The ratio compares the new policy's probability of an action with the old policy's. Multiplied by the advantage, it says how much better the new policy is on that data. Clipping…

Mostly it fixes the scale of the update. Measured on a cart-pole policy, standardising cuts the total variance of the gradient estimator by a factor of thousands, but it shrinks…

Fine-tune on demonstrations, train a reward model on human comparisons, then optimise the model with RL to maximise that reward. The KL penalty keeps the policy close to the…

Whether the reward is the only thing improving. Track KL from the starting model alongside it: reward rising while KL rises and held-out evaluation is flat or falling is…

A fixed forward process gradually adds noise to data. A model learns to reverse one step by predicting the noise added. Generation starts from pure noise and applies the learned…

Training stability. A GAN's adversarial objective suffers mode collapse and oscillation, while diffusion has a simple regression objective that trains reliably and scales…

Running the diffusion process in a compressed latent space rather than on raw pixels, so each denoising step is far cheaper. It is what made high-resolution text-to-image…

Step count, primarily, since latency scales roughly with it. Modern samplers reach good quality in around twenty steps, and distilled models go much lower at a real quality cost.…

Contrasting the model's conditioned and unconditioned predictions and pushing away from the unconditioned one, which strengthens how closely output follows the prompt. Too low…

A LoRA adapter for most cases, since it is cheap and composable. Textual inversion if you only need a new concept token and want to avoid touching the weights. Full fine-tuning…

Sample a clean example, a random step t and Gaussian noise, form x_t as sqrt(alphabar_t) times the clean example plus sqrt(1 minus alphabar_t) times the noise, and train the model…

Too little training, a network too small for the data, a wrong noise schedule so the final step is not near pure noise, a bug in the sampling formula, or the time step not…

Fewer steps with a better sampler, a distilled few-step model, smaller resolution with upscaling, 16-bit weights and compiled models, batching, and caching repeated prompts.…

Reverse-engineering the computation a network performs, identifying circuits of components that implement a specific behaviour. More ambitious than attribution or probing, which…

Models represent more features than they have dimensions by packing them into overlapping directions, so one neuron responds to several unrelated things. That is why reading…

Specific circuits and specific behaviours have been explained, mostly in smaller models. Explaining an arbitrary output from a frontier model is not yet possible. It is real…

Probing shows information is present in a representation, which is correlational and compatible with the model never using it. Patching replaces an activation from one run into…

Models represent more features than they have dimensions by packing them into overlapping directions, so individual neurons respond to several unrelated things. Sparse…

It shows which inputs influenced the output, which is consistent with correct reasoning and with pattern matching. It is evidence about where the model attended rather than about…

The per-position vector that runs from embedding to output, which each attention and MLP layer reads from and adds to. Because contributions are added rather than overwritten, you…

Choose a behaviour and a clean prompt, make a corrupted prompt that changes the answer by a minimal edit, pick a metric such as the logit difference between the two answers, then…

Read activations across the whole range, not only the top examples, test predictions on new text where the label says it should and should not fire, and intervene by adding or…

Replicate a known result to validate the tooling, then pick one narrow behaviour in a small model, define a clean and corrupted setup, and run patching with controls against…

Build concrete tasks that require it and try hard to elicit the behaviour, with scaffolding and multiple attempts, then report the elicitation effort alongside the result. A…

Because failure to elicit is not absence of capability. Better prompting, tool access or more attempts may succeed where your evaluation did not, so the claim is always bounded by…

Deliberately trying to make a model behave badly. Manual work finds creative attacks automated methods miss; automated work scales. Findings feed back into training and…

Report the elicitation effort as part of the finding: how many attempts, which prompting strategies, whether tools and scaffolding were provided. Without that, a failure only says…

Capability is a property of the whole system. A model that fails alone may succeed with tool access, retrieval and retries, so bare-model evaluation systematically understates…

Notify the developer, allow time to mitigate, then publish the class of problem rather than a working exploit. The tension between helping defenders and helping attackers is real…

Only that it did not under the conditions tested. The strength of the conclusion depends on elicitation effort: prompting, tools, multiple attempts, fine-tuning. Report every…

Define specific failures for this product, run an automated loop with an attacker model, the real application in a test environment and a judge or code check, push for varied…

Intervene on something that should affect the answer, such as a biasing hint, and check whether the answer changes and whether the reasoning acknowledges the cause. Answers that…

Loss falls as a power law in compute, data and parameters, consistently across many orders of magnitude. They let a lab predict the loss a compute budget will reach before…

Allocating a fixed compute budget between model size and training tokens to minimise loss. The key finding was that they should grow together, and that earlier large models were…

Compute-optimal minimises training cost. If inference volume is large, inference dominates total cost, so a smaller model trained well past that point is cheaper overall despite…

Disputed. Sudden capability jumps are widely reported, and a serious counter-argument holds that they are partly an artefact of discontinuous metrics, where smooth improvement…

Start from compute-optimal scaling, roughly 20 tokens per parameter, which gives the lowest loss for that training budget. Then adjust for inference: if the model will serve heavy…

When specific abilities appear, how a model performs on a particular downstream task, and what happens with a different architecture or data mix than the one the law was fitted on.

The same items, the same prompt and number of examples, the same sampling settings, and the same rule for extracting and grading an answer. The last one is the one that gets…

Contamination, since public benchmarks end up in training data so a high score can be memorisation. Saturation, where remaining differences above about 90 percent are noise and…

Shortlist with public benchmarks, then build a private set of real tasks, including production failures, and compare pass rate, cost and latency. A private set cannot be…

It earns a place on the shortlist. Whether it is right for your task, at your cost and latency requirements, is a separate question that only your own eval answers.

Not on that basis. Two points may be within noise, the benchmark may be contaminated or saturated, and it may not measure my task. I would run it on my own eval set with the same…

Items unlikely to be in training data, a setup identical across models, several prompts and seeds with the spread reported, confidence intervals, and a task that actually measures…