RAG or fine-tuning
The model does not know something it needs to know. Do you retrieve the knowledge at request time, or train it into the weights?
Retrieval (RAG) · Fine-tuning · contested
The decisions AI engineers actually argue about
Each of these is a real decision with real costs on both sides. Written to help you choose rather than to sell an answer, and where the field genuinely disagrees, that is said rather than resolved.
LoRA or a full fine-tune
You have decided to fine-tune. Do you update every weight, or train small adapter matrices and leave the base model frozen?
LoRA and other adapters · Full fine-tune
Prompting, RAG, or fine-tuning
The output is not good enough. Which lever do you pull, and in what order?
Better prompting · Retrieval · Fine-tuning
An agent or a fixed pipeline
Should the model decide what to do next in a loop, or should you write the steps and call the model at fixed points?
Fixed pipeline · Agent loop
Open weights or a hosted API
Do you run a model whose weights you have, or call someone else's endpoint?
Hosted API · Open weights, self-run · contested
A bigger model or a better prompt
Quality is short of what you need. Do you move up a model tier, or spend the time on the prompt you already have?
Move up a tier · Fix the prompt
Self-hosting or a provider
You have chosen an open-weights model. Do you run the serving stack, or rent it from an inference provider?
Inference provider · Run it yourself
A vector database or a library plus your own storage
You need nearest-neighbour search over embeddings. Do you run a dedicated vector database, or add vectors to the store you already have?
Your existing database · A dedicated vector database · A library in your own process
Dense retrieval or keyword search
Should retrieval match on embeddings, on BM25 keyword scoring, or on both?
Dense (embedding) retrieval · Keyword search (BM25) · Hybrid, then rerank
A code grader or an LLM judge
How should an eval decide whether an output is right?
Code grader · LLM judge · contested
Gradient boosting or a neural network on tabular data
You have a table of rows and columns and a target. Which model family?
Gradient-boosted trees · Neural network
Quantization or a smaller model
The model does not fit, or costs too much to serve. Do you compress the one you have, or use a smaller one?
Quantize the larger model · Use a smaller model
Batch or streaming inference
Should predictions be computed ahead of time on a schedule, or on demand when the request arrives?
Batch, precomputed · Online and streamed
Fine-tuning or few-shot prompting
You have a few hundred examples of the behaviour you want. Put them in the prompt, or train on them?
Few-shot prompting · Fine-tuning
Building evals first or shipping first
Do you build a measurement harness before the feature, or ship something and measure what comes back?
Evals first · Ship first, instrument heavily · contested