Training hyperparameters worth knowing
Starting values that are right often enough to be defaults, and what each one interacts with when you change it.
Learning rate
| Pretraining a transformer from scratch | 3e-4 below 1B parameters, falling to around 1e-4 at 7B and below that at larger scale | Peak learning rate falls roughly as models get larger. GPT-3 at 175B used 0.6e-4. If you have no other information, 3e-4 with AdamW is the single most common starting point in the field. |
|---|---|---|
| Full fine-tune of a pretrained model | 1e-5 to 5e-5 | An order of magnitude below pretraining. Too high and you get catastrophic forgetting that looks like the base model getting worse at everything else. |
| LoRA | 1e-4 to 3e-4 | Roughly ten times the full fine-tune rate, because far fewer parameters are moving and they start at zero. |
| A new head on a frozen backbone | 1e-3 | Nothing pretrained is at risk, so the rate can be high. |
| Finding one empirically | the largest rate that does not diverge, divided by about 3 | A short range test — sweep the rate upward over a few hundred steps and watch where loss turns up — costs minutes and beats guessing. |
Optimiser
| AdamW betas for language models | 0.9 and 0.95 | The framework default for beta2 is 0.999. Large-scale language model training generally uses 0.95, which reacts faster to changes in gradient scale and is more stable at large batch. |
|---|---|---|
| Adam epsilon | 1e-8 | Raise to 1e-6 if you see instability in low precision; it is a numerical guard, not a tuning knob. |
| Weight decay | 0.1 for pretraining · 0.0 to 0.01 for fine-tuning | Exclude biases and normalisation parameters. Decaying them is a common bug that costs a little quality quietly. |
| Gradient clipping | global norm 1.0 | Near universal for transformers. If clipping fires on most steps your learning rate is too high; if it never fires you can stop worrying about it. |
Schedule and batch
| Warmup | 1-2% of total steps, or a flat 2,000 steps | Adam's second-moment estimate is unreliable early on, and warmup stops that turning into a large step into nonsense. Pre-norm architectures tolerate no warmup; post-norm usually will not train without it. |
|---|---|---|
| Decay | cosine to about 10% of the peak rate | Cosine requires knowing the total step count in advance. Where you do not, a constant rate with a linear cooldown at the end gets close and lets you stop anywhere. |
| Batch size and learning rate | scale the rate with the square root of batch size for Adam; linearly for SGD | Both are approximations that hold over perhaps an 8× range, not indefinitely. When in doubt, re-tune the rate after any large batch change. |
| Critical batch size | past it, larger batches buy wall-clock time but not sample efficiency | It grows during training as gradients get less noisy, which is why large runs ramp batch size up rather than fixing it. |
| Fine-tuning epochs | 1-3 | Beyond three epochs on a small instruction set, training loss keeps falling and held-out quality usually does not. Watch the eval, not the loss. |
| Dropout | 0.0 for large-scale pretraining · 0.05-0.1 for fine-tuning on a small set | With far more data than parameters seen once, dropout mostly costs you. With a few thousand examples it is doing something. |
LoRA
| Rank | 8-16 for style and format · 32-64 for new capability · 128+ before considering a full tune | Rank is the budget for how much behaviour you can add. Most reports of LoRA underperforming are rank 8 attempting a rank 64 job. |
|---|---|---|
| Alpha | commonly 2 × rank | The update is scaled by alpha/rank, so setting alpha to twice the rank keeps the effective scale constant as you change rank — which means the learning rate does not need re-tuning with it. |
| Which modules | all linear projections, not just query and value | The original paper adapted q and v to save memory. Adapting the attention output and the MLP projections too is consistently better and costs little at these ranks. |
| LoRA dropout | 0.05-0.1 | On the adapter input. Matters on small datasets and is roughly irrelevant on large ones. |
Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.
Where these numbers come from