Decoder-only model

Also: decoder-only transformer, causal language model

A transformer built from a single causally-masked stack that continues its input. Nearly every chat and completion model is one.

What people get wrong

The name survives from the encoder-decoder translation models it came from. There is no encoder anywhere in the model, and no decoding in the cryptographic sense.

A definition is not an understanding. This is taught properly in From attention to a full language model, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE