Encoder-only model

Also: BERT-style model, bidirectional model

A transformer with no causal mask, so every position sees the whole input at once. Good for classification, embedding and reranking, and incapable of generating text left to right.

A definition is not an understanding. This is taught properly in From attention to a full language model, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE