Transformer Basics

Most tokenizers prepend a BOS token.

• ↳ fixed landmark or “attention sink”
• space prefix gives you the wrong token

Two sublayers at every layer

1. MLP → move info
2. Attention → transform info

$[x_1,x_2,x_3]\cdot W_E=\left[\begin{array}{c}{x}_{10}\\ x_{11}\\ x_{12}\\ ⋮\end{array}\right]\}\text{ embedded token}$

All attention heads write to same residual stream.

$r^{l+}=r^l+{\text{Attn}}^l(r^l)$

MLP ⇒

$r^{l+1}=r^{l+}+{\text{MLP}}^l(r^{l+})$

↳ NO cross token communication

$\text{Residual stream}=r^L=r^0+{\sum }_{l=0}^{L-1}({\text{Attn}}^l+M$

$\text{Logits}=r^L\cdot W_U$

$p({\text{token}}_i)=\frac{e^{{\text{logit}}_i}}{{\sum }_j{e}^{{\text{logit}}_j}}$

Cross entropy loss

$\mathcal{L}=-\frac{1}{n}{\sum }_{i=0}^{n-1}\log (P(t_{i+1}\mid t_0,\dots ,t_i))$

Notation Reference

symbol	meaning
$x$	token embedding / a $d_{model}$-dim vector
$W$	weight matrix
$r$	residual stream
$W_Q,W_K,W_V,W_O$	query, key, value, output matrices
$d_{model}$	residual stream dimension
$H$	number of attention heads