Transformer Architecture Explained for Developers

Transformer architecture explains how a model turns a sequence of tokens into context-aware representations and, for generative models, predicts the next token. The useful way to understand it is to follow one token through embeddings, attention, a feed-forward network, and the output layer.

Why Transformers changed sequence processing

Recurrent neural networks process a sequence step by step, so each hidden state depends on the one before it. The 2017 Attention Is All You Need paper removed recurrence from its encoder-decoder model and used attention to connect positions directly.

Training can calculate attention for many positions in parallel because the input tokens are available together, although generation remains sequential for an autoregressive decoder because token N+1 cannot be produced until token N exists.

Attention also has a cost. Standard self-attention creates a score for every pair of positions, so the score matrix grows quadratically with sequence length.

Tokens need embeddings and position information

A tokenizer converts text or code into token IDs, then an embedding table maps each ID to a vector whose learned coordinates carry features the model can use.

Attention alone does not know whether one token appeared before another, so the original Transformer adds positional encodings to token embeddings and later designs may use learned positions or rotary position embeddings.

Order changes meaning in code. A model needs position information to distinguish arguments in function(a, b) from the reversed call function(b, a), even when both sequences contain the same tokens.

Self-attention builds context with query, key, and value vectors

Each token representation is projected into three vectors. The query describes what the token is looking for, the key describes what it can be matched on, and the value carries the information that will be mixed into the result.

Scaled dot-product attention turns query-key dot products into similarity scores, divides by the square root of the key dimension to keep large vectors from pushing softmax into extreme values, and converts each row into weights that sum to one.

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V

The output for each token is a weighted sum of value vectors. A high weight means that the corresponding token contributes more to the new context vector, but an attention weight is not a complete explanation of a model prediction.

A small self-attention calculation in NumPy

The following example uses the same three-by-four matrix for queries, keys, and values so you can inspect the calculation without trained projection matrices. Each row represents one token embedding.

import numpy as np

np.set_printoptions(precision=3, suppress=True)

tokens = ["function", "returns", "value"]
embeddings = np.array([
    [1.0, 0.0, 1.0, 0.0],
    [0.0, 1.0, 1.0, 0.0],
    [1.0, 1.0, 0.0, 1.0],
])

query = embeddings
key = embeddings
value = embeddings

scores = query @ key.T / np.sqrt(key.shape[-1])
scores -= scores.max(axis=-1, keepdims=True)
weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)
context = weights @ value

print("Tokens:", tokens)
print("\nAttention weights (rows query, columns key):")
print(weights)
print("\nEach row sums to:", weights.sum(axis=-1))
print("\nContext vectors:")
print(context)

Running the script with Python 3.13.5 and NumPy 2.5.1 produces a three-by-three attention matrix whose rows sum to one, then multiplying those weights by the value matrix yields one context vector per token.

The diagonal weights are larger because each sample embedding is most similar to itself. A trained attention layer learns separate projection matrices, so its query, key, and value vectors are not usually identical.

Masks control which tokens can communicate

An encoder can usually let every token attend to every other input token. A decoder used for next-token prediction needs a causal mask that blocks each position from reading tokens to its right.

Padding masks stop placeholder positions in a padded batch from contributing to attention, and framework APIs may use opposite Boolean conventions for causal and padding masks.

PyTorch documents that scaled_dot_product_attention treats True as allowed participation, but MultiheadAttention key_padding_mask treats True as masked out. That difference can produce plausible tensor shapes with incorrect outputs, so inspect the API contract before reusing a mask.

Multi-head attention learns several views of the sequence

One attention head produces one set of weights. Multi-head attention splits the representation across several heads, runs attention for each head, concatenates the results, and applies an output projection.

Different heads can learn different relationships, but assigning a fixed human label such as syntax or variable flow to a head is often too strong. The model learns whichever features reduce its training loss.

More heads do not remove the quadratic score matrix. If long-context latency is your concern, the practical bottlenecks are covered in the LLM latency optimization guide.

A Transformer block does more than attention

Attention moves information between token positions. A position-wise feed-forward network then transforms each token representation independently, usually expanding to a larger hidden dimension before projecting back.

Residual connections preserve the block input by adding it to a sublayer output. Layer normalization keeps activations on a manageable scale, and dropout can regularize training.

Stacking blocks lets later layers operate on representations assembled by earlier layers. Fine-tuning methods such as low-rank adaptation (LoRA) modify selected weight updates without retraining every parameter.

Encoder-only, decoder-only, and encoder-decoder models

Transformer architecture is a family of layouts rather than one fixed model. The layout determines what information a token may use and what task the network is trained to perform.

• Encoder-only models build bidirectional representations, which suit classification, retrieval, and token labeling.
• Decoder-only models use causal self-attention and predict one token after another, which suits text and code generation.
• Encoder-decoder models encode a source sequence and generate a target sequence through decoder self-attention plus cross-attention to encoder outputs.

PyTorch describes nn.Transformer as a reference implementation of the original encoder-decoder architecture with limited features compared with newer designs. Treat it as a learning tool before choosing optimized building blocks for production.

How Transformer architecture maps to developer tools

A code assistant still depends on context selection outside the neural network. Retrieval-augmented generation (RAG) can place relevant files or documentation into the prompt, then attention relates those supplied tokens during inference.

Attention cannot read a repository that was never included in the model input, so the RAG guide for codebases covers that retrieval step and the code embeddings guide explains how semantic search finds candidate chunks.

When a model generates code, the decoder predicts a distribution over the next token from its final hidden representation. Sampling or decoding logic chooses a token, appends it to the sequence, and repeats the forward pass with the longer context.

Questions about Transformer architecture

A Transformer turns token IDs into embeddings, adds position information, mixes context through attention, transforms each position through feed-forward layers, and maps the result to a task output. Trace those tensors and masks before treating a model name as an explanation of how it works.

Use the NumPy example to verify the attention equation, then add learned projection matrices or a causal mask. That next step exposes the difference between a classroom calculation and the layer used for generation.