Almost every transformer people meet today is a decoder-only stack: GPT and everything downstream of it. The paper that started all of it, “Attention Is All You Need,” describes something larger: an encoder-decoder model built for machine translation. The decoder-only architecture is what you get when you take that model, throw away the encoder and the cross-attention, and causally mask the self-attention that remains. I built it from scratch in PyTorch, no nn.Transformer, as exactly that reduction, because the fastest way to understand what is essential for autoregressive generation is to remove everything that is not.

The short version up front: the spine of the 2017 model survives intact. Attention, the residual stream, and one specific scaling factor are still the core. What you drop is the entire reading half of the network, and what you change since is repair work around the edges for training stability and serving cost. Here is the build, and the deltas.

The shape

The full Transformer is two stacks. An encoder reads the source sequence and produces context vectors, and a decoder generates the target one token at a time while attending both to its own past output and to the encoder. A decoder-only model keeps only the second stack, and only part of it.

Concretely, a full decoder block has three sublayers: masked self-attention, cross-attention to the encoder, and a feed-forward block. Remove the encoder and there is nothing for cross-attention to attend to, so it goes too. The block collapses to two sublayers:

class DecoderLayer(nn.Module):
    def forward(self, x, mask):
        x = self.sub[0](x, lambda x: self.self_attn(x, x, x, mask))  # causal self-attention
        return self.sub[1](x, self.ff)                               # feed-forward

That is the whole structural change. A decoder-only block is just a transformer block whose self-attention is not allowed to look forward. Everything else in this post is either a piece carried over unchanged from the paper or a detail that is easy to get subtly wrong.

Scaled dot-product attention, and the square root

The center of the model is four lines, identical to the original:

scores = (query @ key.transpose(-2, -1)) / math.sqrt(d_k)
scores = scores.masked_fill(mask == 0, -1e9)
attn = scores.softmax(dim=-1)
return attn @ value, attn

The one part of this people skip past is the division by the square root of d_k. It is not cosmetic. The dot product of two vectors with unit-variance components and dimension d_k has variance d_k. Without the scaling, as you widen the head dimension the raw scores grow, the softmax saturates, it puts almost all of its mass on a single key, and the gradient through it collapses toward zero. The square root puts the variance back near one so the softmax stays in a regime where it can learn. This factor has survived every architecture change since 2017, and it is invisible until you write attention out yourself.

Causal masking is now the whole point

In the full model the mask was one detail among several. In a decoder-only model it is the defining feature, because it is the only thing distinguishing this block from an encoder block. The rule is that position i may attend to positions up to and including i, never beyond. That is a lower-triangular mask, combined by a logical AND with a padding mask when you batch ragged sequences:

def make_mask(seq, pad_idx):
    pad = (seq != pad_idx).unsqueeze(1)              # (B, 1, seq)
    causal = subsequent_mask(seq.size(1))            # (1, seq, seq), lower-triangular
    return pad & causal                              # (B, seq, seq)

Get the broadcasting wrong by one axis and the model still runs, still trains, and silently lets position i see token i+1. There is no error, just a loss that looks suspiciously good because the model is reading the answer it is supposed to predict. The only defense is understanding exactly what each dimension means. This is the single bug that the from-scratch exercise is most worth doing to internalize.

The details carried over from the paper

Three more choices come straight from 2017 and are easy to drop. Embeddings are multiplied by the square root of the model dimension before positions are added, which keeps their magnitude comparable to the position signal. Positions themselves are fixed sinusoids, a bank of sines and cosines at geometrically spaced frequencies, rather than learned parameters. And the LayerNorm sits after the residual add: LayerNorm(x + sublayer(x)), which the paper calls post-norm.

Hold onto that last one, because post-norm is the decision that aged the worst, and it is the first thing a modern decoder-only model changes.

Checking it actually works

A from-scratch model that runs is not a from-scratch model that is correct. The full Transformer has a clean smoke test, the copy task, where the encoder reads a sequence and the decoder reproduces it. A decoder-only model has no encoder and no source, so that test does not apply directly. The right analog is the in-context copy task: each training example is a random sequence, a separator token, then the same sequence again. The model is trained on next-token prediction over only the second copy, so to do well it has to look back past the separator and reproduce the first occurrence.

This is a real capability test, not a trivial one. Reproducing an earlier span requires the model to form what the interpretability literature calls an induction head, and that provably needs at least two attention layers, one to find the previous occurrence and one to copy what followed it. A two-layer model learns it in a few hundred steps. Loss falls from about 6 down to roughly zero, and greedy generation reproduces held-out prompts token for token. If your causal mask is wrong, this is again where it surfaces, because a model that can see the future copy will drive its loss to zero through a route that has nothing to do with attention.

What changed since 2017, and why decoder-only won

With the model built, the interesting part is holding it next to one from this year and listing what moved.

Normalization moved first. Post-norm, the original choice, is hard to train at depth. Its gradients are badly enough behaved that the paper needed a learning-rate warmup to keep early training from diverging. Pre-norm, x + sublayer(LayerNorm(x)), puts the normalization inside the residual branch instead of across it, which keeps a clean gradient path straight down the stack. It trains deeper and more stably, often without warmup, and it is now the default in essentially every decoder-only model. This is the one place the original is simply worse, and it is a one-line change.

Positions moved twice. Fixed sinusoids gave way to learned absolute positions, and learned absolute gave way to rotary embeddings, which encode position by rotating the query and key vectors so attention scores depend on relative offset. Rotary extrapolates better to longer contexts and is what most current open models use.

The reason the decoder-only shape won at all is partly simplicity and how cleanly it scales, but a large part is serving. An autoregressive decoder generates one token per step, and because of causal masking the representation of every earlier token is fixed once computed. So you cache the keys and values you have already produced and never recompute them. That KV cache turns generation from quadratic recompute into a linear append, and it makes a single causal stack the natural, efficient unit of generation in a way an encoder-decoder model is not. The architecture that is cheapest to serve is the one that took over.

That same serving pressure produced the next change to attention itself. The original gives every head its own keys and values. Multi-query and grouped-query attention share keys and values across heads, which shrinks the KV cache by a large factor and is the main reason long-context generation is affordable. That change exists almost entirely for inference, not for quality.

Smaller deltas round it out. The ReLU in the feed-forward block became GELU and then gated variants like SwiGLU, usually with the inner dimension trimmed to hold the parameter count roughly fixed despite the extra gate. None of these touch the skeleton.

Takeaway

A decoder-only transformer is the 2017 model reduced to exactly what autoregressive generation needs, plus a handful of patches for stability and serving. The encoder and cross-attention are gone, the self-attention is causally masked, and the spine that remains, scaled dot-product attention over a residual stream with normalization and added positions, is the same one the paper drew eight years ago. The square root that keeps the softmax sane is unchanged.

That is the argument for building it by hand as a reduction rather than from a template. Once you have removed the encoder yourself, placed the causal mask, and watched the loss collapse on a task that depends on it, every modern architecture diagram stops looking like a new thing and starts looking like this block with the LayerNorm moved and the KV cache shrunk. You can read which part is load-bearing and which part is someone optimizing the serving bill.

https://ieeexplore.ieee.org/abstract/document/11235449

Abstract: Support Vector Machine approaches provide efficient classification in data sets with high dimensionality. In prior studies we have presented classification results in a high dimensional a metric space constructed from data dependency graphs. These features were demonstrated to be tied to ground truth class labels, and therefore correlated to operational semantics. The dimensionality of the metric space is derived from the quantity of isomorphically unique data dependency graphs within the data set. While successive refinement can reduce the dimensionality and search space, dimensionality at the most coarse-grained level remains very high. We present results from Support Vector Machine classifiers and show high accuracy with low false positive rates using features correlated to operational semantics. We train classifiers on the Kaggle 2015 Microsoft Malware data set using a linear SVM classifier (One-vs-Rest and One-vs-One), SVM classifier with RBF kernel, SVM classifier with polynomial kernel, and an SVM classifier with custom kernel based on computing the pairwise Hamming distance. This study obtains a total accuracy of over 93% for a multi-class classification problem using a Linear SVM (One-vs-Rest) classifier. The Linear SVM classifier has the lowest false positive rate and high precision, with 6 of 9 classes having precision above 90%, high F1 scores, and a ROC AUC of 0.98. Non-linear SVM kernels show a decrease in total accuracy which indicates linearity in the associated feature space. The classifier was trained on features correlated with binary operational semantics which are demonstrated to be tied to ground truth class labels.

https://ieeexplore.ieee.org/document/10670673

Abstract: “Explainability of classification results is dependent upon the features used in classification. Data dependency graph features representing data movement are directly correlated with operational semantics, and subject to fine grained analysis. This study obtains accurate classification from the use of features tied to structure and semantics. By training an accurate model using labeled data, this feature representation of semantics is shown to be correlated with ground truth labels. This was performed using non-parametric learning with a novel feature representation on a large scale dataset, the Kaggle 2015 Malware dataset. The features used enable fine grained analysis, increase in resolution, and explainable inferences. This allows for the body of the term frequency distribution to be further analyzed and to provide an increase in feature resolution over term frequency features. This method obtains high accuracy from analysis of a single instruction, a method that can be repeated for additional instructions to obtain further increases in accuracy. This study evaluates the hypothesis that the semantic representation and analysis of structure are able to make accurate predictions that are also correlated to ground truth labels. Additionally, similarity in the metric space can be calculated directly without prior training. Our results provide evidence that data dependency graphs accurately capture both semantic and structural information for increased explainability in classification results.”

@INPROCEEDINGS{10670673,
  author={Musgrave, John and Ralescu, Anca},
  booktitle={NAECON 2024 - IEEE National Aerospace and Electronics Conference}, 
  title={kNN Classification of Malware Data Dependency Graph Features}, 
  year={2024},
  volume={},
  number={},
  pages={206-213},
  keywords={Training;Measurement;Accuracy;Semantics;Aerospace electronics;Feature extraction;Malware;machine learning;feature extraction;malware analysis},
  doi={10.1109/NAECON61878.2024.10670673}
}

Abstract: “In this study, we present a novel representation for binary programs which captures semantic similarity and structural properties. This representation enables the search and retrieval of binary executable programs based on their similarity of behavioral properties. The proposed representation is composed in a bottom-up approach: we begin by extracting data dependency graphs (DDG), which are representative of both program structure and operational semantics. We then encode each program as a set of graph hashes representing isomorphic uniqueness, a method we have labeled DDG Fingerprinting. We present experimental results of search using k-Nearest Neighbors in a metric space constructed from a set of binary executables. Searches in the dataset are based on the operational semantics of specific malware examples.”

http://dx.doi.org/10.54364/aaiml.2024.41117

@article{musgrave2024search, title={Search and Retrieval in Semantic-Structural Representations of Novel Malware}, author={Musgrave, John and Campan, Alina and Messay-Kebede, Temesguen and Kapp, David and Wang, Boyang}, journal={Advances in Artificial Intelligence and Machine Learning}, volume={4}, number={1}, pages={117}, year={2024} }

Empirical Network Structure of Malicious Programs

Abstract: “A modern binary executable is a composition of various types of networks. Control flow graphs are a commonly used representation of an executable program used for classification tasks. Control flow and term frequency representations are widely adopted, but provide only a partial view of program semantics and present challenges to increases in resolution. By performing a quantitative analysis of program networks, we enable the identification of patterns within these features that are correlated to structure. This allows for increases in feature resolution and pattern recognition in classification tasks. These are necessary steps in order to obtain greater explainability in classification results. We demonstrate the presence of Scale-Free properties of network structure for program data dependency and control flow graphs, and show that data dependency graphs also have Small-World structural properties. We show that program data dependency graphs have a degree correlation that is structurally disassortative, and that control flow graphs have a neutral degree assortativity, indicating the use of random graphs to model the structural properties of program control flow graphs would show increased accuracy. An increase in feature resolution allows for the structural properties of program classes to be analyzed for patterns as well as their component parts. By providing an increase in feature resolution within labeled datasets of executable programs we provide a quantitative basis to interpret the results of classifiers trained on CFG graph features. By capturing a complete picture of program networks we can enable future work in mapping a program’s operational semantics to its structure.“

http://dx.doi.org/10.54364/aaiml.2024.41112

@article{musgrave2024empirical, title={Empirical Network Structure of Malicious Programs}, author={Musgrave, John and Campan, Alina and Messay-Kebede, Temesguen and Kapp, David and Wang, Boyang}, journal={Advances in Artificial Intelligence and Machine Learning}, volume={4}, number={1}, pages={112}, year={2024} }