OpenAI has released openai/circuit-sparsity — a model on Hugging Face and an accompanying toolkit on GitHub — packaging the models and circuits from its research paper “Weight-sparse transformers have interpretable circuits.” The release is aimed at interpretability researchers: it provides working examples of language models whose internal computations can actually be read and understood, along with the tools to study them.
In plain language, the problem this work attacks is that ordinary neural networks are tangled: every neuron connects to thousands of others, so nobody can say which specific connections implement a given behavior. OpenAI’s approach is to train models that are forced to be mostly empty — and in those models, the wiring behind specific behaviors becomes small enough to draw on a whiteboard.
What is a weight-sparse transformer?
The released models are GPT-2-style decoder-only transformers trained exclusively on Python code. The sparsity is not added after training — it is enforced during optimization. After each AdamW step, the training loop keeps only the largest-magnitude entries in each weight matrix and bias (including the token embeddings) and zeros the rest, with every matrix holding the same fraction of non-zero elements. In the sparsest models, roughly one in a thousand weights is non-zero, and mild activation sparsity is enforced as well.
Models start dense, and the allowed non-zero budget tightens gradually toward the target during training. This design lets the team hold the number of non-zero parameters constant while varying sparsity and model size, and then measure the trade-off between capability and interpretability. The headline result: at matched pretraining loss, the circuits recovered from sparse models are roughly 16 times smaller than those recovered from dense baselines.
So what is a sparse circuit?
The central object of the research is the sparse circuit, defined precisely. Each node is a single neuron, attention channel, residual read channel or residual write channel. An edge is a single non-zero weight-matrix entry connecting two nodes. Circuit size is measured by the geometric mean number of edges across tasks.
To test the models, the team constructed 20 simple Python next-token binary tasks, each forcing the model to choose between two completions that differ by one token. Examples include single_double_quote (predict whether a string closes with a single or double quote), bracket_counting (decide between ) and )) based on nesting depth), and set_or_string (track whether a variable was initialized as a set or a string).
For each task, the model is pruned to the smallest circuit that still achieves a target loss of 0.15 on that task distribution. Pruning operates at the node level: deleted nodes are mean-ablated, with activations frozen at their pretraining-distribution average, and a learned binary mask per node is optimized with a straight-through-style surrogate to balance task loss against circuit size.
Example circuits: quote closing and bracket counting
The simplest example is the single_double_quote circuit, where the model must emit the correct closing quote type. The pruned circuit contains just 12 nodes and 9 edges, organized in two stages. In an early MLP layer, two neurons specialize: a quote detector that activates on both " and ', and a quote-type classifier that is positive on " and negative on '. A later attention head uses the detector channel as its key and the classifier channel as its value; the final token carries a constant positive query, so attention copies the correct quote type into the last position and the model closes the string correctly.
bracket_counting yields a slightly larger circuit with an even clearer algorithm. The embedding of ( writes into residual channels that act as a bracket detector; a value channel in a layer-2 attention head averages that detector across the context, effectively computing nesting depth and storing it in a residual channel; a later attention head thresholds the depth and activates “close nested list” channels only when the list is nested, leading the model to output )).
A third circuit, for set_or_string_fixedvarname, shows type tracking: one attention head copies the embedding of the initializing token (set() or "") alongside the variable name, and a later head uses that stored embedding as query and key to retrieve the type when choosing between .add and +=.
Bridges connecting sparse models to dense models
The release also introduces bridges, which connect a sparse model to an already-trained dense model. Each bridge is an encoder–decoder pair mapping dense activations to sparse activations and back, once per sublayer — the encoder is a linear map with an AbsTopK activation, the decoder is linear. A training loss encourages the hybrid sparse–dense forward pass to match the original dense model. This lets researchers perturb an interpretable sparse feature (such as the quote-type classifier channel) and transfer that perturbation into the dense model, changing its behavior in a controlled way — a step toward relating clean, readable circuits to the messy internals of production-scale models.
What exactly was released?
The openai/circuit-sparsity model on Hugging Face is a 0.4-billion-parameter checkpoint (tagged custom_code, corresponding to csp_yolo2 in the paper) used for the qualitative bracket-counting and variable-binding results. The openai/circuit_sparsity codebase on GitHub ships under Apache 2.0 and includes model checkpoints, task definitions and a circuit-visualization UI.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
if __name__ == "__main__":
PROMPT = "def square_sum(xs):n return sum(x * x for x in xs)nnsquare_sum((1, 2, 3))n"
tok = AutoTokenizer.from_pretrained("openai/circuit-sparsity", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
"openai/circuit-sparsity",
trust_remote_code=True,
torch_dtype="auto",
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
inputs = tok(PROMPT, return_tensors="pt", add_special_tokens=False)("input_ids").to(
model.device
)
with torch.no_grad():
out = model.generate(
inputs,
max_new_tokens=64,
do_sample=True,
temperature=0.8,
top_p=0.95,
return_dict_in_generate=False,
)
print(tok.decode(out(0), skip_special_tokens=True))
``` :contentReference(oaicite:14){index=14}
Key takeaways
Sparsity is imposed during training, not recovered afterward: most weights are zero, so each neuron has only a few connections, and interpretability is a property of the model itself rather than of a post-hoc analysis. The recovered circuits are small and concrete — often tens of nodes and a handful of edges for the 20 Python tasks — and for tasks like quote closing they constitute fully understood algorithms. The bridge mechanism connects this readable world to standard dense models, which is where the approach could eventually matter for real systems.
Limitations and what to watch
The caveats are significant and acknowledged in the work itself. These are small models (0.4B parameters) trained on a narrow domain — Python code — and evaluated on deliberately simple binary tasks; nothing here demonstrates that frontier-scale models can be made interpretable this way. Weight-sparse training is also computationally inefficient relative to dense training at matched capability, so the technique currently buys understanding at the cost of performance. Whether bridges can carry meaningful interpretability from toy sparse models into production systems like the frontier models now being benchmarked on real work remains an open research question — and the one most worth watching.
