# Stuck in a Loop

Runaway credit ratcheting in attention matrices

Source: https://hyperstition.cc/stuck-in-a-loop

September 4, 2026

Authors: Mason Eyler, AJ, Nahom Seyoum, Max Misterka, Deven Pietrzak, Adit Srivastava

## TL;DR

- Current attention weights act as a strong filter during backpropagation, creating a positive feedback loop in which certain keys can gain exponentially more attention.

- This feedback can lock in early differences in attention.

- This leaves some semantically relevant keys weakly attended to; we construct an oracle to identify them.

- Different heads within a layer have mostly compatible attention patterns, suggesting less specialized coordination between routing and value representations than we might naively expect.

- We formulate a method to democratize the backward pass using a natural-gradient approach.

- We find that the new gradient broadens attention patterns and gives previously weakly attended keys more weight, but worsens validation loss, suggesting that the dynamics concentrating attention on a few winners may also support learning.

## One Attention Head's Forward Pass

Fix an attention head within a transformer, and let $a_{ij}$ be token $j$'s attention score during token $i$'s forward pass.

$\def\arraystretch{0.88}\begin{align*}
a_{ij} &\ge 0, \quad \text{and}\tag{1}
\\[0.176100em]
\sum_{j \le i} a_{ij} &= 1.\tag{2}
\end{align*}$

We can express the output of the attention mechanism as

$y_i = \sum_{j\le i} a_{ij} v_j.\tag{3}$

Let $\mathcal L$ be the loss and let $\delta_i$ be the gradient at the head output:

$\delta_i:=\nabla_{y_i}\mathcal L.\tag{4}$

Backpropagation through the head gives the following formulas for the gradients:

$\def\arraystretch{0.88}\begin{align*}
(\nabla_{v_j}\mathcal L)_i &= a_{ij}\delta_i,\tag{5}
\\[0.206600em]
\nabla_{q_i}\mathcal L &= \frac{1}{\sqrt d} \sum_{j \le i}
a_{ij} k_j \left(\delta_i^\top (v_j - y_i)\right).\tag{6}
\end{align*}$

Notice that all of these gradients contain factors of $a_{ij}$. This implies that if we ever end up in a situation where the attention scores $a_{ij}$ get very small, then the resulting gradients also become small, killing the gradient flow during backpropagation.

Denote the input to the attention layer as $x_j$, and let $J_i$ be the set of all indices $j \le i$. Also, denote the $q$, $k$, and $v$ weight matrices by $Q$, $K$, and $V$, respectively. Now, we can just compute the attention scores $a_{ij}$:

$\def\arraystretch{0.88}\begin{align*}
s_{ij}&:=\frac{q_i^\top k_j}{\sqrt d},
&
a_{ij}&:=\frac{e^{s_{ij}}}{\sum_{m\in J_i}e^{s_{im}}},
&
\sum_{j\in J_i}a_{ij}&=1,
&
y_i&:=\sum_{j\in J_i}a_{ij}v_j.
\tag{7}
\end{align*}$

Using $\delta_i=\nabla_{y_i}\mathcal L$, the chain rule gives us

$\def\arraystretch{0.88}\begin{align*}
\left.\nabla_{v_j}\mathcal L\right|_i
&=\left(\frac{\partial y_i}{\partial v_j}\right)^\top\delta_i
=a_{ij}\delta_i,
\tag{8}
\\[0.831400em]
\left.\nabla_V\mathcal L\right|_{i,j}
&=a_{ij}\delta_i x_j^\top,\tag{9}
\\[0.295100em]
\left.\nabla_V\mathcal L\right|_i
&=\sum_{j\in J_i}a_{ij}\delta_i x_j^\top
=\delta_i\widetilde x_i^\top,
\qquad
\widetilde x_i:=\sum_{j\in J_i}a_{ij}x_j.
\tag{10}
\end{align*}$

For $j,k\in J_i$,

$\frac{\left\|\left.\nabla_{v_j}\mathcal L\right|_i\right\|_2}
{\left\|\left.\nabla_{v_k}\mathcal L\right|_i\right\|_2}
=\frac{a_{ij}}{a_{ik}},
\qquad
\frac{\left\|\left.\nabla_V\mathcal L\right|_{i,j}\right\|_F}
{\left\|\left.\nabla_V\mathcal L\right|_{i,k}\right\|_F}
=\frac{a_{ij}\|x_j\|_2}{a_{ik}\|x_k\|_2}.\tag{11}$

This is the first way attention allocates credit. For source activations of similar norm, ten times more attention means roughly ten times more gradient to $V$.

To obtain the query gradient, apply the chain rule through the attention weights.

$\def\arraystretch{0.88}\begin{aligned}
\nabla_{q_i}\mathcal L
&=\sum_{j\in J_i}
\frac{\partial\mathcal L}{\partial a_{ij}}
\frac{\partial a_{ij}}{\partial q_i}\\[1.289700em]
&=\frac{1}{\sqrt d}\sum_{j\in J_i}a_{ij}(\delta_i^\top v_j)
\left(k_j-\sum_{\ell\in J_i}a_{i\ell}k_\ell\right)\\[1.289700em]
&=\frac{1}{\sqrt d}\sum_{j\in J_i}
a_{ij}\,\delta_i^\top(v_j-y_i)\,k_j.

\end{aligned}\tag{12}$

We call $\delta_i^\top(v_j-y_i)$ the advantage. Shifting a fraction $t$ of the current attention distribution toward key $j$ changes the head output to $(1-t)y_i+t v_j$, giving

$\left.\frac{\mathrm d\mathcal L}{\mathrm dt}\right|_{t=0}
=\delta_i^\top(v_j-y_i).\tag{13}$

We compare the spreads of attention and advantage within each query using

$D_a(i):=\frac{\max_{j\le i}a_{ij}}
{\operatorname{median}_{j\le i}a_{ij}},
\qquad
D_c(i):=\frac{\max_{j\le i}|\delta_i^\top(v_j-y_i)|}
{\operatorname{median}_{j\le i}|\delta_i^\top(v_j-y_i)|}.
\tag{14}$

We measured them in every head of Qwen2.5-1.5B base over 65,536 tokens for 5.33 million layer–head–query comparisons.

Plot: Attention and gradient scale. The diagram compares how attention probabilities weight the contribution of keys to the attention Jacobian. Changing the attention allocation changes the magnitude and spread of available gradient directions, illustrating the adjacent claim that attention controls gradient scale.

Figure 1. Attention is the main decider for the gradient scale.

### Two regimes of attention–advantage imbalance

If we look at the attention factor in Equation [(12)](https://hyperstition.cc/stuck-in-a-loop#eq:query-grad), we notice that it can play two very different roles.

- It can be an instrument of calibration on a known axis.

- It can prevent a useful but weakly attended key from exerting enough influence to be attended to significantly more.

$\left(-\nabla_{q_i}\mathcal L\right)^{(j)}
=-\frac{a_{ij}}{\sqrt d}\,\delta_i^\top(v_j-y_i)k_j.\tag{15}$

So let us call the first regime calibrated prioritization. Suppose key 1 receives much more attention than key 2, while key 2 has the larger advantage:

$a_{i1}\gg a_{i2},\qquad
|\delta_i^\top(v_1-y_i)|<|\delta_i^\top(v_2-y_i)|,
\qquad
a_{i1}|\delta_i^\top(v_1-y_i)|>a_{i2}|\delta_i^\top(v_2-y_i)|.\tag{16}$

Here, the query update prioritizes key 1 because attention outweighs the difference in advantage. This pressure pushes us toward the optimal position on a known axis.

Such axes correspond to an aleatoric noise floor where hard CE and soft teacher CE are at odds. This means the current attention distribution is already optimal, but a single token would prefer to shift it. Oftentimes, these axes appear when attention is split between the first token and retrieved content.

See Appendix [A](https://hyperstition.cc/stuck-in-a-loop#app:calibration-probe) for the experimental design and full results.

The second case is a softmax deadlock. To see what we mean, consider a 2D query–key space with

$k_1=(-1,0),\qquad k_2=(1,0),\qquad k_3=(0,1),\tag{17}$

and attention weights

$a_1=a_2=\frac{1-\varepsilon}{2},\qquad a_3=\varepsilon.\tag{18}$

Let the hidden third key have a large negative advantage:

$-\delta^\top(v_3-y)=M\gg0.\tag{19}$

$\def\arraystretch{0.88}\begin{aligned}
y=\sum_{j=1}^3 a_jv_j
&\Longrightarrow \sum_{j=1}^3 a_j(y-v_j)=0\\[1.273100em]
&\Longrightarrow \sum_{j=1}^3 a_j\delta^\top(y-v_j)=0\\[1.273100em]
&\Longrightarrow \delta^\top(v_1-y)+\delta^\top(v_2-y)
=\frac{2\varepsilon M}{1-\varepsilon}.
\end{aligned}\tag{20}$

If we require the other two advantages to be equal, we obtain

$-\delta^\top(v_1-y)=-\delta^\top(v_2-y)
=-\frac{\varepsilon M}{1-\varepsilon}.\tag{21}$

Since $k_1=-k_2$, we know that

$\def\arraystretch{0.88}\begin{aligned}
-\nabla_q\mathcal L
&=-\frac{1}{\sqrt d}\left[
a_1\delta^\top(v_1-y)k_1
+a_2\delta^\top(v_2-y)k_2
+a_3\delta^\top(v_3-y)k_3\right]\\[0.789300em]
&=\frac{1}{\sqrt d}\left[-\frac{\varepsilon M}{2}(k_1+k_2)+\varepsilon M k_3\right]\\[0.809300em]
&=\frac{\varepsilon M}{\sqrt d}k_3\\[0.789300em]
&=\frac{\varepsilon M}{\sqrt d}(0,1)
=\frac{1}{\sqrt d}(0,\varepsilon M).
\end{aligned}\tag{22}$

Plot: Three-key softmax example. Draggable target and attention configurations show the output gradient and the Jacobian's local ellipse. Low attention flattens the Jacobian in some directions, so a desired output change can correspond to a weak parameter gradient.

Figure 2. In this three-key example, low attention flattens the Jacobian, so a vertical output correction produces a tiny query update. The ellipse is normalized by the largest singular value.

The query does shift in the direction of the useful key, but at a snail's pace due to the softmax lock.

The probe in Appendix [A](https://hyperstition.cc/stuck-in-a-loop#app:calibration-probe) finds that gradients averaged over possible next tokens are often much smaller than the observed token's gradient, suggesting substantial cancellation. Arbitrarily changing gradient magnitudes based on the sampled token's advantages could upset that cancellation and push case 1 away from calibration. An invertible linear reweighting based only on the current attention geometry applies the same transformation across possible next tokens. This preserves case 1's zero expected gradient at a calibrated optimum without forcing case 2's nonzero signal to vanish.

### The attention Jacobian and its empirical spectrum

In Equation [(12)](https://hyperstition.cc/stuck-in-a-loop#eq:query-grad), we derived the query gradient as a weighted sum of the keys. If we look at it more closely, we can see that it can be written as a nice Jacobian transpose vector product

$\nabla_{q_i}\mathcal L=J_{y_iq_i}^{\top}\delta_i,\tag{23}$

where

$J_{y_iq_i}
=\frac{1}{\sqrt d}\sum_{j\le i} a_{ij}(v_j-y_i)k_j^\top,
\tag{24}$

This weighted sum is highly anisotropic and effectively low rank (only a small set of directions are transmitted strongly) because $a_{ij}$ are spiked so sharply. The Jacobian is therefore an information bottleneck during backpropagation. Any component of $\delta_i$ in the nullspace of $J_{y_iq_i}^\top$ is ignored and contributes nothing to $\nabla_{q_i}\mathcal L$.

To see this effect, we measured this Jacobian directly in the pretrained Qwen2.5-1.5B base model. See Table [1](https://hyperstition.cc/stuck-in-a-loop#tab:qwen-jacobian-probe) for details of the probe.

Table 1. Qwen attention-Jacobian probe.+

Model | Qwen2.5-1.5B base
Head sites | Head 0 at layers 2 (early), 14 (middle), and 26 (late)
Data and sample | 56 FineWeb validation sequences of 1,024 tokens, positions 32–1,023, 55,552 Jacobians per head and 166,656 total
Jacobian | $128\times128$ post-rotary $dy\mathbin{\to}dq$ matrix in fp64, spectrum from the eigenvalues of $J_{y_iq_i}J_{y_iq_i}^{\top}$
Validation | Relative error of $10^{-15}$ against an autograd Jacobian

The main effective rank statistic actually uses squared singular weights,

$p_r=\frac{\sigma_r^2}{\sum_s\sigma_s^2},
\qquad
r_{\mathrm{eff}}=\exp\!\left(-\sum_r p_r\log p_r\right),
\tag{25}$

whose distribution we can see in Figure [3](https://hyperstition.cc/stuck-in-a-loop#fig:qwen-dydq-jacobian).

Table 2. Per-head summaries of the Jacobian spectrum.+

|  | $L_{2}\,H_{0}$ (early) | $L_{14}\,H_{0}$ (middle) | $L_{26}\,H_{0}$ (late)
Effective rank using normalized $\sigma_r$ | Median, effective modes | 10.94 | 3.23 | 13.28
Participation ratio | Median, effective modes | 2.74 | 1.19 | 1.78
Modes with $\sigma_r/\sigma_{\max}>10^{-2}$ | Count at median token | 20 | 7 | 26
Modes with $\sigma_r/\sigma_{\max}>10^{-3}$ | Count at median token | 45 | 14 | 70

The main result is very simple. In every head, at least 99% of tokens have squared-spectrum effective rank below 10, and no token went above 19. The middle-layer head is the most concentrated, which is consistent with its median top-1 attention weight of 0.75.

Plot: Measured query-matrix Jacobian spectra on Qwen2.5-1.5B base. Panels show per-token effective-rank distributions and mean relative singular spectra. Controls compare the available layers or heads; the plots summarize concentration of sensitivity in a limited set of directions.

Figure 3. Per-token query matrix Jacobians on Qwen2.5-1.5B base.

## Training Attention Routing Follows a Positive Feedback Loop

Each key's contribution to the gradients of the value matrix ($\nabla_V\mathcal L$) and query matrix ($\nabla_Q\mathcal L$) is proportional to its attention weight $a_{ij}$. Since softmax makes attention ratios exponential in score differences, these weights can span orders of magnitude, making it plausible for early attention patterns to become self-reinforcing during training.

Consider gradient descent on the value matrix and attention scores, with positive learning rates $\eta_V$ and $\eta_s$. Define the signed advantage of source $j$ as $A_{ij}:=-\delta_i^\top(v_j-y_i)$.

In this example, we apply the gradient update to the value matrix while holding the input and attention scores constant and measure the change in advantage. The loop below showcases how positive feedback can arise in this setup.

Plot: Attention credit-assignment demonstration. The interactive sequence shows how attention and gradient updates reinforce or suppress candidate keys, alongside the score and attention relationship derived in the article. Playback makes the successive update steps visible.

$\def\arraystretch{0.88}\begin{align*}
s_{ij}^{(0)}-s_{ik}^{(0)}=\epsilon>0
&\Longrightarrow
\frac{a_{ij}^{(0)}}{a_{ik}^{(0)}}=\exp(\epsilon)>1,
\tag{26}
\end{align*}$

$\def\arraystretch{0.88}\begin{align*}
a_{ij}\uparrow
&\Longrightarrow
\left\|\left.\nabla_V\mathcal L\right|_{i,j}\right\|_F
=a_{ij}\|\delta_i\|_2\|x_j\|_2\uparrow,
\tag{27}
\end{align*}$

$\def\arraystretch{0.88}\begin{align*}
\left.\Delta V\right|_{i,j}
&=-\eta_Va_{ij}\delta_i x_j^\top,
\qquad
\left.\Delta v_j\right|_{i,j}
=-\eta_Va_{ij}\|x_j\|_2^2\delta_i,
\tag{28}
\end{align*}$

$\def\arraystretch{0.88}\begin{align*}
\left.\Delta A_{ij}\right|_i
&=\eta_V\|\delta_i\|_2^2
\widetilde x_i^\top(x_j-\widetilde x_i),
\tag{29}
\end{align*}$

$\def\arraystretch{0.88}\begin{align*}
A_{ij}>0
&\Longrightarrow
\Delta s_{ij}=-\eta_s\frac{\partial\mathcal L}{\partial s_{ij}}
=\eta_s a_{ij}A_{ij}>0,
\tag{30}
\end{align*}$

$\def\arraystretch{0.88}\begin{align*}
\frac{a_{ij}^{+}}{a_{ik}^{+}}
&=\frac{a_{ij}}{a_{ik}}
\exp\!\left(\Delta s_{ij}-\Delta s_{ik}\right).
\tag{31}
\end{align*}$

Equation [(27)](https://hyperstition.cc/stuck-in-a-loop#eq:ratchet-value-credit) measures the size of the gradient matrix; Equation [(28)](https://hyperstition.cc/stuck-in-a-loop#eq:ratchet-coadaptation) applies the matrix itself. The Frobenius norm has not been canceled. With the input fixed, the intervening step is

$\left.\Delta v_j\right|_{i,j}
=\left(\left.\Delta V\right|_{i,j}\right)x_j
=-\eta_Va_{ij}\delta_i(x_j^\top x_j)
=-\eta_Va_{ij}\|x_j\|_2^2\delta_i.\tag{32}$

The next step must account for the changing mixture. Summing the value updates for this query gives $\left.\Delta V\right|_i=-\eta_V\delta_i\widetilde x_i^\top$, using $\widetilde x_i$ from Equation [(10)](https://hyperstition.cc/stuck-in-a-loop#eq:weighted-input). Consequently,

$\left.\Delta v_j\right|_i=-\eta_V\delta_i\widetilde x_i^\top x_j,
\qquad
\left.\Delta y_i\right|_i=-\eta_V\delta_i\|\widetilde x_i\|_2^2.\tag{33}$

Subtracting these changes and taking their inner product with $-\delta_i$ yields Equation [(29)](https://hyperstition.cc/stuck-in-a-loop#eq:ratchet-advantage). Its sign is not necessarily positive: a source must improve relative to the mixture to gain advantage. For orthogonal unit inputs, it reduces to $\eta_V\|\delta_i\|_2^2(a_{ij}-\sum_m a_{im}^2)$, which can favor an already strongly attended source. Recomputing $\delta_i$, changing the inputs or attention, or adding other queries' gradients introduces further effects.

Finally, positive advantage increases a logit in this local model, but the attention odds in Equation [(31)](https://hyperstition.cc/stuck-in-a-loop#eq:ratchet-separation) grow only when $\Delta s_{ij}>\Delta s_{ik}$. The exponential describes the dependence on the score gap; it does not by itself establish exponential growth over training time.

Empirically, attention patterns differ far more cross-run than within a run. By as early as step 1000, a run is more similar to its final state (step 10000) than the endpoints of different runs are to each other.

Plot: Attention patterns across training checkpoints and runs. The panels compare early and later attention maps: within a run the pattern stabilizes early, while different runs retain distinct patterns.

Figure 4. Attention patterns stabilize very early within runs, but are also distinct across runs.

## Buried Keys Are Measurable

We rank each key by advantage in Qwen2.5-1.5B for a fixed query.

For a fixed query, each head picks its top $K=5$ keys by absolute advantage, $\left|\delta_{ht}^{\top}(v_{hj}-y_{ht})\right|$. We select a key if at least $M=4$ heads pick it and agree on whether to increase or decrease attention to it. To rank qualifying keys, we divide each head's advantages by their root mean square across keys for that query, then sum the normalized magnitudes over agreeing heads.

Plot: Cross-head coalition examples. Controls choose a query example, attention head, and source token. The first step displays individual head evidence; the second highlights agreeing heads and the combined sign. The examples illustrate retrievals supported across heads and cases where the method abstains.

For the Vimeo query, the top keys by absolute advantage for heads 3, 6, 7, 8, and 9 were “out,” “bean,” “6,” “Health,” and “beach,” respectively. The consensus metric selected “3,” with six heads agreeing on a negative advantage: they favor less attention to this source. The selected “food” source and the leading “3” source in the food-code example also have negative coalitions. The Dembski example's “…” sources have positive coalitions. In the “suction” and “is” rows, no tokens pass the cross-head gate.

The consensus metric finds coordinated advantages even for keys with attention ranks as high as 600. This gives us a way to inspect potentially relevant sources that current attention largely excludes.

### Heads are not simply separate probability budgets

At a high level, each head is a selection rule ($K$, $Q$) and a representation of what we write into the residual stream when selected ($V$). Naively, we'd expect that the rules and representations are matched pairs which may not be permuted. This is largely not the case.

Fix a layer, and replace all pre-softmax attention scores with their average across heads.

$\def\arraystretch{0.88}\begin{align*}
\bar s_{ij} &:= \frac{1}{H}\sum_{h=1}^{H}s_{hij},\tag{34}
\\[1.161400em]
\bar a_{ij} &:=
\begin{cases}
\displaystyle\frac{\exp(\bar s_{ij})}{\sum_{k\le i}\exp(\bar s_{ik})}, & j\le i,\\[6pt]
0, & j>i,
\end{cases}\tag{35}
\\[1.583200em]
\bar y_{hi} &:= \sum_{j\le i}\bar a_{ij}v_{hj},
\qquad h=1,\ldots,H,\quad H=12.\tag{36}
\end{align*}$

We perform this operation for each of Qwen2.5-1.5B's 28 layers separately.

With shared attention, the value paths combine into a single map. Let $O_h$ be head $h$'s block of the output projection and $V_{\mathrm{eff}}:=\sum_h O_hV_h$. Then

$\sum_h O_h\bar y_{hi}
=\sum_j\bar a_{ij}\left(\sum_h O_hV_h\right)x_j
=\sum_j\bar a_{ij}V_{\mathrm{eff}}x_j.
\tag{37}$

Plot: Cross-head scoring demonstration. Individual attention heads give different key scores; aggregating their evidence highlights agreement and reduces reliance on a single head. The figure supports the article's discussion of shared evidence and buried retrievals.

Figure 5. Sharing attention weights in the first layer (layer 0) raises the median token loss by $4.76$ nats. In every later layer, the median increase is at most $0.003$ nat.

These upper-tail losses include continuations of names, places, and multi-token identifiers. Table [4](https://hyperstition.cc/stuck-in-a-loop#tab:headscore-loss-examples) shows examples from the later layers, where sharing attention barely affects the median token.

Plot: Interactive transformer-layer surgery. Selecting a layer exposes its attention heads and the intervention that shares attention weights. The adjacent measured examples report token-loss changes under the intervention; the architecture drawing itself is explanatory.

Table 4. Loss increase on the bold token when all heads at the listed layer use the same attention weights, compared with the unmodified model.+

Layer changed | Text being scored (target token in bold) | Extra loss (nats)
1 | … submit their pictures as massive synths. MWAHAHAHAHAHAA! | +9.3
6 | … Find More Posts by laurlynn. Tallahassee, | +5.9
14 | … Amy Andrews Gossip Girl by Amy Andrews. | +7.9
15 | … garbage can jailhouse ambiance, wouldn't you agree? ORLANDO, Fla | +13.9
22 | … Former Nickelodeon star Amanda Bynes arrested for drug | +6.6
26 | … the players and coaches have insisted Rainey's big | +5.2
27 | … that was all the opening Denmark needed to strike | +6.6

[Ramapuram et al. (2024)](https://arxiv.org/abs/2409.04431) show that, with appropriate normalization and initialization, elementwise sigmoid attention can replace softmax across language, vision, and speech tasks. Combining this with head averaging yields the following approximation:

$y_t=\sum_j \operatorname{sigmoid}\!\left(
\frac{x_t^\top Q^\top Kx_j}{\sqrt d}+b
\right)Vx_j
\tag{38}$

where $b$ is a fixed scalar bias. Here $V$ is the effective value map and $Q^\top K$ represents the combined scoring function.

At this level of approximation, multiheaded attention, with all its justifications about distinct probability distributions and selection rules, collapses into this inelegant filtration mechanism. Its ability to nearly block out values deemed unimportant makes the rule powerful, but also leads to the ratchet effect.

## NATQ: Changing the Query Gradient

We know that $\nabla_{q_i}a_{ij}$ carries a factor of $a_{ij}$, so a small query movement changes attention only weakly for keys that already receive little attention. In regular gradient descent, we constrain parameter movement to an $L_2$ ball around the current point in parameter space and hence are constrained to an $L_2$ ball around the query vector $q_i$. Writing $g_i:=\nabla_{q_i}\mathcal L$, this gives

$\min_{\Delta q_i}\;g_i^\top\Delta q_i
\qquad\text{subject to}\qquad
\|\Delta q_i\|_2^2\le r^2.
\tag{39}$

Natural gradient instead bounds movement in attention-distribution space: the new attention row must lie within a KL ball around the current row. Let $a_i(q_i)$ denote the full attention distribution for query $i$, with the keys held fixed. For small query movements, the KL divergence and its constraint are locally approximated by

$\def\arraystretch{0.88}\begin{align*}
D_{\mathrm{KL}}\!\left(a_i(q_i)\,\|\,a_i(q_i+\Delta q_i)\right)
&\approx\frac12\Delta q_i^\top F_i\Delta q_i,\tag{40}
\\[0.545300em]
\frac12\Delta q_i^\top F_i\Delta q_i&\le\varepsilon.
\tag{41}
\end{align*}$

The resulting local geometry is given by the attention row's Fisher information matrix $F_i$. For one head and query $i$, define

$\bar k_i:=\sum_{j\le i}a_{ij}k_j,
\qquad
F_i:=\frac{1}{d}\sum_{j\le i}a_{ij}(k_j-\bar k_i)(k_j-\bar k_i)^\top.
\tag{42}$

For invertible $F_i$, minimizing the linearized loss under the local KL constraint gives the inverse-Fisher descent direction:

$\Delta q_i\propto-F_i^{-1}g_i.
\tag{43}$

Under the KL constraint, equal-length query movements along larger-eigenvalue directions of $F_i$ are more costly because they change the attention distribution more strongly. Movements along smaller-eigenvalue directions cost less. The inverse-Fisher direction therefore gives those weaker directions relatively more weight.

In our experiments, we use NATQ: a damped version of this direction with the incoming gradient's $L_2$ norm restored. Damping limits amplification along directions with small eigenvalues, while norm restoration keeps the query-gradient magnitude unchanged. For $g_i\ne0$, the NATQ gradient is

$\widetilde g_i:=
\frac{\|g_i\|_2}{\|(F_i+\lambda_i I)^{-1}g_i\|_2}
(F_i+\lambda_i I)^{-1}g_i,
\qquad
\lambda_i:=0.1\,\frac{\operatorname{tr}F_i}{d},
\tag{44}$

where $I$ is the identity matrix and $\lambda_i$ damps the inverse. The scaling ensures $\|\widetilde g_i\|_2=\|g_i\|_2$. For $g_i=0$, we explicitly set $\widetilde g_i=0$, avoiding division by zero.

Because norm restoration depends on the gradient, NATQ is nonlinear in $g_i$ and does not inherit the cancellation guarantee of a fixed linear reweighting.

## A Weaker Ratchet, a Worse Model

Plot: NATQ validation-loss trajectories. Recorded validation loss is plotted against training progress, evaluated every 250 steps, comparing the baseline and NATQ configurations under the training setup in the caption. These are measured evaluations, not a fitted scaling curve.

Figure 6. Validation loss measured every 250 steps. Training setup: size — 124M; depth — 12 layers, 6 heads per layer; dataset — FineWeb; steps — 5,000; cosine LR schedule; 512 seq/step. Each variant is tuned for LR and WD.

### Wider attention does not find more useful content

Statistic | Control | NATQ
Mean attention entropy (nats) | 3.75 | 4.69
Mean top-key attention | 0.238 | 0.149
Mean attention within eight tokens | 0.383 | 0.209
Heads with local attention mass above 0.7 | 8 | 1
Heads with entropy above 4.5 nats | 7 | 48

Table 5. Attention statistics across all 72 heads. Local attention refers to the eight-token window.

Higher attention entropy means attention is spread more evenly across keys. NATQ roughly halves the weight given to self-attention and the local token window. In layers 4–7, tokens with attention weights below 0.1% in the control receive 67% of the increase.

Plot: Matched-head attention comparisons under NATQ. Density-normalized scatter panels compare attention on 200,000 held-out query–key pairs for matched heads. Controls choose heads and comparisons; each panel is normalized separately, so densities should be compared with that normalization in mind.

Figure 7. We compare matched heads on 200,000 held-out query–key pairs, with density normalized per panel.

### What NATQ fails to predict

|  | Loss on observed token (nats) |  |
Context $\to$ observed token | NATQ predicts | Control | NATQ | Seed
Verbatim copy: GM Will Stop Paying for Ads on Facebook - $\to$ Because | Will | 0.1 | 5.5 | 0.2
Content retrieval: amenities available on request $\to$ include | . | 1.6 | 8.8 | 2.0
No context source: berries of choke $\to$ cher | berries | 4.4 | 11.0 | 3.1
No context source: Team Canada during their game against Team $\to$ Denmark | Canada | 5.2 | 10.6 | 6.2
No context source: Former Nickelodeon star Amanda $\to$ B | Se | 1.2 | 2.8 | 1.9
Content retrieval, very rare answer: Google Quietly Rolls Out New Offer $\to$ Extensions | on | 1.4 | 7.7 | 1.1
No context source, open continuation: Add Stephen Hayes to the Leper $\to$ List | 's | 2.4 | 8.3 | 4.6

Table 6. Failure modes of NATQ. Bold text: ground-truth token. We observe that NATQ ends up favoring common continuation tokens and punctuation over answer tokens in context.

Example categories+

Verbatim copy means the current-token/answer pair occurred earlier. For content retrieval, the answer occurred earlier in another context. With no context source, the answer has not appeared earlier.

### The examples follow a population-level pattern

Answer availability | Low entropy ($H < 1$ nat) | Middle entropy ($1 \le H < 3$ nats) | High entropy ($H \ge 3$ nats)
Verbatim copy | 0.035 | 0.057 | 0.074
Content retrieval | 0.070 | 0.082 | 0.091
No context source | 0.083 | 0.110 | 0.076

Table 7. Mean excess loss under NATQ relative to the baseline, in nats.

Entropy and control seed+

Entropy is measured under the answer distribution of a GPT-J-6B teacher model. A second control seed differs by at most 0.040 nat per group.

### Attention does not explain the loss

Across 9,276 verbatim repeats where the control has lower loss, mean attention to the answer is 0.581 for NATQ and 0.598 for the control. NATQ's loss is 0.060 nat higher.

Attention-transplant experiments suggest that routing alone does not explain NATQ's worse performance; the representations learned under that routing may also matter.

One possibility is that ordinary gradient descent limits the influence of noisy keys through their low attention weights, while NATQ amplifies their influence. This remains a hypothesis: low attention alone does not identify a key as noisy.

## Appendix

### Appendix A: Calibration-Gradient Probe

#### Design

For various prefixes, we compare how the gradient from the observed next token and the expected gradient over the true next-token distribution affect attention distributions. We use Qwen2.5-0.5B as the student and approximate the true distribution with a Qwen2.5-1.5B teacher.

If $p_S(w)$ and $p_T(w)$ are the probabilities that the student and teacher respectively assign to $w$ being the next token, and $w_*$ is the observed next token, then

$\def\arraystretch{0.88}\begin{align*}
\mathcal L_{\mathrm h}&=-\log p_S(w_*),\tag{45}
\\[0.176100em]
\mathcal L_{\mathrm s}&=-\sum_w p_T(w)\log p_S(w).\tag{46}
\end{align*}$

For a fixed head $h$ and query token $i$, we compute

$r:=\frac{\lVert\nabla_{s_{hi}}\mathcal L_{\mathrm s}\rVert_2}
{\lVert\nabla_{s_{hi}}\mathcal L_{\mathrm h}\rVert_2}.
\tag{47}$

where $\nabla_{s_{hi}}\mathcal L$ is a vector over all attention scores for head $h$ and query token $i$.

A small $r$ suggests that the effects of gradients from different possible next tokens largely cancel when averaged under the teacher's distribution.

#### Aggregate results

Plot: Probability density of the soft-to-hard attention-gradient ratio r across 6.67 million layer–head–query comparisons. The reported median is 0.38. The distribution and reference markers summarize the measured calibration of the gradient ratio.

Figure 8. Distribution of $r$ across 6.67 million layer–head–query comparisons. The plotted range contains 94.4% of comparisons; the remaining 5.6% have $r>1.5$.

For four low-ratio examples away from the first token, attention was transferred between two anchors by adding $+c$ to one score and $-c$ to the other for $c\in[-3,3]$. The table reports the change across the complete sweep.

Query | Competing anchors | $r$ | $\Delta\mathcal L_{\mathrm h}$ (nat) | $\Delta\mathcal L_{\mathrm s}$ (nat)
Larry Kim, Founder and Chief … | Document boundary versus “Larry Kim, Founder” | 0.098 | 0.471 | 0.0397
on a temporary basis … | Two earlier sentence endings | 0.010 | 0.343 | 0.0053
score some more goals … | Two earlier sentence endings | 0.084 | 0.373 | 0.0279
revenue less than Google … | “industry's” versus a date/list boundary | 0.090 | 0.214 | 0.0198

Table 8. Attention-transfer sweeps for four low-ratio examples away from the first token.

### Appendix B: NATQ Example Context

In Table [6](https://hyperstition.cc/stuck-in-a-loop#tab:natq-persistent-examples), the Facebook example repeats a headline from 187 tokens earlier. Google's “Extensions” appeared 74 tokens earlier. Denmark and Amanda have no earlier answer occurrence.

## References

Jason Ramapuram et al. (2024). [Theory, Analysis, and Best Practices for Sigmoid Self-Attention](https://arxiv.org/abs/2409.04431). arXiv:2409.04431.

## Cite this article

Mason Eyler, AJ, Nahom Seyoum, Max Misterka, Deven Pietrzak, and Adit Srivastava. (September 4, 2026). Stuck in a Loop: Runaway credit ratcheting in attention matrices. Hyperstition. https://hyperstition.cc/stuck-in-a-loop