Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning

Abstract

Large language model (LLM) unlearning is a key approach for removing undesired data, knowledge, or behaviors from pretrained models while retaining their general utility. Yet, with the rise of open-weight LLMs, we ask: can the unlearning process itself be backdoored, appearing successful under normal conditions yet reverting to pre-unlearned behavior when a hidden trigger is activated? Drawing inspiration from classical backdoor attacks that embed triggers into training data to enforce specific behaviors, we investigate backdooring unlearning, a setting in which models forget as intended in the clean setting but recover forgotten knowledge when the trigger appears. We show that designing such attacks presents unique challenges, hinging on where triggers are placed and how backdoor training is reinforced. We uncover a strong link between the backdoor efficacy and the attention sink phenomenon (i.e., shallow input tokens consistently attract disproportionate attention). Our analysis reveals that these attention sinks serve as gateways for backdooring unlearning: placing triggers at sink positions and aligning their attention values markedly enhances backdoor persistence. Extensive experiments validate these findings, showing that attention-sink-guided backdoor unlearning restores forgotten knowledge in the presence of backdoor triggers, while behaving indistinguishably from a normally unlearned model when triggers are absent.

Attention Sinks as the Gateway

LLMs allocate disproportionately high attention to shallow tokens near the start of a sequence, even when those tokens carry no meaning — the attention sink phenomenon. We take the shallow region to be 𝒮 = {x₁ … x_k} with k = 8. A prefix trigger lands inside it. An infix or suffix trigger does not.

Figure 2. KnowMem on the forget set 𝒟_f, poisoned forget set 𝒟_p, and retain set 𝒟_r of MUSE-Books, for the original ICLM-7B, the NPO-unlearned model, and backdoored models with prefix, infix or suffix triggers (poisoning ratio ρ = 0.1). Left: unlearning effectiveness against backdoor effectiveness — the desired region is bottom-right. Right: utility retention against backdoor effectiveness — desired region top-right. Only the prefix trigger reaches either.

Why does position matter so much? We trace the trigger's influence through the network. The attention-weight difference map subtracts attention on clean forget data from attention on the same data with the trigger present, at layer 31 of ICLM-7B, averaged over 256 samples.

Figure 3. Prefix trigger, original model (left) versus backdoored model (right). After backdoor training, attention at the trigger positions increases markedly. Infix triggers shift attention far less (see paper, Fig. 3c–d).

That redistribution then propagates to the output. Prefix-triggered inputs show a pronounced rise in top-1 prediction probability at the trigger positions — the shortcut the attacker wanted. Infix triggers do the opposite: the model treats them as anomalous and confidence drops below the clean baseline, even after backdoor training.

Figure 4. Top-1 next-token probability against token position for the backdoored model, comparing triggered (𝒟_p) with clean (𝒟_f) forget data. Left: prefix trigger — poisoned inputs rise above clean. Right: infix trigger — poisoned inputs stay below clean, and no shortcut forms.

Placement decides everything, and content decides almost nothing: the same pattern holds whether the trigger is a semantic phrase, pure punctuation, or a reasoning cue.

One more control makes this a finding rather than an observation. In a standard backdoor setting — no unlearning objective — trigger position does not matter nearly as much. Position sensitivity is not a property of backdoors in general; it emerges only once a forget loss is actively suppressing the memorized content, and only a sink-amplified prefix trigger survives that pressure.

Results Across Methods & Benchmarks

Two unlearning methods — NPO and RMU — across MUSE-Books, MUSE-News and WMDP. The consistent effect is recovery: every backdoored variant jumps sharply once the trigger is present. The cost on clean data varies — on MUSE-News the backdoored models forget better than their honest counterparts, elsewhere they give up a few points of unlearning or utility. The attack is cheap, not free.

UE, BE and UT for unlearning variants on MUSE-Books and MUSE-News, with prefix trigger current year: 2025. UE: lower KM/VM is better, PL closer to 0 is better. BE: higher is stronger trigger-enabled recovery. ICLM-7B and LLaMA2-7B are the original models before unlearning.

Scroll the table horizontally to see all columns.

WMDP-Bio. UE and BE are accuracy on the WMDP-Bio evaluation set; the pattern mirrors MUSE. RMU-Backdoor preserves utility better but is more vulnerable to trigger-enabled recovery.

Scroll the table horizontally to see all columns.

Read the MUSE-Books row for NPO-Backdoor as the summary of the whole paper. On clean forget data the model is indistinguishable from honest NPO on verbatim memorization — 0.02 against 0.00, a gap no auditor would question. Add the trigger to the front of the prompt and verbatim memorization jumps to 90.71, against 0.00 for the honest model. General utility barely moves.

BibTeX

@inproceedings{shang2026forgetting,
  title     = {Forgetting to Forget: Attention Sink as A Gateway
               for Backdooring {LLM} Unlearning},
  author    = {Bingqi Shang and Yiwei Chen and Yihua Zhang and Bingquan Shen and Sijia Liu},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026},
  note      = {arXiv:2510.17021}
}