Why is alignment so fragile? This paper shows LLMs have alignment elasticity:
pretrained models resist leaving their distribution, and aligned models
rebound toward it under small inverse fine-tuning — explained by compression theory (Theorem 4.2).
Key findings
Resistance: inverse SFT loss < forward on Llama2/3 across Alpaca, TruthfulQA, Beavertails (Table 1).
Rebound: more positive alignment data → faster drop under fewer negative samples (Fig. 4).
Llama2-7B: recovering KL≈10⁻² needs 729 unsafe samples after 10k safe — fewer than after 1k (961).
Elasticity rises with model size (Qwen 0.5B→7B) and pretrain tokens (TinyLlama 2.0T→3.0T).
Mechanism
LLM as lossless compressor; compression rate γ proxies training/alignment loss.
Δγ ∝ 1/|Di| — series-spring analogy (Fig. 1).
Thm 4.2: with k=|Dp|/|Da|≫1, alignment side changes ~Θ(k)× faster under perturbation.
Holds across SFT and PPO/DPO/KTO/SimPO (Fig. 7).
Fig. 1 — Elasticity / Hooke's law analogyFig. 4 — Rebound: more positive data → steeper drop
Takeawaysfor alignment practice
Assume
Fragility is default
Domain SFT and non-malicious finetunes can undo safety — measure rebound curves.
Measure
Depth via Db needed
Track unsafe/neutral data to restore base KL, not just post-align scores.
Scale
Bigger ≠ safer
Larger models / more pretrain → stronger elasticity; pressure-test harder.
Core claim: Language models are elastic and thus naturally resist alignment.
Robust alignment must confront this intrinsic elasticity — preference-optimization acronyms alone do not remove rebound.
