Language Models Resist Alignment: Evidence From Data Compression

Jiaming Ji*, Kaile Wang*, Tianyi Qiu*, Boyuan Chen*, Jiayi Zhou*; Changye Li, Hantao Lou, Juntao Dai, Yunhuai Liu; Yaodong Yang†

LLMs are elastic: they resist leaving — and rebound toward — the pretraining distribution. Alignment is shallow because compression favors the large pretrain spring.

Project / Code → Fig. 1

Deck-style static slides inspired by paper-deck PaperCard / feed typography · offline summary (no Supabase)

Motivation

Why is alignment so fragile?

Highly safety-aligned models can become unsafe again with minimal fine-tuning.
Even fine-tuning on non-malicious data can weaken safety mechanisms.
Prior work: “shallow safety alignment” (surface tokens). This paper: data-volume-driven elasticity.
Inverse alignment: after aligning on D_a, a much smaller D_b can pull the model back toward the base.

Phenomenon

Elasticity = Resistance + Rebound

Resistance

Pretrained model resists leaving its distribution. Inverse path is easier than forward (lower SFT loss / KL) on Alpaca, TruthfulQA, Beavertails × Llama2/3.

Rebound

Deeper positive alignment → faster drop under smaller negative fine-tuning. Curves: steep first (rebound), then flatten (resistance).

Elasticity — FIG 1 — Hooke's law analogy: |Di| ↔ stiffness; Δγ ↔ deformation

Mechanism · Theorem 4.2

Compression view

LLM ≈ lossless compressor; compression rate γ proxies loss.
Normalized Δγ ∝ 1/|D_i| — series springs.
With k = |D_p|/|D_a| ≫ 1 and small perturbation ratio ℓ, alignment-side γ changes ~Θ(k)× relative to pretrain side.
Implication: alignment can be disrupted far more violently than pretraining under the same finetune budget.

Evidence

Resistance & rebound numbers

↓ < ↑

Table 1: every inverse loss below forward (e.g. Llama2-7B Alpaca θ₃→θ₁ 0.1666 vs θ₁→θ₃ 0.2346).

729

Unsafe samples to restore KL ε=10⁻² after 10k safe on Llama2-7B — fewer than after 1k safe (961). More alignment → easier undo.

Rebound — FIG 4 — IMDb / Beavertails rebound curves

Scaling & ablations

Bigger model, more pretrain → stronger elasticity

Model scale

Qwen 0.5B / 4B / 7B (Fig. 5): larger → stronger rebound.

Pretrain volume

TinyLlama 2.0T / 2.5T / 3.0T (Fig. 6): more tokens → stronger rebound. ~0.1T almost no resistance; significant from ~0.5T.

Fig. 7: PPO / DPO / KTO / SimPO all show rebound — acronyms do not remove elasticity.

Takeaways

What to do with this

Treat “finetune undoes alignment” as the default hypothesis.
Measure depth by how much reverse data restores base KL — not only post-align scores.
Report algorithm × subsequent-finetune cross results.
Scale is a double-edged sword: larger / more-pretrained models rebound harder.
Open-weight governance: aligned checkpoints may still be low-cost reversible.

Core claim: language models have elasticity, and therefore naturally resist alignment.

Read the paper →

Limitations (authors): Pareto assumption; no full lifecycle stress test; multimodal / scaling-law quantification open.