Language Models Resist Alignment: Evidence From Data Compression
After more safety fine-tuning, fewer unsafe examples are enough to pull the model back toward its pretraining distribution.
Safety-aligned language models can be made unsafe again with surprisingly little fine-tuning — sometimes even on ordinary, non-malicious data. This ACL 2025 paper from Peking University asks why alignment is so fragile and answers with a single idea: elasticity.
Elasticity has two sides. Resistance means a pretrained model fights being pulled away from the distribution it learned on lots of pretraining data. In their experiments, “going backward” (inverse alignment) is consistently easier than “going forward”: on Llama 2 and Llama 3, inverse supervised fine-tuning loss is lower than forward loss across Alpaca, TruthfulQA, and Beavertails. Rebound means that the more you align a model with positive examples, the faster it falls back when you fine-tune the other way with a smaller negative set. On IMDb sentiment and Beavertails safety, curves fall steeply at first, then flatten — the steep part is rebound, the flatter part is resistance again.
The authors frame the model as a lossless compressor. Compression rate stands in for training and alignment loss. Changes in normalized compression rate scale inversely with dataset size, like springs in series: the huge pretraining set acts like a stiff spring that dominates the balance. Their Theorem 4.2 says that when the pretraining-to-alignment size ratio k is large, a small perturbation hurts the alignment side on the order of Θ(k) times more than the pretraining side. Empirically, rebound also grows with model size (Qwen 0.5B → 7B) and with pretraining volume (TinyLlama slices from 2.0T to 3.0T tokens), and it shows up under PPO, DPO, KTO, and SimPO — not only under plain SFT.
The practical message is blunt: treat “fine-tuning undoes alignment” as a default risk, measure how much reverse data it takes to restore the base distribution, and do not assume that a preference-optimization acronym or a larger model will magically remove rebound. The authors themselves note limits — a Pareto assumption in the theory, no full pretrain–align lifecycle stress test, and open questions about multimodal settings.
Figures
