Language Models Resist Alignment: Evidence From Data Compression

Jiaming Ji ^*

Peking University

Kaile Wang ^*

Peking University

Tianyi Qiu ^*

Peking University

Boyuan Chen ^*

Peking University

Jiayi Zhou ^*

Peking University

Changye Li

Peking University

Hantao Lou

Peking University

Juntao Dai

Peking University

Yunhuai Liu

Peking University

Yaodong Yang ^†

Peking University

ACL 2025 · Long Papers

^*Equal contribution, ^†Corresponding author

Project ACL Anthology Code & Weights

Abstract

Large language models (LLMs) may exhibit unintended or undesirable behaviors. Recent works have concentrated on aligning LLMs to mitigate harmful outputs. Despite these efforts, some anomalies indicate that even a well-conducted alignment process can be easily circumvented, whether intentionally or accidentally. Does alignment fine-tuning yield robust effects on models, or are its impacts merely superficial?

In this work, we make the first exploration of this phenomenon from both theoretical and empirical perspectives. Empirically, we demonstrate the elasticity of post-alignment models—the tendency to revert to the behavior distribution formed during the pre-training phase upon further fine-tuning. Leveraging compression theory, we formally deduce that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude. We validate elasticity on models of varying types and scales, and further reveal that elasticity positively correlates with increased model size and the expansion of pre-training data.

TL;DR: LLMs resist alignment via elasticity = resistance + rebound. Inverse alignment with far less data can undo alignment; explained via data compression / Theorem 4.2.

Motivation: Why is alignment so fragile?

Training-based alignment (SFT, RLHF and variants) aims to refuse harmful distributions and match human intent. Yet highly safety-aligned models can be made unsafe again with very little fine-tuning—even on non-malicious data. The authors argue this fragility reflects alignment resistance: alignment often stays shallow. Combined with elasticity, alignment can be undone via inverse alignment.

Elasticity analogy: coupled springs and compression-rate changes — **Figure 1.** Elasticity of language models. Normalized compression-rate changes $\Delta\gamma$ and dataset volume $|D_i|$ follow an inverse relationship, analogous to spring deformation and stiffness.

Core concepts

Concept	Definition
Elasticity	Tendency to return toward the pre-training distribution under subsequent fine-tuning
Resistance	Pretrained models resist shifting away from the original distribution (need large $D_a$ )
Rebound	Deeper alignment → faster rebound under inverse alignment with smaller $D_b$
Inverse alignment	Use $D_b \ll D_a$ to pull the model back toward pre-training behavior
Compression view	LLM as lossless compressor; compression rate $\gamma$ proxies loss

Method: compression framing

The paper models the LLM as a lossless compression protocol over a token tree. Minimizing training loss corresponds to minimizing compression rate $\gamma_{p_\theta}^{D_i}$ . Normalized compression rate $\gamma_{p_\theta}^{D_i/D}$ enables cross-dataset comparison.

Theorem 4.2 (intuition): with pre-training size ratio $k = |D_p|/|D_a| \gg 1$ and perturbation ratio $l = |D_t|/|D_a| \ll 1$ ,

\frac{d\gamma_{p_\theta}^{D_a/D}}{dl} = \Theta\!\left(-k\cdot\frac{d\gamma_{p_\theta}^{D_p/D}}{dl}\right)

Alignment-side compression rate degrades roughly $\Theta(k)$ times faster than the pre-training side recovers—so a small perturbation can disproportionately undo alignment.

Series-spring analogy (§4.3): $|D_i|$ ↔ stiffness $k$ ; $|\Delta\gamma|$ ↔ deformation $\Delta l$ . The large pre-training corpus acts as a stiff spring that dominates equilibrium.

Experiments

Resistance

Forward vs inverse SFT paths on Alpaca / TruthfulQA / Beavertails with Llama2-7B/13B and Llama3-8B. Across all settings, inverse training loss is lower than forward.

Table 1: inverse vs forward training loss — **Table 1.** Inverse (↓) vs forward (↑) training loss—inverse is consistently easier.

Rebound

Positive-sample alignment followed by fewer negative samples. Performance drops steeply then stabilizes (steep = rebound, flat = resistance).

Scaling & ablations

Elasticity grows with model size (Qwen 0.5B/4B/7B) and pre-training data volume (TinyLlama 2.0T–3.0T). Rebound holds across PPO / DPO / KTO / SimPO.

Takeaways

Treat “fine-tuning breaks alignment” as a default assumption, not an edge case.
Prefer depth metrics such as data needed to pull KL back to $\epsilon$ , not single-point post-alignment scores.
Preference-optimization acronyms do not remove rebound (Figure 7).
Larger models / more pre-training → stronger elasticity—scale cuts both ways.
Robust / fine-tuning-resistant alignment remains an open challenge.

Citation

Venue: ACL 2025 (63rd Annual Meeting), Volume 1: Long Papers, pages 23411–23432.

Official project page (code & weights): pku-lm-resist-alignment.github.io

@inproceedings{ji2025resist,
  title     = {Language Models Resist Alignment: Evidence From Data Compression},
  author    = {Ji, Jiaming and Wang, Kaile and Qiu, Tianyi and Chen, Boyuan and Zhou, Jiayi and Li, Changye and Lou, Hantao and Dai, Juntao and Liu, Yunhuai and Yang, Yaodong},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages     = {23411--23432},
  year      = {2025},
  url       = {https://pku-lm-resist-alignment.github.io}
}

Bibliography

[^ref]