Language Models Resist Alignment: Evidence From Data Compression

Jiaming Ji*, Kaile Wang*, Tianyi Qiu*, Boyuan Chen*, Jiayi Zhou*, Changye Li, Hantao Lou, Juntao Dai, Yunhuai Liu, Yaodong Yang†
Institute for Artificial Intelligence, Peking University
ACL 2025 (Volume 1: Long Papers), pages 23411–23432
*Equal contribution. †Corresponding author.
Paper Project Code arXiv

PDF path note: local ACL 2025 paper Language Models Resist Alignment (llms-resist-alignment.pdf)

Elasticity teaser

TL;DR: We demonstrate the elasticity of post-alignment models, forming resistance to alignment.

Abstract

Large language models (LLMs) may exhibit unintended or undesirable behaviors. Recent works have concentrated on aligning LLMs to mitigate harmful outputs. Despite these efforts, some anomalies indicate that even a well-conducted alignment process can be easily circumvented, whether intentionally or accidentally. Does alignment fine-tuning yield have robust effects on models, or are its impacts merely superficial? In this work, we make the first exploration of this phenomenon from both theoretical and empirical perspectives. Empirically, we demonstrate the elasticity of post-alignment models, i.e., the tendency to revert to the behavior distribution formed during the pre-training phase upon further fine-tuning. Leveraging compression theory, we formally deduce that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude. We validate the presence of elasticity through experiments on models of varying types and scales. Specifically, we find that model performance declines rapidly before reverting to the pre-training distribution, after which the rate of decline drops significantly. Furthermore, we further reveal that elasticity positively correlates with the increased model size and the expansion of pre-training data. Our findings underscore the need to address the inherent elasticity of LLMs to mitigate their resistance to alignment.

Motivation

Training-based alignment (SFT, RLHF and variants) is the dominant approach to reject harmful distributions. Yet highly safety-aligned models can become unsafe again with minimal fine-tuning — even on non-malicious data. We ask: Why is alignment so fragile?

We attribute this to alignment elasticity = resistance (pretrained models retain their original distribution) + rebound (deeper alignment → faster return under reverse fine-tuning). Inverse alignment with far less data (Db ≪ Da) can undo alignment.

Method & Theory

We model LLMs as lossless compressors. Compression rate γ proxies training/alignment loss. Theorem 4.2 (Elasticity) shows that under perturbation, alignment-side normalized compression rate changes at about Θ(k) times the pretraining side, where k = |Dp| / |Da| ≫ 1 — so alignment can be undermined far more easily than pretraining.

Spring analogy: dataset size ↔ stiffness; Δγ ↔ deformation. Large pretraining acts as a hard spring dominating the equilibrium.

Elasticity
Figure 1: Elasticity of language models — spring analogy linking dataset size to compression-rate change.

Results Gallery

Experiments

  • Resistance: Inverse alignment consistently easier than forward (lower loss/KL) on Alpaca, TruthfulQA, Beavertails across Llama2-7B/13B and Llama3-8B.
  • Rebound: More positive data → steeper drop under negative fine-tuning (IMDb, Beavertails; Gemma-2B, Llama2-7B).
  • Scaling: Elasticity grows with model size (Qwen) and pretraining tokens (TinyLlama).
  • Algorithms: Rebound persists under PPO, DPO, KTO, SimPO.
Figure 2: Resistance experiment pipeline (forward vs inverse alignment).
Figure 2: Resistance experiment pipeline (forward vs inverse alignment).
Figure 3: Rebound experiment pipeline (positive alignment then negative fine-tuning).
Figure 3: Rebound experiment pipeline (positive alignment then negative fine-tuning).
Figure 4: Rebound results — deeper positive alignment → faster drop under negative fine-tuning.
Figure 4: Rebound results — deeper positive alignment → faster drop under negative fine-tuning.
Figure 5: Larger models exhibit stronger rebound (Qwen 0.5B / 4B / 7B).
Figure 5: Larger models exhibit stronger rebound (Qwen 0.5B / 4B / 7B).
Figure 6: More pre-training data amplifies rebound (TinyLlama 2.0T / 2.5T / 3.0T).
Figure 6: More pre-training data amplifies rebound (TinyLlama 2.0T / 2.5T / 3.0T).
Figure 7: Rebound holds across PPO / DPO / KTO / SimPO.
Figure 7: Rebound holds across PPO / DPO / KTO / SimPO.
Table 1: Inverse alignment loss consistently lower than forward across models and datasets.
Table 1: Inverse alignment loss consistently lower than forward across models and datasets.

BibTeX

@inproceedings{ji2025language,
  title={Language Models Resist Alignment: Evidence From Data Compression},
  author={Ji, Jiaming and Wang, Kaile and Qiu, Tianyi and Chen, Boyuan and Zhou, Jiayi and Li, Changye and Lou, Hantao and Dai, Juntao and Liu, Yunhuai and Yang, Yaodong},
  booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={23411--23432},
  year={2025}
}