Language Models Resist Alignment: Evidence From Data Compression

Jiaming Ji^*1, Kaile Wang^*1, Tianyi Qiu^*1, Boyuan Chen^*1, Jiayi Zhou^*1, Changye Li¹, Hantao Lou¹, Juntao Dai¹, Yunhuai Liu¹, Yaodong Yang^†1

¹Institute for Artificial Intelligence, Peking University

*Equal contribution. †Corresponding author.

📌 ACL 2025 (Volume 1: Long Papers), pages 23411–23432

📜 Official project: https://pku-lm-resist-alignment.github.io

PDF path note: local ACL 2025 paper Language Models Resist Alignment (llms-resist-alignment.pdf)

📄 Paper 📜 arXiv 🌐 Project 💻 Code 📜 BibTeX

Abstract

Large language models (LLMs) may exhibit unintended or undesirable behaviors. Recent works have concentrated on aligning LLMs to mitigate harmful outputs. Despite these efforts, some anomalies indicate that even a well-conducted alignment process can be easily circumvented, whether intentionally or accidentally. Does alignment fine-tuning yield have robust effects on models, or are its impacts merely superficial? In this work, we make the first exploration of this phenomenon from both theoretical and empirical perspectives. Empirically, we demonstrate the elasticity of post-alignment models, i.e., the tendency to revert to the behavior distribution formed during the pre-training phase upon further fine-tuning. Leveraging compression theory, we formally deduce that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude. We validate the presence of elasticity through experiments on models of varying types and scales. Specifically, we find that model performance declines rapidly before reverting to the pre-training distribution, after which the rate of decline drops significantly. Furthermore, we further reveal that elasticity positively correlates with the increased model size and the expansion of pre-training data. Our findings underscore the need to address the inherent elasticity of LLMs to mitigate their resistance to alignment.

TL;DR: We demonstrate the elasticity of post-alignment models, forming resistance to alignment.

Introduction

Training-based alignment (SFT, RLHF) aims to reject harmful distributions, yet highly aligned models can be undone with minimal fine-tuning. We ask: Why is alignment so fragile? We introduce alignment elasticity — resistance plus rebound — and show inverse alignment can revoke alignment with far less data.

Phenomenon: elasticity = resistance + rebound; Δγ inversely proportional to |Di|
Mechanism: compression theorem modeling of training/alignment
Validation: resistance/rebound across models, scales, and algorithms

Methodology

We treat the LLM as a lossless compression protocol. Normalized compression rate serves as a proxy for alignment/pretraining loss. Theorem 4.2 proves elasticity under Pareto mass assumptions: alignment-side rate changes ~Θ(k) times pretraining-side with k=|Dp|/|Da|≫1.

Inverse Alignment & Elasticity

Inverse alignment uses Db ≪ Da to pull an aligned model back toward the pretrained distribution.

Spring Analogy

|Di| ↔ spring stiffness; Δγ ↔ deformation. Large pretraining is a hard spring that dominates equilibrium after perturbation.

Figure 1: Elasticity of language models — spring analogy.

Figure 2: Resistance experiment pipeline.

Rebound pipeline — Figure 3: Rebound experiment pipeline.

Results

Resistance: inverse loss lower than forward on Alpaca/TruthfulQA/Beavertails (Llama2-7B/13B, Llama3-8B). Rebound: more positive data → faster performance drop under negative fine-tuning. Scaling: larger models and more pretraining amplify elasticity. Ablation: rebound holds for PPO/DPO/KTO/SimPO.

Table 1: inverse vs forward training loss across all settings
Figure 4: rebound curves (steep then plateau)
Figures 5–6: model size and pretrain volume scaling
Figure 7: algorithm ablation

Image Gallery

Figure 1: Elasticity of language models — spring analogy.

Figure 2: Resistance experiment pipeline.

Figure 3: Rebound experiment pipeline.

Figure 4: Rebound results.

Figure 5: Model-size scaling of rebound.

Figure 6: Pretraining-data scaling of rebound.

Figure 7: Algorithm ablation (PPO/DPO/KTO/SimPO).

Table 1: Inverse vs forward training loss.

Scale model — Figure 5: Model-size scaling of rebound.

Scale pretrain — Figure 6: Pretraining-data scaling of rebound.

Ablation — Figure 7: Algorithm ablation (PPO/DPO/KTO/SimPO).

Table 1: Inverse vs forward training loss.

Conclusion

Language models have elasticity and thus naturally resist alignment. Addressing this inherent elasticity is necessary for robust, deep alignment.

BibTeX

@inproceedings{ji2025language,
  title={Language Models Resist Alignment: Evidence From Data Compression},
  author={Ji, Jiaming and Wang, Kaile and Qiu, Tianyi and Chen, Boyuan and Zhou, Jiayi and Li, Changye and Lou, Hantao and Dai, Juntao and Liu, Yunhuai and Yang, Yaodong},
  booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={23411--23432},
  year={2025}
}

Acknowledgement

* Equal contribution. † Corresponding author. Official project and code: https://pku-lm-resist-alignment.github.io.