Language Models Resist Alignment: Evidence From Data Compression
🎯 LLMs exhibit alignment elasticity (resistance + rebound): alignment is shallow and compressible; inverse alignment with far less data can undo it — explained via compression / Theorem 4.2.
💡 对你的启发：Treat finetune-undoes-alignment as default; measure rebound depth (Db to restore base KL); larger models rebound harder.
展开详细解读
🧩 背景与动机
Training-based alignment (SFT, RLHF, variants) is mainstream, yet highly safety-aligned models can become unsafe again with minimal fine-tuning, including on non-malicious data. The authors ask: why is alignment so fragile?
🔬 方法
Frame LLMs as lossless compressors; define inverse alignment and elasticity (resistance + rebound); prove Theorem 4.2 under Pareto mass assumptions linking normalized compression-rate changes to dataset sizes (spring analogy).
📊 主要结果
Inverse SFT loss < forward across Llama2/3 × Alpaca/Truthful/Beavertails (Table 1). More positive data → steeper rebound under fewer negatives (Fig. 4; Table 3: 729 unsafe samples to restore KL after 10k safe vs 961 after 1k). Elasticity increases with model size and pretrain volume; holds for PPO/DPO/KTO/SimPO.
✨ 创新点
First systematic theory+experiments treating alignment fragility as elasticity from compression; unifies resistance and rebound under Δγ ∝ 1/|Di|.
💡 深度启发
Do not rely on preference-optimization acronyms alone; pressure-test open-weight aligned models for low-cost inverse alignment; track 'data to undo' as a depth metric.