The Alignment Problem & Preference Optimization

The Alignment Triad: Helpful, Honest, Harmless (HHH)

A raw base LLM trained only on internet text predicts the most probable continuation—which may be toxic, biased, or untruthful. Alignment steers models to follow user intent safely.

---

Evolution of Alignment Methods:

1. SFT (Supervised Fine-Tuning): Training on curated (Instruction, Ideal Answer) pairs.
2. RLHF (Reinforcement Learning from Human Feedback):
   • Humans rank multiple model responses (y_win > y_lose).
   • A Reward Model is trained to score responses.
   • PPO (Proximal Policy Optimization) optimizes the LLM policy against the reward model with a KL-divergence penalty.
3. DPO (Direct Preference Optimization - Rafailov et al., 2023):
   • Derives a closed-form mathematical substitution that optimizes the policy directly on preference pairs without needing a separate reward model or complex reinforcement learning loops!

DPO Loss Formulation:
L_DPO(pi_theta; pi_ref) = -E_{(x, y_w, y_l)} [ log sigma( beta * log( pi_theta(y_w|x) / pi_ref(y_w|x) ) - beta * log( pi_theta(y_l|x) / pi_ref(y_l|x) ) ) ]

Where:
- x: Input prompt
- y_w: Winning (preferred) response
- y_l: Losing (rejected) response
- pi_ref: Frozen reference policy model
- beta: Regularization parameter controlling deviation from reference