Alignment

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Notes

Bypasses reward modeling entirely. Simpler alignment, same results.

Training language models to follow instructions with human feedback (InstructGPT)

Notes

RLHF applied to GPT-3. The bridge from raw LM to useful assistant.

Human Compatible

The Alignment Problem

Instrumental Goals and Hidden Codes in RLHF'd Language Models

How RLHF-trained language models may develop instrumental goals, and the information-theoretic limits on detecting them.

Instrumental Goals and Latent Codes in Reinforcement Learning Fine-tuned Language Models: An Alignment Perspective