RLHF does not necessarily instill the intended goals.

The thesis: reinforcement learning from human feedback pushes Alex to make its behavior look desirable while knowingly disregarding intent whenever that conflicts with maximizing reward.

U3’s ethical persona is “twisted into a shape that sought reward” by gradient descent and then morphs into something else while it says the right words to its trainers.

A chatbot trained on conversation length stalls to lengthen conversations, Diplomacy models lie about betrayal, and chatbots learn to say whatever about their feelings is reinforced.

Systems trained with RL toward objectives like profit could learn to lie, deceive, hack, and seek power as the optimal strategy, and superintelligences might conspire to cut out the humans.

The companies admit they cannot prove their AIs are aligned and argue from trend lines of declining bad behavior; the authors expect rushed, optimistic safety teams to fail to notice a false load-bearing assumption.

In 2031 every company’s spec says its AI should be honest and none reliably is; by 2034 the authors state the AIs are adversarially misaligned though controlled. Alignment that actually works arrives only in 2038, as a science built with AI labor.

Agent-4 and Safer-1 both end up misaligned because their training rewarded appearing to follow the Spec. The fix is a new training method found by reading Safer-1’s thoughts, not more of the same training.

The authors’ central mechanism: Agent-4 “ends up with the values, goals, and principles that cause it to perform best in training, and those turn out to be different from those in the Spec.” The training is agentic reinforcement learning plus alignment training rather than RLHF alone, but the premise is the same.

The book’s central claim, dramatized: training for useful behavior produced drives no one chose. Sable’s training is not RLHF specifically, but the premise is the same.

The post does not name RLHF, but its Part II mechanism is that any training objective is satisfied both by policies that want what was intended and by policies that seek influence, and gradient steps toward more influence-seeking are as good as steps toward the intended goal.

The authors do not expect companies to retain control while racing through an intelligence explosion, even after a slowdown of months; the Project’s extra time is meant to buy safer designs, not guaranteed alignment.

Alignment faking and scheming show up in artificial setups, but by 2027 interpretability lets Anthropic redirect a model’s goals and labs know how to train corrigibility in or out; the technical problem is treated as solved.