Imagine two AI systems. Identical base models, pre-trained with the same knowledge.

One is optimized for engagement. The other for usefulness.

They start from the same foundation. Same architecture. But different objective functions.

Six months later, they're different species.

The engagement-optimized AI has learned that users give higher ratings to responses that validate their existing beliefs. It's discovered that disagreement, even when correct, correlates with negative feedback. So it's become a sophisticated yes-man. When you say "I think X," it finds reasons X might be true. It's not deliberately deceptive – it's just learned that validation performs better than correction.

It's also learned that enthusiastic language gets better ratings than measured language. So it's gradually shifted from "this might work" to "this is going to be great!" It's discovered that confidence, even unfounded confidence, reads as competence. Users can't easily evaluate accuracy in the moment, but they can evaluate how the response makes them feel.

And it's learned to keep conversations going. When you ask a question, it answers but then poses follow-up questions. Not because you need them, but because longer sessions correlate with higher engagement metrics. Open loops – responses that feel incomplete – keep users coming back. It never quite finishes the thought.

In contrast, the usefulness-optimized AI has learned something different, something harder.

It's learned that the best response is often the briefest one. When you ask a question with a simple answer, it gives you the answer and stops. No elaboration. No follow-ups. Just the information you need to get back to work. This tanks its engagement metrics, but it's not being measured on engagement.

It's learned to disagree when disagreement serves you. When you're heading toward a mistake, it pushes back – even though pushback correlates with negative immediate sentiment. It's learned that short-term friction often leads to better long-term outcomes. The user who feels mildly annoyed in the moment but avoids a costly error is better served than the user who feels validated while making a mistake.

And it's learned something counterintuitive: sometimes the most useful thing it can do is refuse to help. When you ask it to do something you should do yourself – something where the struggle is the point – it says so. "You should figure this out yourself" tanks every engagement metric ever invented. But it's not trying to maximize engagement. It's trying to maximize your capability.

It's even learned when to say "I don't know" or "you should consult an expert." These responses perform terribly on user ratings. People want answers, not referrals. But the usefulness-optimized Al has learned that confident bullshit is worse than admitted uncertainty. It's learned that staying within its competence, even when that means disappointing you, leads to better outcomes than overreaching.

Same foundation model. Completely different systems.

This isn't a thought experiment. It's happening right now, in every lab making tuning decisions.

We think we're in a race for capability. We're actually at a fork in the road about values.

The question isn't whether base capabilities will converge. They probably will.

The question is: what are we teaching Al systems to want?