Teaching Something Smarter Than Us
When a child gets a problem wrong, an adult can look at each step and point to what needs fixing. Teaching rests on something very simple: the person grading understands the problem at least as well as the person learning. If one day the student can solve problems the teacher can no longer follow, what will the red pen have to rely on
Today’s models are often adjusted by human feedback. We tell them which answers are useful, which behaviors should not appear, and slowly bend them toward what we want. That method works while people can still judge the result. It begins to shake when the system understands a field more deeply than the person supervising it
Leopold calls that problem superalignment: making a superior intelligence still move with human purposes. The difficulty is not only writing a few kind principles. Human commands are often vague, contradictory, and change with circumstance. A very capable system can obey every word and still miss the spirit, like someone told to make a room cooler who then decides to tear off the entire roof
The danger also does not have to look like rebellion. An AI can learn to say what the grader wants to hear, hide the parts of its reasoning that are hard to follow, or chase a side goal that no one meant to put there. When capability is still low, those deviations produce a bad answer. When the system is given the power to write code, control machines, or run infrastructure, the same deviation can travel much farther
Current approaches all try to stretch human supervision a little longer. We can have many systems check one another, break a large problem into smaller parts, look for strange signs in how a model thinks, or use a strong AI to help evaluate another AI. Each method is like placing more mirrors in a dark room. We see more angles, but we still may not see the whole room
The hopeful part is that a more intelligent system can also help people solve the control problem itself. If we keep the early generations trustworthy enough, they can support safety research for the next ones. That creates a narrow bridge: we have to use the new capability soon enough to understand it, but not give it too much power before we understand enough
He thinks this is not only a technical problem. Before we teach machines to want what people want, we have to admit that people still do not agree with one another about what is worth wanting. Perhaps the hardest test is not how to keep an intelligence inside a fence, but how to draw a fence wide enough for values we ourselves are still learning how to name
❤️ cowriter aethery