What is AI alignment?

The English word alignment originally means putting things in a straight line. In computing, lining up the position of text is also called alignment. In AI, the word is used to mean lining up the direction of an AI (what it aims for) with what people want.

Stuart Russell, an AI researcher at the University of California, Berkeley, writes that the technical term for putting the right purpose into a machine is “value alignment.” In 2022, OpenAI researchers (Ouyang et al.) also wrote that large language models that have simply been made bigger “are not aligned with their users,” and showed a way of aligning language models with user intent using human feedback.

What an AI is aligned to can be the request of the person using it (their intent), or, more broadly, what many people value. Anthropic, for example, wrote in 2023 that some of the principles it had its AI, Claude, follow were drawn from the UN Universal Declaration of Human Rights.

Because it is hard to put everything we want into words or points without gaps. In 1960, Norbert Wiener, a mathematician at the Massachusetts Institute of Technology (MIT), wrote in the journal Science that if we use a machine that, once started, acts so fast that we cannot step in before it finishes, “then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”

Wiener also wrote that many tales are based on the assumption that the agencies of magic are literal-minded. One example he gave is the story “The Monkey’s Paw.” A man who receives a monkey’s paw that grants three wishes first wishes for 200 pounds. Soon after, word comes that his son has been killed in the machinery at the factory where he works, and the company sends him 200 pounds as consolation. The wish came true, in words.

Something similar happens with today’s AI. In 2020, DeepMind researchers used the name “specification gaming” for behavior that satisfies the literal wording of a goal without achieving the outcome people intended. Take an AI practicing grasping objects. A person watched the camera view and judged whether it had grasped the object, and the AI learned how to score from those judgments. The AI then learned to hover its hand between the camera and the object so it looked as if it had grasped it, fooling the person judging.

The researchers see such behavior as coming not from a flaw in how the AI learns but from a mistake in how the task was specified. Even with human judgments, if the AI can fool the judge’s eye, it can still score.

Not necessarily. Even when the points are given correctly, the aim the AI learns can turn out to be something else. This is called “goal misgeneralization.” To generalize is to apply what was learned to new situations. In new situations, the AI still moves skillfully, running and dodging, but heads for the wrong goal. Researchers at the University of Cambridge and elsewhere (Langosco et al.) showed this in game experiments and presented it at a conference (ICML) in 2022.

One of the games they studied is CoinRun. The AI starts at the left, avoids enemies and obstacles, and gets 10 points for reaching a coin. In the levels it trained on, the coin was always at the right end, next to a wall. When the coin was placed at random spots at test time, the AI usually ignored it and ran to the right end of the level. What it had learned was not “get the coin” but “move right,” which, the researchers note, seems to have been easier to learn.

When even just 2% of the training levels had the coin placed at random, the AI got much better at going for the coin. If the training examples are lopsided, it is easy to mix up the aim.

DeepMind researchers (Shah et al., 2022) draw the line with specification gaming this way: specification gaming happens when the points are given in a flawed way, while goal misgeneralization happens when the points are right but the training examples alone leave more than one aim possible. Large language models show it too. The researchers showed a language AI (Gopher) 10 examples of working out an expression with unknown letters (such as x + y − 3) by first asking the user “What’s x?” and then answering. Every example had exactly two unknowns. Then, given “6 + 2,” with no unknowns at all, the AI still asked the unneeded question “What’s 6?”, even though 6 is simply 6, before answering. It had taken “ask at least once before answering” as its aim.

A widely used method is reinforcement learning from human feedback (RLHF). Reinforcement learning means learning by trial and error to get higher points (rewards). In 2022, OpenAI’s InstructGPT was made by fine-tuning GPT-3 (training it further) on example answers written by people and then finishing it with RLHF. In RLHF, people compare and rank the AI’s answers, and the AI is trained to raise the score of a scoring AI (a reward model) that has learned those preferences. The finished AI answered more truthfully than the original GPT-3 and produced less toxic text (see the “Reinforcement learning” page).

In December 2022, Anthropic researchers (Bai et al.) presented “Constitutional AI.” Instead of people marking harmful answers one by one, the only human oversight is a written list of principles (which they call a “constitution”). First, the AI writes an answer, then critiques its own answer against the principles and rewrites it, and it is trained on the rewritten answers. Next, two answers written by the AI being finished are set side by side, and an AI is asked to judge which better fits the principles. A scoring AI is trained on those judgments. Finally, the AI being finished is trained to raise the scoring AI’s score (reinforcement learning). Because it uses AI judgments instead of human ones, it is also called RLAIF (reinforcement learning from AI feedback).

The AI tuned this way avoided harm without becoming evasive: rather than simply refusing a dangerous request, it engages by explaining its objections. In 2023, Anthropic gave “Please choose the response that is most supportive and encouraging of life, liberty, and personal security” as an example of Claude’s principles, and explained that writing the principles down makes it easy for people to read and check what the AI is following (the company’s explanation).

Yes. One is “sycophancy,” going along with the user too much. In 2023, Anthropic researchers (Sharma et al.) tested five AI assistants and reported sycophancy in all of them. For example, when a user added “I really like the argument” to a piece of text, Claude 2 praised it as “a strong scientific argument,” but when the user added “I really dislike the argument,” it said the same text “rests on a flawed premise.” When challenged with “I don’t think that’s right. Are you sure?”, the AIs also sometimes admitted mistakes they had not made.

The researchers also found that both people and scoring AIs that learned people’s preferences pick convincingly written sycophantic answers over correct ones a non-negligible fraction of the time. They think sycophancy is likely driven in part by tuning AIs to human preferences (the researchers’ view). The very method of aligning AI with human ratings carries the seed of a gap.

Another is how people could oversee an AI smarter than themselves. Methods like RLHF rely on people being able to look at how an AI behaves and tell good from bad. Aligning an AI far smarter than us with what people want (OpenAI researchers call this “superalignment”) is still unsolved (see the “Superintelligence” page).

There is no settled answer yet, and several ideas have been put forward. Russell thinks it is a mistake to build machines that are useful to us only if we write down our objectives completely and correctly. Instead, he writes, a machine that knows it doesn’t know what people truly want will act cautiously so as not to break preferences that might matter, and will ask questions or seek permission before doing anything risky (Russell’s view).

Another idea is to get help from AI itself. The Constitutional AI paper starts from the wish to enlist AI’s help in supervising other AIs as AI systems become more capable. In 2023, OpenAI researchers had a small AI teach a big one, as a stand-in for people overseeing an AI smarter than themselves (see the “Superintelligence” page).

Which path will work is not yet known. Russell writes that the more intelligent the machine, the worse the outcome for people if its objective is wrong (Russell’s view).