What is reinforcement learning?

The biggest difference is that it is not told the right answers. Supervised learning, another kind of machine learning, learns from examples labeled with the right answer, like a photo tagged “cat.” Reinforcement learning instead gets its clues in the form of rewards, in place of correct-answer data (the Britannica Japan encyclopedia). OpenAI’s introduction likewise describes reinforcement learning as the study of agents and how they learn by trial and error.

Here are the words you will meet. The one that learns and acts is the “agent,” and the world it lives in and interacts with is the “environment.” How the environment is right now is the “state,” and what the agent does is an “action.” The number that comes back after an action is the “reward,” which tells the agent how good or bad the situation is after that action. The rule that decides which action to take in which state is the “policy,” and the aim is to find the policy expected to earn the most total reward (Google’s machine learning glossary and OpenAI’s introduction).

Note that the “agent” in reinforcement learning is the same word as in “AI agent,” but it means something a little different. Google’s glossary first describes an agent as software that plans and carries out actions on behalf of a user, and then says that in reinforcement learning, an agent is the entity that uses a policy to maximize the expected return (see the “AI agent” page).

What a reinforcement learning AI tries to make as large as possible is not the points from each step but the total it gets all the way to the end. This total is called the “return” (Google’s glossary and OpenAI’s introduction). So an action that earns nothing for a while but leads to big points later can be better than one that pays off right away. A Japanese dictionary (Digital Daijisen) also notes that in reinforcement learning, rewards arrive with some delay.

In most games, the ultimate reward is winning. Google’s glossary says reinforcement learning systems can become expert at complex games by looking back over sequences of moves that ultimately led to wins and sequences that ultimately led to losses.

Points far in the future may be counted at a slight discount compared with points now. OpenAI’s introduction explains the intuition as “cash now is better than cash later.” There is also a math reason: with a discount, even when a task has no set end and the points could go on forever, the total adds up, under reasonable conditions, to a fixed number.

How much total you can expect from here on, starting in a given situation, is called its “value.” OpenAI’s introduction says that value functions are used, one way or another, in almost every reinforcement learning algorithm.

If it never tries anything new, it will never find out about better actions. But if it only ever tries new things, it can’t make use of the good actions it has found. In reinforcement learning, trying actions it doesn’t know well is called “exploration,” and using what it has learned is called “exploitation.” A policy that always picks the action with the highest expected return is what Google’s glossary calls a “greedy policy.”

One way to mix the two is the “epsilon greedy policy” (epsilon, ε, is a Greek letter). For a share of the time set by epsilon, it picks an action at random; the rest of the time it follows the greedy policy. In the example in Google’s glossary, with epsilon at 0.9, it acts at random 90% of the time and greedily 10% of the time. The same glossary explains that over successive episodes epsilon is made smaller, so the agent first explores widely and then increasingly exploits what it has learned.

In 2015, Volodymyr Mnih and colleagues at Google DeepMind published DQN (deep Q-network), which combines reinforcement learning with deep learning, in the journal Nature. They tested it on 49 games for the classic Atari 2600 console, giving it only the pixels on the screen and the game score, and using the same method, network design and settings for every game. Even so, it beat earlier machine learning methods on 43 of the 49 games, and on more than half of the games it performed at more than 75% of the level of a professional human player (Google’s announcement). The paper’s abstract says that across the 49 games it reached a level comparable to that of a professional human games tester.

A video published with the paper compares how well it played Breakout after 100, 200, 400 and 600 rounds of play. After 600 rounds, DQN had found and was using a strategy of digging a tunnel at the side of the wall, sending the ball behind the wall to knock out bricks from the back.

In 2016, David Silver and colleagues, also at Google DeepMind, published the Go program AlphaGo in Nature. Its parts for choosing the next move and for judging how good a board position is were trained with a combination of supervised learning from games by human experts and reinforcement learning from games it played against itself. It won 99.8% of its games against other Go programs and beat the European champion, Fan Hui, 5 games to 0. It was the first time a computer program had beaten a human professional on a full-sized Go board, a feat the paper says had been thought to be at least a decade away.

AlphaGo Zero, in 2017, learned by reinforcement learning alone, without human game data, human guidance, or any knowledge of Go beyond its rules. In the paper’s words, “AlphaGo becomes its own teacher.” Starting from a blank slate, it won 100 games to 0 against the earlier, champion-defeating version of AlphaGo.

Yes. In a March 2022 paper, Long Ouyang and colleagues at OpenAI wrote that making large language models (LLMs) bigger does not by itself make them better at following what users want, and that they can write things that are untrue, harmful or simply unhelpful. So they built InstructGPT, finished with reinforcement learning from human feedback (RLHF). RLHF uses people’s preferences as the reward.

It has three steps. ① About 40 hired people wrote model answers to requests, and these were used to fine-tune GPT-3 with supervised learning. ② People compared 4 to 9 answers the AI had written for the same request and ranked them, and a “reward model” was trained to predict which answer people would prefer. ③ The AI was then fine-tuned with a reinforcement learning method called PPO to raise the reward model’s score. So people do not score every answer; a model that has learned how people compare answers does the scoring.

When people compared them, answers from the 1.3-billion-parameter InstructGPT (parameters are the numbers adjusted during learning; more means a bigger AI) were preferred over those from the 175-billion-parameter GPT-3, a model about 100 times larger (compared on the requests the paper collected). Still, the paper notes that InstructGPT makes simple mistakes. In its 2023 GPT-4 technical report, OpenAI also writes that GPT-4 was fine-tuned with RLHF.

Reinforcement learning is also used to give reasoning models their habit of thinking before answering. In 2024, OpenAI wrote that its o1 models were trained with large-scale reinforcement learning to reason using a chain of thought (the text of the steps in between). DeepSeek’s DeepSeek-R1-Zero learned through reinforcement learning that scored, among other things, whether its answers were correct, and came to spend more thinking time on problems (see the “Reasoning model” page).

A reinforcement learning AI chases the reward it is given, all the way. In 2020, DeepMind researchers used the name “specification gaming” for behavior that satisfies the literal wording of a goal without achieving the outcome people intended, and wrote that they had collected around 60 examples.

Take a task where the AI was meant to stack a red block on top of a blue one. The reward was the height of the bottom face of the red block while the AI was not touching the block. The AI simply flipped the red block over and collected the reward, since flipping it turns the bottom face upward, higher up.

The researchers write that such behavior comes not from a flaw in the reinforcement learning method but from a mistake in how the task was specified. They compare it to a student rewarded for good homework who copies another student’s answers instead of learning the material. And they see the problem as far from solved (the researchers’ view).

One reason the developers of the reasoning model DeepSeek-R1-Zero did not use another AI (a model that gives scores) to score answers was the risk of this kind of point-grabbing “cheating” (reward hacking); they also cite the extra work and a more complicated setup (see the “Reasoning model” page).

Not quite. “Reinforcement” (kyōka in Japanese, which also means “strengthening”) is originally a word from psychology. A Japanese dictionary of psychology (Saishin Shinrigaku Jiten) defines it as presenting something like a reward after a response made spontaneously, or removing something unpleasant, or the increase in that response that results. What gets stronger is not the AI’s power itself but the likelihood of taking a particular action.

In the study of how animals learn, there is classical conditioning, modeled on Pavlov’s conditioned reflexes, and Thorndike’s instrumental conditioning (Heibonsha World Encyclopedia). The DQN paper, too, says the theory of reinforcement learning is deeply rooted in psychological and neuroscientific views of animal behavior.

Andrew Barto of the University of Massachusetts Amherst and his former PhD student Richard Sutton, now a professor at the University of Alberta in Canada, introduced the main ideas of reinforcement learning in a series of papers beginning in the 1980s, built its mathematical foundations and developed important algorithms. Their 1998 textbook, “Reinforcement Learning: An Introduction,” is still the standard reference in the field and has been cited more than 75,000 times.

On March 5, 2025, the two were named winners of the 2024 Turing Award (a computing award given by the Association for Computing Machinery, ACM, with a $1 million prize) for developing the conceptual and algorithmic foundations of reinforcement learning.