Understanding Reinforcement Learning — A Primer

Author(s): Ayo Akinkugbe Originally published on Towards AI. Understanding Reinforcement Learning — A Primer Photo by Girl with red hat on Unsplash Introduction: Learning by Trial and Error Imagine teaching a dog to fetch a ball. You don’t hand the dog a manual titled “The Complete Guide to Ball Retrieval.” Instead, you throw the ball, and when the dog brings it back, you give it a treat. When the dog gets distracted and wanders off, you withhold the treat. Over dozens of repetitions, the dog learns that bringing the ball back leads to rewards, while ignoring the ball doesn’t. This process of learning through interaction, experimentation, and feedback is exactly what reinforcement learning does for artificial intelligence. Teaching a dog to fetch a ball A Different Type of Learning : Supervised, Unsupervised, Reinforced Reinforcement learning is fundamentally different from the other types of machine learning you might be familiar with. In supervised learning, we show the algorithm thousands of examples with correct answers, like showing a child flashcards where one side has a picture of an apple and the other side has the word “apple.” In unsupervised learning, we give the algorithm data without answers and ask it to find patterns, like asking someone to organize a messy drawer without telling them how. But in reinforcement learning, we do something more interesting: we place an agent in an environment, give it a goal, and let it figure out how to achieve that goal through experimentation. The agent doesn’t know the right answer in advance. It doesn’t have a dataset of correct moves to learn from. Instead, it takes actions, observes what happens, receives rewards or penalties, and gradually learns which actions tend to lead to good outcomes and which ones don’t. This is how DeepMind’s AlphaGo learned to beat world champions at Go, how robotic arms learn to grasp objects, and how autonomous vehicles learn to navigate roads. The agent learns by doing, making mistakes, and slowly improving its strategy based on the consequences of its actions. “In reinforcement learning, the agent doesn’t know the right answer in advance. It doesn’t have a dataset of correct moves to learn from. Instead, it takes actions, observes what happens, receives rewards or penalties and gradually learns which actions tend to lead to good outcomes and which ones don’t.” The Core Components of Reinforcement Learning At the heart of every reinforcement learning problem are 5 fundamental components that work together in a continuous loop. Understanding each of these components and how they interact is essential to grasping how reinforcement learning actually works. Agent The agent is the learner or decision-maker. In our dog example, the dog is the agent. In a video game, the agent might be the character you control. In a self-driving car, the agent is the AI system making decisions about steering, acceleration, and braking. The agent exists to make decisions, and its entire purpose is to learn which decisions lead to the best outcomes. The agent doesn’t start out knowing anything; it begins with a blank slate and learns entirely from experience. Environment The environment is everything the agent interacts with. It’s the world in which the agent operates. For the dog, the environment includes the room, the ball, you as the trainer, and all the physical laws that govern how balls bounce and roll. For a chess-playing agent, the environment is the chessboard and the rules of chess. For a trading algorithm, the environment is the stock market with all its complexity, volatility, and rules. The environment responds to the agent’s actions and provides feedback. It’s important to note that the agent doesn’t control the environment; it can only influence it through its actions. “The agent doesn’t control the environment; it can only influence it through its actions.” State A state represents a specific situation or configuration of the environment at a particular moment in time. When you’re teaching the dog to fetch, one state might be “ball has just been thrown and is in the air,” another state might be “ball has landed fifteen feet away,” and another might be “dog has ball in mouth and is five feet from owner.” States capture all the relevant information the agent needs to make a decision. In a video game, the state might include the positions of all characters, their health levels, available items, and the current score. The quality of the state representation is key: if you don’t include important information in your state, the agent won’t be able to make good decisions.\ “A state represents a specific situation or configuration of the environment at a particular moment in time.” Action An action is something the agent can do to interact with the environment. Actions are the agent’s way of influencing its world. For the dog, actions might include “run toward ball,” “pick up ball,” “run toward owner,” or “lie down and take a nap.” For a chess agent, actions are the legal moves available given the current board position. For a robot learning to walk, actions are the specific motor commands sent to each joint and actuator. The set of available actions can change depending on the current state. In chess, the legal moves change with every move made. In the fetch example, the dog can’t pick up the ball if the ball isn’t within reach. “An action is an agent interacting or influencing the environment” Reward The reward is the feedback signal that tells the agent whether its action was good or bad. Rewards are numbers: positive numbers for good outcomes and negative numbers (penalties) for bad outcomes. When the dog brings the ball back, it gets a positive reward (the treat, which we might represent as +10). When it ignores the ball, it gets zero or even a small negative reward (no treat, perhaps represented as -1 or 0). The reward is the only way the environment communicates value to the agent. The agent’s entire learning process is driven by a single objective: maximize […]