Reinforcement Learning: Agents & Rewards

Imagine trying to learn to ride a bicycle from a book that lists the "correct instruction" for every possible wobble — lean 3 degrees left, apply 0.4 kg of pressure to the right pedal, and so on. No such book exists, and no one could write one. Instead, you get on the cycle, push off, wobble, maybe fall, feel your knee scrape the ground, get back on, wobble a little less, and after a few evenings of this you simply know how to balance. Nobody handed you the "right answer" for any single moment. You learned entirely from the consequences of what you tried.

This is a fundamentally different way of learning from the one you have probably already met in earlier machine learning topics, where a model is shown thousands of labelled examples — this email is "spam", this photo is "cat" — and learns to copy the given answer. That style of learning is called supervised learning, because a supervisor (the label) tells the model the correct output for every single example. Riding a bicycle, or learning to bat in cricket, or training a robot to walk, does not work that way. There is no labelled dataset of "correct" bicycle-balancing instructions. There is only trial, consequence, and adjustment. The branch of machine learning that studies exactly this kind of learning is called reinforcement learning (RL), and it is the third major paradigm of machine learning alongside supervised and unsupervised learning.

Meet the Vocabulary: Agent, Environment, State, Action, Reward

To talk about reinforcement learning precisely, we need five words. Let's build them from a situation every Indian student watching a Ranji Trophy or gully cricket match will recognise instantly: a batter facing an unfamiliar bowling attack, trying to maximise the runs scored in an innings.

• Agent — the decision-maker that is learning. In our example, the batter. In software, this is the program whose behaviour we are trying to improve.
• Environment — everything the agent interacts with and does not fully control: the bowler, the pitch, the fielding placement, the match situation. The agent acts on the environment, and the environment responds.
• State — a snapshot of the current situation the agent must decide from: which bowler is running in, how many overs are left, how many wickets are down, what the current score is. In RL notation we write the state at time step t as s(t).
• Action — a choice the agent makes from the current state: play a defensive block, drive through covers, go for a pull shot, leave the ball. Written a(t).
• Reward — a single number the environment sends back after the action, telling the agent how good or bad that action turned out to be right now. Runs scored is a positive reward; losing a wicket is a strongly negative reward. Written r(t+1), because it arrives just after the action is taken.

Notice something important that is easy to miss: the reward is not the same thing as a supervised-learning label. A label says "the correct answer here was X." A reward only says "here is a number describing how that turned out" — it never directly tells the agent which action would have been better. The agent has to work that out for itself, by trying different actions across many attempts and comparing the rewards that follow.

After the batter acts, two things happen simultaneously: the environment produces a reward for that action, and it also moves into a new state (one ball fewer, possibly a new score, possibly a new bowler about to start their next over). The agent then picks its next action from this new state, and the cycle continues. This repeating exchange is the heart of reinforcement learning, and it is worth drawing out precisely.

Every reinforcement learning problem, no matter how complex — a robot vacuum mapping a room, a chess engine, a self-driving car deciding when to brake — is built from exactly this loop repeated over and over. What differs between problems is only what counts as a state, what actions are available, and how reward is defined.

A Worked Example: The Six-Cell Corridor

Cricket is great for building intuition, but to actually do arithmetic we need something smaller. Picture a narrow corridor with six cells, numbered 0 to 5, laid out in a straight line. An agent starts at cell 0. At every step it can move Left or Right by one cell. Cell 5 is the goal. Every single move costs a reward of −1 (representing effort or time), except the move that lands the agent exactly on cell 5, which gives a reward of +10 and ends the episode (an episode is simply one complete attempt, from start to goal, after which a fresh attempt begins from cell 0 again).

This tiny world is small enough to trace by hand and complex enough to show every important idea in reinforcement learning: delayed reward, credit assignment, and the difference between an immediate reward and a genuinely good decision.

How Rewards Add Up: Return and the Discount Factor

A single reward tells you almost nothing on its own. What actually matters to the agent is the total reward it collects over an entire episode, called the return. If an agent starting at cell 3 goes Right, Right (reward −1, then +10), its return for that episode is −1 + 10 = 9.

Now, should a reward promised two steps into the future count exactly as much as a reward available right now? In most RL problems, the answer is no — the future is less certain than the present, so we shrink future rewards slightly using a number called the discount factor, written γ (gamma), which is always between 0 and 1. A reward that arrives one step from now is multiplied by γ, a reward two steps away is multiplied by γ², and so on. When γ is close to 1 (say 0.9 or 0.99), the agent plans almost as carefully for the distant future as for the present — useful when reaching a far-off goal matters a lot. When γ is close to 0, the agent becomes short-sighted, caring mostly about whatever reward is available immediately, even if a much bigger reward was one step further away. For our corridor example we will use γ = 0.9.

Turning Experience into a Policy: Q-Values

The agent's ultimate goal is to learn a policy — a rule for choosing the best action from every state it might find itself in. One of the most direct ways to build a policy is to keep a running estimate, for every (state, action) pair, of "how good is it, in total future return, to take this action from this state, and then act well afterward?" This estimate is called a Q-value, written Q(s, a), and the table holding all of them is called a Q-table.

We start every Q-value at 0, because the agent knows nothing yet. Every time the agent actually takes an action and observes what happens, it nudges that Q-value slightly toward a better estimate using the Q-learning update rule:

• Q(s, a) ← Q(s, a) + α × [ r + γ × max_a' Q(s', a') − Q(s, a) ]

This formula looks dense, so let's unpack every piece in plain words before using it. Q(s, a) on the left is the value we are about to update. On the right, r is the reward we actually just received. Q(s', a') is the current estimate for the best action available in the new state s' we landed in, and max_a' means we take the highest such value across all possible next actions — this is how information about the goal, several steps away, gets pulled backward into earlier states. α (alpha) is the learning rate, a number between 0 and 1 that controls how much each new experience is allowed to shift the old estimate; α = 0.5 means "move the estimate halfway toward what this new experience suggests."

Tracing the Math by Hand

Let's actually run this. We use α = 0.5 and γ = 0.9, and we start the agent at cell 3, always choosing Right for this trace so the arithmetic stays simple.

Episode 1, first move: state = 3, action = Right, next state = 4, reward = −1 (cell 4 is not the goal). Since every Q-value starts at 0, max_a' Q(4, a') = 0. Applying the formula:

• Q(3, Right) ← 0 + 0.5 × [ −1 + 0.9 × 0 − 0 ] = 0.5 × (−1) = −0.5

Notice what just happened: the very first time the agent tries the correct move toward the goal, its Q-value estimate goes negative. That is not a bug — from where the agent stands after only one experience, all it has seen is a step that cost −1, with no idea yet that this path leads anywhere good. This is exactly why a single reward can be badly misleading, and why we need many episodes before the Q-table becomes trustworthy.

Episode 1, second move: state = 4, action = Right, next state = 5 (the goal), reward = +10. Because state 5 is terminal (the episode ends there, no further action follows), the future-value term is 0:

• Q(4, Right) ← 0 + 0.5 × [ 10 + 0.9 × 0 − 0 ] = 0.5 × 10 = 5.0

Now the agent restarts a fresh episode from cell 0 (in a real run it would wander through cells 0, 1 and 2 first, but let's fast-forward to when it again reaches cell 3, to see how the earlier mistake gets corrected).

Episode 2, at state 3 again: state = 3, action = Right, next state = 4, reward = −1. This time, max_a' Q(4, a') is no longer 0 — we just learned Q(4, Right) = 5.0, so the maximum over state 4's actions is 5.0:

• Q(3, Right) ← −0.5 + 0.5 × [ −1 + 0.9 × 5.0 − (−0.5) ]
• = −0.5 + 0.5 × [ −1 + 4.5 + 0.5 ]
• = −0.5 + 0.5 × 4.0 = −0.5 + 2.0 = 1.5

Q(3, Right) has jumped from −0.5 to 1.5 — still not its final value, but now correctly positive, correctly signalling "this is a good move." Nothing changed about cell 3 itself between episode 1 and episode 2. What changed is that the knowledge of the goal propagated one step backward through the max_a' term. This backward flow of information, from the reward at the goal toward the states that lead to it, is called credit assignment, and it is the single most important mechanical idea in this chapter: with enough repeated episodes, the +10 at cell 5 gradually "leaks" backward through cell 4, then cell 3, then cell 2, then cell 1, until even cell 0 has an accurate Q-value pointing the way to the goal, even though cell 0 is five steps away from ever seeing the +10 directly.

Explore or Exploit? The Batting Dilemma

Suppose after a few episodes the agent has learned that, from cell 3, Right looks good. Should it now always pick Right from cell 3, forever? If it does, it will never find out whether some other path might have been even better in situations it has not yet tried enough times, and it will never discover a mistake if its early, noisy estimates happened to be wrong. This is the exploration versus exploitation trade-off, and it shows up constantly outside RL as well: a batter who has scored well cutting the ball past point against a particular type of bowler will keep exploiting that shot, but a good batter also occasionally explores a different shot against a bowler they have not truly figured out yet, because always repeating the "currently known best" action means never discovering a better one.

A simple, widely used rule for balancing this is called epsilon-greedy: define a small probability ε (epsilon), such as 0.2. At every decision point, with probability ε the agent picks a completely random action (explore), and otherwise, with probability 1 − ε, it picks whatever action currently has the highest Q-value for that state (exploit, using the "greedy" choice). Early in training, when the Q-table is mostly unreliable, this occasional randomness is what lets the agent discover the corridor's goal at all; later, once the Q-values are trustworthy, the exploit branch dominates and the agent behaves close to optimally.

From Hand Trace to Code

The hand trace above did exactly two updates, with the agent always choosing Right on purpose. A real Q-learning program automates this over hundreds of episodes, using epsilon-greedy so the agent can also discover the goal starting from cells that were never manually walked through. Here is the corridor world and the Q-learning update rule, written using only lists, loops, conditionals and functions:

import random

NUM_STATES = 6
LEFT = 0
RIGHT = 1

# Q[state][action] holds our current value estimate.
# We build it one row at a time with a loop.
Q = []
for state in range(NUM_STATES):
    Q.append([0.0, 0.0])   # [value of Left, value of Right]

def best_action(state):
    # compare the two action values for this state directly
    if Q[state][LEFT] >= Q[state][RIGHT]:
        return LEFT
    else:
        return RIGHT

def take_step(state, action):
    if action == LEFT:
        next_state = max(0, state - 1)
    else:
        next_state = min(5, state + 1)

    if next_state == 5:
        reward = 10
    else:
        reward = -1

    return next_state, reward

alpha = 0.5     # learning rate
gamma = 0.9     # discount factor
epsilon = 0.2   # exploration probability

for episode in range(200):
    state = 0
    while state != 5:
        if random.random() < epsilon:
            action = random.choice([LEFT, RIGHT])   # explore
        else:
            action = best_action(state)              # exploit

        next_state, reward = take_step(state, action)

        if next_state == 5:
            future_value = 0   # episode ends, no action follows
        else:
            future_value = max(Q[next_state][LEFT], Q[next_state][RIGHT])

        old_value = Q[state][action]
        Q[state][action] = old_value + alpha * (reward + gamma * future_value - old_value)

        state = next_state

print(Q)

Trace the structure against what you already did by hand: Q[state][action] is exactly Q(s, a); future_value is exactly max_a' Q(s', a'); and the single assignment line inside the while loop is a direct, literal translation of the update formula, applied automatically after every step instead of by hand. Run this for 200 episodes and Q[3][RIGHT] will settle close to a stable number well above zero, and Q[0][RIGHT] will also become clearly positive and larger than Q[0][LEFT], even though state 0 is five moves from the goal — direct evidence of credit assignment working across the whole corridor, not just the one step we traced by hand.

Two Misconceptions Worth Correcting

Misconception 1: "The agent is told the correct action, like a label." This is wrong, and it is the single most common confusion between reinforcement learning and supervised learning. A labelled dataset says, for each example, "the correct output here was X." A reward never says this. Reward only reports a number describing how one particular action turned out; it never states what the best action would have been. The agent must run the same state through many different actions, across many episodes, and compare the resulting Q-values itself, the way we watched Q(3, Right) climb from −0.5 to 1.5 purely through repetition — nobody ever told the agent directly that Right was correct at cell 3.

Misconception 2: "The action with the best immediate reward is always the best action." Our own hand trace disproves this directly. After only one experience, Q(3, Right) was −0.5 — if the agent had judged purely by immediate reward, Right from cell 3 would have looked like a bad move, worse than doing nothing. It only looked negative because immediate reward alone ignores what happens afterward. The max_a' Q(s', a') term in the update rule exists specifically to fix this: it lets a small or even negative immediate reward still be recognised as part of a good overall plan, as long as it leads to states with high future value. This is exactly why reinforcement learning problems are described in terms of maximising total return over an episode, not the reward of any single step in isolation.

Where This Shows Up

The corridor and the cricket analogy are simplified on purpose, but the same loop — agent, environment, state, action, reward — scales up to genuinely hard real-world problems. The best-known milestone is DeepMind's AlphaGo, which in March 2016 defeated professional Go player Lee Sedol; AlphaGo was trained substantially through reinforcement learning, including playing enormous numbers of games against itself and updating its value estimates from the outcomes, conceptually the same backward flow of credit you traced by hand in the corridor, just at a vastly larger scale with millions of states instead of six. Robotics labs building walking or grasping robots use a similar approach: instead of hand-coding every joint movement, the robot is placed in a simulated environment with a reward for staying upright or successfully grasping an object, and it discovers a working policy through repeated trial, exactly the way the corridor agent discovered that Right was the way to the goal.

Summary

• Reinforcement learning is a distinct paradigm from supervised learning: instead of labelled correct answers, the agent only receives a scalar reward after each action, and must discover good behaviour through repeated trial and error.
• The core loop is agent → action → environment → new state and reward → agent, repeating at every time step.
• Return is the total reward collected over an episode; the discount factor γ controls how strongly future rewards are weighted against immediate ones.
• A Q-value Q(s, a) estimates the total future return of taking action a from state s and acting well afterward; the Q-learning update rule nudges this estimate toward the observed reward plus the discounted best value of whatever state comes next.
• Because of the max_a' Q(s', a') term, information about a distant goal propagates backward through the states that lead to it over repeated episodes — this is credit assignment, and it is why an action's true value can differ sharply from its immediate reward.
• Epsilon-greedy balances exploration (trying random actions to discover new information) against exploitation (using the current best-known action), which is essential because early Q-value estimates are unreliable.

Check Your Understanding

• In the corridor world, why did Q(3, Right) start out negative even though Right is the correct direction toward the goal? What specifically changed between episode 1 and episode 2 to fix this?
• If we had used γ = 0 instead of 0.9, recompute Q(3, Right) in episode 2. (Hint: the max_a' Q(4, a') term gets multiplied by 0, so only the immediate reward survives — what does that tell you about a short-sighted agent's behaviour in a world where the goal is several steps away?)
• Suppose epsilon-greedy used ε = 0. What would happen to an agent starting fresh, with every Q-value still at 0, when two actions from a state are exactly tied? Would it ever discover a better path if its very first random guess happened to be a poor one?
• Explain, in your own words, why a reward is not the same thing as a supervised-learning label, using the difference between "here is a number describing how that action turned out" and "here is the correct answer."
• A friend claims: "the agent should always pick whichever action gives the highest reward on the very next step." Using the corridor example, construct a short argument for why this claim is false in general.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where reinforcement learning: agents & rewards is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting reinforcement learning: agents & rewards to at least 3 other topics you have studied.