Recurrent Neural Networks and Sequences

The commentator who resets every ball

Picture two commentators watching the same over of a T20 match. Bowler bowls six balls: 2 runs, 1 run, a dot ball, then a six, then two more dot balls. The first commentator has no memory at all — for every single ball, she only looks at that one ball in isolation and says something like "two runs scored" or "no run." The second commentator remembers everything that happened earlier in the over and says things like "after that six, the required rate has eased, and the bowler is under pressure after two low-scoring balls in a row." Both commentators saw the exact same six balls. But only the second one can talk about momentum, pressure, and trend — because those ideas only make sense when you track how the situation changes step by step, in order.

This is exactly the difference between a plain feedforward neural network (the kind you may have already met, where an input goes in, passes through some layers, and an output comes out, with no notion of "before" or "after") and a Recurrent Neural Network (RNN). An RNN is a network built specifically to process data that arrives one piece at a time, in order — a sequence — while carrying forward a summary of everything it has seen so far. That carried-forward summary is called the hidden state, and it is the single most important idea in this chapter.

Sequences are everywhere once you start looking for them: the runs scored ball by ball in an over, the words you type one after another into a WhatsApp message (which is why your keyboard can guess the next word), the daily closing value of the Sensex, the temperature readings from a weather station over a week, or the stream of status updates on an IRCTC ticket as it moves from "under processing" to "confirmed." In every one of these, order carries information. The sequence of runs 2, 1, 0, 6, 0, 0 describes a very different over than 0, 0, 6, 0, 1, 2 — same six numbers, same total, completely different story about how the pressure built or eased. Any model that just adds up the runs, or averages them, or looks at them one at a time with no memory, throws away exactly the information that made cricket commentary interesting in the first place. This is the problem RNNs are built to solve.

Why a plain feedforward network struggles here

You might ask: why not just feed the whole over into a normal feedforward network as one big input — six numbers in, one output out? Two real problems show up immediately. First, sequences don't come in a fixed length. One over has six balls; a full innings has hundreds of balls; a sentence might have four words or forty. A feedforward network needs a fixed-size input, decided in advance, so it cannot naturally handle "sometimes six numbers, sometimes six hundred." Second, and more subtly, if you did pad every sequence to some huge fixed length, the network would need to learn a completely separate rule for "what does ball 1 mean," "what does ball 2 mean," and so on, all the way up — it could never reuse what it learned about "a dot ball right after a six" whether that pattern happened at position 3 or position 300. An RNN fixes both problems with one idea: instead of looking at the whole sequence at once, it looks at one element at a time, updates a small running summary, and reuses the exact same update rule at every single step, no matter how long the sequence is or where in the sequence you are.

The core idea: a hidden state that updates one step at a time

Let's build the update rule concretely. Suppose we want to track a batter's "momentum score" — a single running number, call it h, that goes up when the batter is scoring freely and fades when nothing is happening. After every ball, we want a new momentum score that depends on two things: the runs scored on this ball, and the momentum score carried over from before this ball. In plain algebra, if x is the runs on the current ball and h_prev is the momentum score going into this ball, a simple update rule looks like:

h_new = wx * x + wh * h_prev + b

Here wx, wh, and b are just numbers (weights and a bias, exactly like the weights you've already seen in a single artificial neuron) that the network learns during training so that the momentum score actually ends up being useful for whatever it's predicting. wx controls how much a fresh run this ball should count; wh controls how much of the old momentum survives into the new step. This formula — new state depends on current input and old state, using the same fixed weights every time — is the entire mathematical heart of a recurrent neural network. Everything else in this chapter is detail on top of this one line.

Written in the general notation you'll see in textbooks and later grades, this is h_t = f(x_t, h_{t-1}), read as "the hidden state at time step t is some function of the input at time t and the hidden state at the previous time step, t-1." Real RNNs squeeze the result through a nonlinear "squashing" function (commonly tanh) to keep the numbers well-behaved over long sequences — we'll skip that squashing step in our hand-worked example below to keep the arithmetic simple with plain algebra, but keep it in mind as the version used in practice.

Worked example: momentum, ball by ball

Let's pick concrete numbers and trace through them by hand, the way a CBSE numerical problem would expect. Use wx = 1 (every run scored counts fully toward momentum), wh = 0.5 (half of the previous momentum survives into the next ball — it decays if nothing new happens), b = 0, and a starting momentum of h_0 = 0 before the first ball is bowled. Suppose the runs scored on eight consecutive balls are:

x = [2, 1, 0, 3, 0, 0, 0, 1]

Now apply h_t = 1 * x_t + 0.5 * h_{t-1} one ball at a time:

• Ball 1: h_1 = 1*2 + 0.5*0 = 2.0 — two runs, no prior momentum to add, so momentum is simply 2.0.
• Ball 2: h_2 = 1*1 + 0.5*2.0 = 1 + 1.0 = 2.0 — only 1 run this ball, but half of the previous momentum (1.0) carries over, so the total stays exactly 2.0.
• Ball 3: h_3 = 1*0 + 0.5*2.0 = 0 + 1.0 = 1.0 — a dot ball adds nothing, and momentum decays to half of what it was.
• Ball 4: h_4 = 1*3 + 0.5*1.0 = 3 + 0.5 = 3.5 — a big scoring ball spikes the momentum up.
• Ball 5: h_5 = 1*0 + 0.5*3.5 = 1.75
• Ball 6: h_6 = 1*0 + 0.5*1.75 = 0.875
• Ball 7: h_7 = 1*0 + 0.5*0.875 = 0.4375
• Ball 8: h_8 = 1*1 + 0.5*0.4375 = 1 + 0.21875 = 1.21875

Notice the shape of the story this single number tells: momentum holds steady through balls 1–2, decays through the dot ball at 3, spikes hard on the four at ball 4, then decays steadily through three dot balls (5, 6, 7) before ticking back up on ball 8. That decay pattern — losing half its value every ball with no new runs — is not an accident of this example; it is the direct, mechanical consequence of multiplying by wh = 0.5 at every step, and it will matter a lot in a few sections when we discuss why simple RNNs struggle with very long sequences.

Here is the same computation as a short Python function, matching the hand trace exactly:

def rnn_step(h_prev, x, wx=1, wh=0.5, b=0):
    return wx * x + wh * h_prev + b

h = 0
signals = [2, 1, 0, 3, 0, 0, 0, 1]
for x in signals:
    h = rnn_step(h, x)
    print(round(h, 5))

Running this executes print() eight separate times — once per ball — so it prints, one value per line: 2.0 / 2.0 / 1.0 / 3.5 / 1.75 / 0.875 / 0.4375 / 1.21875. Each line is the momentum score right after that ball, and each one was computed using only the current input and the single number left over from the line before it — the function never looks back at the original list x to "re-read" earlier balls. All of history that matters is squeezed into that one running variable h.

Turning the hidden state into a prediction

A momentum score is only useful if it eventually produces an actual prediction. Just as we had a rule turning (x, h_prev) into a new h, an RNN typically has a second, separate rule turning the hidden state into an output y at each step — for example, a predicted "riskiness" score for the next ball, using its own weight wy and bias c:

y_t = wy * h_t + c

With, say, wy = 2 and c = 0, the output after ball 4 would be y_4 = 2 * 3.5 = 7.0, while after ball 7 (deep in the dot-ball drought) it would be y_7 = 2 * 0.4375 = 0.875 — a much lower predicted value, matching the quieter momentum. The key structural point is that wx, wh, and wy are three separate, fixed numbers, each doing one clear job (how much a new input matters, how much old memory survives, how the memory becomes an output), and none of them change from one time step to the next.

Seeing it as a diagram: unrolling the recurrence

The recurrence relation is easiest to see once it's drawn out "unrolled" across time — one column per time step, with the same three weights reappearing in every column:

Reading the diagram left to right: at every time step, the current input x(t) combines with the hidden state carried over from the previous step through the arrow labelled Wh, producing the new hidden state h(t), which then produces an output y(t). The horizontal arrows between the h circles are the "memory line" — they are what make this network recurrent. If you covered up the horizontal arrows, you would be left with three completely independent feedforward networks, one per time step, with no way to know that x(2) came after x(1). It is precisely those horizontal connections, carrying h(t-1) forward, that let the network build up momentum, notice trends, and remember context.

A misconception worth correcting directly

Looking at an unrolled diagram like the one above, with a separate box for each time step, it is very natural to assume that each column has its own private set of weights — as if there's one "ball 1 predictor," a different "ball 2 predictor," and so on. This is wrong, and it is worth being precise about why. Unrolling is only a way of drawing the same computation repeated across time; it is not a description of separate machinery. There is exactly one Wx, one Wh, and one Wy in the whole network, and the identical three numbers are reused at time step 1, time step 2, time step 3, and every step after that — that's what the caption under the diagram is pointing at. This weight sharing is not a minor implementation detail; it's the entire reason RNNs can handle sequences of any length with a fixed, small number of parameters, and the entire reason a pattern the network learns from balls 1–2 of an over automatically applies when that same pattern shows up at balls 47–48 of an innings. A second, related misconception is treating the hidden state as if it were a growing list that stores every past input separately, like a scoreboard recording each ball. It isn't — h is a single fixed-size number (or, in real networks, a fixed-size vector), overwritten completely at every step. It does not remember individual old balls at all; it only remembers a blended summary of them, and — as the next section shows — that summary fades.

Why memory fades: the vanishing-influence problem

Go back to the momentum trace. The very first ball contributed its full value, 2, directly into h_1. By the time we reach h_8, seven steps later, how much of that original "2" is still influencing the result? Each recurrent step multiplies whatever survived by wh = 0.5, so after seven steps, the surviving fraction of that first ball's contribution is 0.5^7 = 0.0078125, meaning its contribution has shrunk to 2 * 0.0078125 = 0.015625 — down from an original value of 2.0. That's roughly 128 times smaller, or equivalently, only about 0.78% of ball 1's original influence is still detectable in h_8. In other words, this simple RNN has, for all practical purposes, forgotten what happened at the start of the over by the time it reaches the end of it.

This isn't a flaw specific to our toy example — it is a structural property of any plain RNN whose recurrent weight has magnitude less than 1: influence from early steps shrinks geometrically (exponentially) with distance, so long sequences systematically lose track of early context. (If the recurrent weight's magnitude were instead greater than 1, the opposite problem happens — values can blow up instead of vanishing.) This is known as the vanishing-influence problem (closely related to what you'll later see formally called the vanishing-gradient problem when RNNs are trained). It's precisely why a batter's momentum from over 3 tells you almost nothing useful about over 18, and why, for tasks that genuinely need to remember something from far back in a long sequence — like a pronoun in a long sentence referring back to a name mentioned several clauses earlier — plain RNNs of this kind struggle. This limitation is the direct motivation for more advanced recurrent designs such as LSTMs and GRUs, which add extra "gates" that let the network choose to preserve specific pieces of information over long stretches instead of always decaying them; you'll meet those in more advanced work, but the vanishing-influence arithmetic above is exactly the problem they exist to solve.

Where this shows up in real systems

Recurrent processing of sequences was, for years, the standard approach behind machine translation. Google's neural machine translation system (GNMT), introduced in November 2016, used a deep recurrent architecture (a stack of RNN-style layers) to translate whole sentences by reading them in, word by word, into a hidden state, and then generating the translated sentence, word by word, out of that state — a major jump in quality over the older approach of translating short phrases independently, because the recurrent hidden state let the model retain context across the entire sentence rather than losing it at phrase boundaries. GNMT's initial rollout covered a small set of language pairs (English to and from French, German, Spanish, Portuguese, Chinese, Japanese, Korean, and Turkish); as its coverage expanded over the following year, Hindi and other Indian languages were added, benefiting from the same sequence-memory advantage.

Closer to everyday life, the predictive-text suggestions on your phone's keyboard, voice assistants transcribing spoken Hindi or English into text, and apps that forecast a queue's wait time from a stream of past timestamps are all, at their core, dealing with the same problem this chapter has been building toward: given a sequence where order matters, keep a running summary and use it to predict what comes next. Whether the "sequence" is runs per ball, words per sentence, or transaction amounts per day on a UPI account, the underlying computation — a hidden state updated one step at a time by the same small set of weights — is the same recurrence relation you traced by hand in this chapter's worked example.

Check yourself

Use the recurrence rule h_t = wx * x_t + wh * h_{t-1} + b with wx = 1, b = 0, and h_0 = 0 throughout.

1. With wh = 0.3 and inputs x = [4, 0, 2], compute h_1, h_2, and h_3 by hand.
2. If you increased wh from 0.3 to 0.9 (keeping the same inputs), would old information fade faster or slower across steps? Explain using the multiplication in the recurrence rule.
3. Two overs have the exact same six runs — over A: 4, 0, 0, 0, 0, 6; over B: 6, 0, 0, 0, 0, 4. Explain, using the idea of a hidden state, why an RNN would very likely compute a different final momentum score h_6 for these two overs even though the total runs are identical.
4. True or False, with a one-line justification: "In an unrolled RNN diagram with five time steps, there are five different values of Wh, one for each step."

Answers: (1) h_1 = 1*4 + 0.3*0 = 4.0; h_2 = 1*0 + 0.3*4.0 = 1.2; h_3 = 1*2 + 0.3*1.2 = 2.36. (2) Slower — a larger wh closer to 1 means more of the previous hidden state survives each multiplication, so old contributions decay less per step (with wh = 0.9, seven steps back retains 0.9^7 ≈ 48% of its value, versus under 1% at wh = 0.5). (3) Because the recurrence is order-dependent: in over A, the big score (6) is freshest and barely decayed by the end, while in over B, the big score (6) is oldest and has already decayed through five recurrent steps by the time you reach h_6 — same numbers, different order, different final state. (4) False — the same single Wh (along with the same Wx and Wy) is reused at every time step; unrolling only draws the repeated computation across time, it does not create separate weights per step.

Summary

• A sequence is data where order carries meaning — ball-by-ball runs, words in a sentence, daily prices. Reordering a sequence usually changes what it means, so models built for sequences must process items one at a time, in order.
• An RNN keeps a fixed-size hidden state h that is updated at every step using the rule h_t = wx * x_t + wh * h_{t-1} + b — the new state depends on the current input and the previous state, nothing more.
• The same weights (Wx, Wh, Wy) are reused at every single time step — this weight sharing is what lets an RNN handle sequences of any length with a small, fixed number of parameters, and it is a frequently misunderstood point when reading unrolled diagrams.
• The hidden state is a blended summary, not a list of remembered inputs — old contributions shrink geometrically at every step (by a factor of wh each time), which is why plain RNNs struggle to remember information from far back in a long sequence — the vanishing-influence problem.
• An output y_t = wy * h_t + c can be produced at each step from the hidden state, turning the running summary into an actual prediction.
• Recurrent sequence processing powered systems like Google's 2016 neural machine translation engine (GNMT), and the same core idea — an updating hidden state — underlies predictive text, speech transcription, and forecasting over any time-ordered data stream.

Think About It

Think about this: How would you explain recurrent neural networks and sequences to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.