Data Science with Pandas: Analyzing Real Data

Suppose your teacher hands you a file with one month of Indian Railways delay records for your zone: every train, every date, how many minutes late it arrived. Not six rows — three hundred. She asks three questions: What was the average delay? Which single train run was the worst offender? Which train number is, on average, the most consistently late across the month?

Try answering this by manually scrolling a spreadsheet. Finding the average means adding 300 numbers without a mistake. Finding "the worst train on average" means mentally sorting 300 rows into groups by train number, adding each group separately, and dividing — by hand, that is hours of careful, boring, error-prone arithmetic. This is exactly the kind of problem that broke spreadsheets for real analysts and led to the tool this chapter teaches: pandas, a Python library built specifically for loading, inspecting, filtering, grouping, and summarizing tables of data with a few lines of code instead of a few hours of clicking.

We will build every idea in this chapter from a tiny, six-row table you can trace by hand — so that when pandas prints an answer, you already know, before you run it, exactly why that number is correct. Only once the six-row version is fully understood do we scale up to the real 300-row file.

From a Spreadsheet Table to a DataFrame

Here is our starting dataset — delay records for two trains, three days each:

train_no,date,delay_min
12621,2026-01-05,12
12622,2026-01-05,5
12621,2026-01-06,40
12622,2026-01-06,8
12621,2026-01-07,15
12622,2026-01-07,22

Save this exact text as a file named delays.csv — we will use this six-row file for every hand-traced example in this chapter. Each line is one row (one recorded delay). Each comma-separated field belongs to a column (a fact that every row shares: which train, which date, how many minutes late). This row-and-column structure is exactly what a spreadsheet gives you — pandas simply gives that same structure a name and a set of precise tools.

In pandas, a table like this is called a DataFrame. Before reading it from a file, let's build one directly in code, from a Python dictionary, so you can see exactly what a DataFrame is made of:

import pandas as pd

data = {
    "train_no":  [12621, 12622, 12621, 12622, 12621, 12622],
    "date":      ["2026-01-05", "2026-01-05", "2026-01-06",
                   "2026-01-06", "2026-01-07", "2026-01-07"],
    "delay_min": [12, 5, 40, 8, 15, 22]
}
df = pd.DataFrame(data)
print(df)

Output:

   train_no        date  delay_min
0     12621  2026-01-05         12
1     12622  2026-01-05          5
2     12621  2026-01-06         40
3     12622  2026-01-06          8
4     12621  2026-01-07         15
5     12622  2026-01-07         22

Notice the unlabeled leftmost column: 0, 1, 2, 3, 4, 5. Pandas added this automatically. It is called the index — a label attached to every row so that any single row can be found again instantly, without scanning the whole table. You did not ask for it; pandas created a default index (0, 1, 2, …) because the dictionary itself did not specify row labels. Later, you will see how the index becomes essential once we filter and sort — after those operations, row 2 might no longer be the third row printed, but its index label still says "2," so you never lose track of which original record you are looking at.

A DataFrame Is a Dictionary of Series, Sharing One Index

Pull out a single column:

print(df["delay_min"])

0    12
1     5
2    40
3     8
4    15
5    22
Name: delay_min, dtype: int64

This one-dimensional, labeled sequence of values is called a Series. It is not just a plain list — every value carries its index label with it (the 0, 1, 2, … on the left), the same index labels that appear on the full DataFrame. This is the core structural fact of pandas: a DataFrame is a collection of Series, one per column, all sharing the same row index. That shared index is what lets pandas guarantee that "row 3 of the train_no column" and "row 3 of the delay_min column" always refer to the same real-world record, even after you reorder, filter, or combine data.

Reading Real Data With pd.read_csv()

Typing out a dictionary works for six rows. For a real file — a CSV (comma-separated values) exported from a railway system, a school attendance register, or a UPI transaction history — you load it directly:

df = pd.read_csv("delays.csv")

This produces the exact same DataFrame shown above. Before doing anything else with real data, always inspect its shape and structure — never assume you know what you loaded:

print(df.shape)

(6, 3)

(6, 3) means 6 rows, 3 columns — always rows first, then columns, matching how you'd say it in a sentence ("six rows by three columns").

print(df.dtypes)

train_no       int64
date          object
delay_min      int64
dtype: object

This tells you how pandas interpreted each column. train_no and delay_min became whole numbers (int64). date became object — pandas' label for a column holding text (strings), because 2026-01-05 isn't valid arithmetic to pandas unless you explicitly convert it with pd.to_datetime(). This distinction matters constantly: if a numeric-looking column shows up as object instead of int64 or float64, it usually means the file has stray text, a missing value marker, or a formatting symbol hiding inside a column you expected to be pure numbers — the .dtypes check is often the fastest way to catch a dirty file before it corrupts a calculation.

print(df.columns)

Index(['train_no', 'date', 'delay_min'], dtype='object')

The column names themselves are stored as an Index too — the same structural object used for row labels. (Note: this exact output — dtype='object' — is what classic pandas versions print. Very recent pandas releases running with the newer opt-in string dtype backend may instead show dtype='str' for this line; either way, the column names themselves are identical.)

Selecting: Columns, Rows, and the Difference Between loc and iloc

You already saw df["delay_min"] pull out one column as a Series. To pull out one row, pandas gives you two related tools that are a frequent source of confusion, so trace both carefully.

.loc[] selects by label — the index value itself:

print(df.loc[2])

train_no          12621
date         2026-01-06
delay_min             40
Name: 2, dtype: object

.iloc[] selects by position — "the row at this numeric position, counting from zero," regardless of what its index label says:

print(df.iloc[0])

train_no          12621
date         2026-01-05
delay_min             12
Name: 0, dtype: object

Right now, with the default index 0–5, df.loc[2] and df.iloc[2] happen to return the same row, which is exactly why the distinction feels pointless at first — and exactly why it later causes real bugs. The moment you filter or sort a DataFrame (next section), positions and labels come apart: a row's label ("this is originally row 2") stays fixed forever, but its position ("this is now the 4th row printed") changes. Using .iloc when you meant .loc, or the reverse, after a filter or sort is one of the most common pandas mistakes — and it fails silently, quietly returning the wrong row instead of raising an error, which is what makes it dangerous.

Also notice both rows print with dtype: object, even though they contain a mix of numbers and text. That's because a single row, sliced across columns of different types (int64, object, int64), can no longer be represented as one uniformly-typed Series — pandas falls back to the most general type, object, which can hold anything. This is different from df["delay_min"] above, which stayed int64 because every value in that one column really is an integer.

Filtering With Boolean Masks

Now the real payoff: finding rows that satisfy a condition, without writing a single loop.

mask = df["delay_min"] > 15
print(mask)

0    False
1    False
2     True
3    False
4    False
5     True
Name: delay_min, dtype: bool

Read this literally: pandas compared every single value in the delay_min Series to 15, one at a time, and produced a new Series of the same length and the same index, holding True or False for each row. This is called a boolean mask — a Series of True/False values shaped exactly like the column you're filtering.

Feed that mask back into the DataFrame with square brackets, and pandas keeps only the rows marked True:

print(df[df["delay_min"] > 15])

   train_no        date  delay_min
2     12621  2026-01-06         40
5     12622  2026-01-07         22

Notice the surviving rows keep their original index labels, 2 and 5 — not renumbered to 0 and 1. This is the shared-index guarantee from earlier in action: even after filtering, row 2 still means "the third record in the original file," so you can always trace a filtered result back to where it came from.

Descriptive Statistics — and a Misconception That Trips Up Almost Everyone

Pandas can summarize an entire numeric column in one call:

print(df["delay_min"].describe())

count     6.000000
mean     17.000000
std      12.712199
min       5.000000
25%       9.000000
50%      13.500000
75%      20.250000
max      40.000000
Name: delay_min, dtype: float64

Most of these are easy to verify by hand. mean: (12+5+40+8+15+22)/6 = 102/6 = 17.0. min and max are the smallest and largest values, 5 and 40. The 50% row is the median — sort the six values (5, 8, 12, 15, 22, 40); with an even count, pandas averages the two middle values, (12+15)/2 = 13.5, exactly what's printed. (The 25% and 75% rows use the same sorted-and-interpolated logic one quarter and three-quarters of the way through the sorted list — worth knowing they exist, but the mechanics matter less than mean, median, and std for now.)

std — standard deviation — is worth slowing down for, because its formula is where a very common misconception lives.

Standard deviation measures how spread out the values are around the mean. The recipe: find how far each value is from the mean, square that distance (so negative and positive distances don't cancel out), add up all six squared distances, divide by a count, then take the square root to undo the earlier squaring.

mean = 17
deviations:  12-17=-5   5-17=-12   40-17=23   8-17=-9   15-17=-2   22-17=5
squared:        25         144        529        81         4        25
sum of squared deviations = 25+144+529+81+4+25 = 808

Now the question: divide 808 by what — 6 (the count of values) or 5 (one less)? Try the natural guess, dividing by 6: 808/6 = 134.666667, and √134.666667 ≈ 11.6046. But pandas' .describe() printed std = 12.712199, not 11.6046. Where does that number come from?

Common Misconception: Students (and even the plain textbook formula for standard deviation) usually divide by n, the number of values. Pandas' .std() does not — by default, it divides by n − 1. Using n − 1 = 5: 808/5 = 161.6, and √161.6 ≈ 12.712199 — which matches the printed output exactly.

Population std (divide by n):     std = sqrt( Σ(x - mean)² / n )
Sample std   (divide by n - 1):   std = sqrt( Σ(x - mean)² / (n - 1) )   ← pandas default

Why the "−1"? Dividing by n − 1 is called Bessel's correction. In real data analysis, the rows you have are almost always a sample drawn from a larger process — six days of delays out of a whole year of running trains, not every delay that will ever happen on that route. A sample's own mean is computed from that same sample, which makes the sample slightly "tuned" to look less spread out than the true, full population really is. Dividing by a slightly smaller number (n − 1 instead of n) inflates the result just enough to correct for that bias. Pandas assumes you're usually working with a sample, so .std() and .var() default to n − 1 (formally, this is controlled by a parameter called ddof, "delta degrees of freedom," which defaults to ddof=1).

This exact mismatch is a classic bug source when a script mixes libraries: NumPy's .std() defaults the opposite way (ddof=0, dividing by n), so numpy.std(df["delay_min"]) would print ≈11.6046 on this same data, while df["delay_min"].std() prints 12.712199 — two different "correct" answers to what looks like the identical question, purely because of a silent default. Always check which one a function is using before you rely on the number.

Grouping: "Which Train Is Worse, On Average?"

This is the operation that made the 300-row problem from the introduction solvable in one line. groupby() splits the DataFrame into buckets that share a value in some column, applies a calculation separately to each bucket, and combines the results back into one table:

print(df.groupby("train_no")["delay_min"].mean())

train_no
12621    22.333333
12622    11.666667
Name: delay_min, dtype: float64

Trace it: train 12621 appears in rows 0, 2, 4 with delays 12, 40, 15 → mean = 67/3 = 22.333333. Train 12622 appears in rows 1, 3, 5 with delays 5, 8, 22 → mean = 35/3 = 11.666667. Both numbers match the printed output exactly, and both were computed the same way .mean() works on any Series — groupby() just ran that same calculation once per bucket, automatically, instead of you writing a loop to separate the rows yourself.

You can count rows per bucket the same way, without doing math on them at all:

print(df["train_no"].value_counts())

train_no
12621    3
12622    3
Name: count, dtype: int64

(The exact wording of that last label line has changed across pandas versions — older releases print Name: train_no, dtype: int64; pandas 2.x and later print Name: count, dtype: int64 as shown above. The counts themselves, 3 and 3, are unaffected either way.)

Sorting and Finding the Single Worst Record

print(df.sort_values("delay_min", ascending=False))

   train_no        date  delay_min
2     12621  2026-01-06         40
5     12622  2026-01-07         22
4     12621  2026-01-07         15
0     12621  2026-01-05         12
3     12622  2026-01-06          8
1     12622  2026-01-05          5

Notice again: the printed row order changed completely, but every row still carries its original index label (2, 5, 4, 0, 3, 1) — proof that sorting rearranged positions, not identities. To get straight to the single worst row without sorting the whole table, use idxmax(), which returns the index label of the largest value:

worst = df["delay_min"].idxmax()
print(worst)
print(df.loc[worst])

2
train_no          12621
date         2026-01-06
delay_min             40
Name: 2, dtype: object

Putting It All Together: A Realistic Script

Everything above was traced on six rows so every number could be verified by hand. A real monthly extract has the same three columns — train_no, date, delay_min — just far more rows, so save it under its own name, railway_delays_month.csv, rather than reusing delays.csv: it is a larger file with the identical structure, not a bigger version of the six rows we've been tracing. The code below is what you would actually run to answer the teacher's three original questions — the logic is line-for-line what you already traced above, just pointed at 300 rows instead of 6:

import pandas as pd

df = pd.read_csv("railway_delays_month.csv")

print("Shape:", df.shape)
print(df.dtypes)

print("\nOverall average delay (minutes):")
print(df["delay_min"].mean())

print("\nSingle worst delay:")
print(df.loc[df["delay_min"].idxmax()])

print("\nAverage delay per train, worst first:")
print(df.groupby("train_no")["delay_min"].mean().sort_values(ascending=False))

print("\nHow many runs were delayed more than 30 minutes?")
print((df["delay_min"] > 30).sum())

That last line is worth pausing on: (df["delay_min"] > 30) produces a boolean mask, exactly as before — a Series of True/False values. In Python, True behaves as 1 and False as 0 in arithmetic, so calling .sum() on a boolean Series counts how many values were True — a compact, very common pandas idiom for "how many rows satisfy this condition," without ever writing if row.delay_min > 30: count += 1 in an explicit loop.

Six lines of real code just answered, over an entire month of real records, exactly what took paragraphs of manual arithmetic to verify on six rows. That gap — six rows by hand versus 300 rows in six lines — is the entire reason pandas exists, and it is also exactly why understanding the six-row version first matters: you now know precisely what each of those six lines is doing underneath, instead of trusting a black box.

Check Your Understanding

1. Using the six-row delays.csv table, what does df.loc[4] return? Write out the exact Series, including its Name and dtype line.
2. A classmate computes the standard deviation of delay_min using numpy.std() instead of pandas' .std() and gets a different number than the describe() table. Explain precisely why, using the words "ddof" and "n − 1" in your answer.
3. Write the boolean mask (as a list of True/False values, one per row 0–5) that df["delay_min"] <= 12 would produce on the six-row dataset.
4. If you ran df.groupby("train_no")["delay_min"].max() on the six-row dataset, what two numbers would be printed, and next to which train numbers?
5. After sorted_df = df.sort_values("delay_min"), what is the value of sorted_df.iloc[0]["train_no"]? What is the value of sorted_df.loc[0]["train_no"]? Explain why these two answers are different.

Summary

• A pandas DataFrame is a table of rows and columns; a Series is a single labeled column (or row) pulled out of it. Every Series in a DataFrame shares the same row index, which is how pandas keeps track of which values belong to the same original record even after filtering or sorting.
• pd.read_csv() loads a file into a DataFrame; .shape, .dtypes, and .columns are the first things to check before trusting any calculation on it.
• .loc[] selects by index label; .iloc[] selects by position. They agree only while the default 0, 1, 2, … index is untouched by filtering or sorting — after that, confusing the two silently returns the wrong row.
• A boolean mask — a True/False Series produced by a comparison like df["delay_min"] > 15 — is how pandas filters rows, and .sum() on such a mask counts how many rows were True.
• .describe() reports count, mean, std, min, quartiles, and max in one call. Pandas' default .std() divides by n − 1, not n (ddof=1, Bessel's correction) — a different, larger number than the population formula many students expect, and different from NumPy's default.
• .groupby() splits rows into buckets sharing a column value and applies a calculation to each bucket separately — the single most powerful tool for turning "one big table" into "one answer per category" without writing a manual loop.

Think About It

Think about this: How would you explain data science with pandas: analyzing real data to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where data science with pandas: analyzing real data is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting data science with pandas: analyzing real data to at least 3 other topics you have studied.