Imagine you want to compare the price of a scientific calculator across five different online stores before your parents buy one. You open five tabs, note down each price on a piece of paper, and after twenty minutes of clicking and scrolling, you have your answer. Now imagine doing this for 500 products, every single day, to see how prices move during a festival sale like Diwali or the Amazon Great Indian Festival. Doing it by hand would take weeks, and you would still make copying mistakes along the way. This is exactly the problem web scraping solves: writing a program that reads a web page the way you do, finds the exact pieces of information you care about, and pulls them out automatically — thousands of times, without getting tired or mistyping a single digit.
Web scraping is not a vague idea like "using the internet with code." It is a precise process with well-defined steps, and by the end of this chapter you will be able to trace exactly what a scraping program does to a real web page, one line at a time, and predict its exact output — the same way you already trace a loop in Python and predict what it prints.
What a Web Page Actually Is
When your browser shows you a neatly formatted page — headings, tables, buttons — it is not showing you some magical picture. It is showing you the rendered version of a text file written in HTML (HyperText Markup Language). That text file is the actual thing your browser downloaded from the server. Every visible element on the page corresponds to a tag in that text.
Here is a tiny, complete HTML page. This is not a simplification for teaching purposes — this is precisely the kind of text a real web server sends back:
<html>
<body>
<h1>Train Fares</h1>
<p class="route">New Delhi to Mumbai</p>
</body>
</html>A browser reads this and draws a big bold heading "Train Fares" followed by a line of text "New Delhi to Mumbai." A web scraper does something different: instead of drawing anything, it reads this exact same text and asks, "which tags exist, what attributes do they carry, and what text sits inside them?" Web scraping is fundamentally about treating the HTML source — not the pretty rendered picture — as your input data.
You can see this difference yourself: in any browser, right-click a web page and choose "View Page Source" (or press Ctrl+U). What you see is not the pretty page — it is the raw HTML text, tags and all. That raw text is exactly what a scraping program receives.
The Request-Response Cycle
Before a program can read a page's HTML, it must first ask the server for it. This conversation between a program and a server has a name: the request-response cycle. It has two steps:
- Your program sends an HTTP request to a URL, essentially saying "please send me the contents of this page."
- The server sends back an HTTP response, which includes a status code (200 means "here you go, success"; 404 means "not found"; 403 means "you're not allowed") and, if successful, the full HTML text of the page as its body.
In Python, this is done with a library called requests:
import requests
response = requests.get("https://example.com/train-fares")
print(response.status_code) # 200 if the page loaded successfully
html_text = response.text # the raw HTML, as one big stringAfter this runs, html_text is just a Python string — the same kind of string you already work with, except this one happens to contain angle-bracket tags. The diagram below shows the full round trip:
Notice what requests.get() does not do: it does not click buttons, does not scroll, does not run any JavaScript that the page might use to load extra content after the initial page arrives. It fetches exactly the HTML text the server sent in that first response — nothing more. This becomes important later, when we discuss why some pages are hard to scrape.
Turning HTML into a Tree: The DOM
Once you have the HTML as a string, you could try to find data using string tricks — searching for the position of <td>, cutting out the text after it, and so on. This is exactly what beginners try first, and it breaks constantly, because HTML is not really "flat text with some tags sprinkled in." It has a nested, tree-like structure, where every tag can contain other tags inside it, like boxes inside boxes.
This nested structure is called the DOM (Document Object Model). A parsing library reads the raw HTML string and builds this tree in memory, so your program can ask tree-shaped questions like "give me every <td> tag that has the class fare" instead of fragile string-position questions. In Python, the standard tool for this is a library called BeautifulSoup.
Consider this HTML — a small table of train fares, exactly the kind of table you might find while comparing IRCTC bookings:
html_page = """
<html>
<body>
<h1>Train Fares</h1>
<table>
<tr>
<td class="train">Rajdhani Express</td>
<td class="fare">2500</td>
</tr>
<tr>
<td class="train">Shatabdi Express</td>
<td class="fare">1800</td>
</tr>
</table>
</body>
</html>
"""Once parsed, this text becomes the tree shown below. Notice that <td> is not a single kind of node — there are four separate <td> tags, and the only thing distinguishing "train name" cells from "fare" cells is the class attribute written on each one.
This is the whole point of parsing: the tree lets your program say "walk down from the root and collect every <td> node whose class attribute equals fare," and it will correctly skip the "train name" cells even though both are <td> tags, because it is checking the attribute, not just the tag name.
Your First Scraper: Extracting Train Fares
Now let's actually write and trace the code. Using the same html_page string from before:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_page, "html.parser")
trains = soup.find_all("td", class_="train")
fares = soup.find_all("td", class_="fare")
for t, f in zip(trains, fares):
print(t.text, "->", "Rs.", f.text)Let's trace this exactly, the way you would trace any loop:
BeautifulSoup(html_page, "html.parser")builds the tree from the previous section and stores it insoup.soup.find_all("td", class_="train")walks the whole tree and collects every<td>node withclass="train", in the order they appear in the document. That gives a list of two tags: the one containing "Rajdhani Express" and the one containing "Shatabdi Express".- Similarly,
faresbecomes a list of two tags: "2500" and "1800", in that order. zip(trains, fares)pairs them up by position: (Rajdhani-tag, 2500-tag), then (Shatabdi-tag, 1800-tag).- For each pair,
t.textgives you the tag's inner text as a plain Python string (not the tag object, just the words between<td>and</td>).
So the exact printed output is:
Rajdhani Express -> Rs. 2500
Shatabdi Express -> Rs. 1800Notice something important: this worked correctly only because "train name" cells and "fare" cells were tagged with different class names. If the website's author had used the same class for both, or no class at all, your program would have no reliable way to tell them apart — it would need to fall back on position ("the second <td> in every row is the fare"), which is far more fragile, as we'll see shortly.
Extracting Attributes, Not Just Text
Sometimes the data you want is not the visible text but an attribute — like a link's destination. Suppose you're scraping headlines and their article links from a news page:
news_html = """
<div class="news">
<a href="/article/isro-launch" class="headline">ISRO launches new satellite</a>
<a href="/article/exam-dates" class="headline">CBSE releases board exam dates</a>
</div>
"""
soup2 = BeautifulSoup(news_html, "html.parser")
links = soup2.find_all("a", class_="headline")
for link in links:
print(link.text, "|", link["href"])Tracing this: find_all("a", class_="headline") returns the two <a> tags in document order. For each one, link.text gives the visible headline words, and link["href"] reaches into the tag's attributes dictionary and pulls out the value of href — the URL the link actually points to, which is never shown as visible text on the page. The output is:
ISRO launches new satellite | /article/isro-launch
CBSE releases board exam dates | /article/exam-datesThis is a different extraction target from before — .text reads what's between the tags, while ["attribute_name"] reads what's written inside the opening tag itself. Confusing these two is one of the most common beginner mistakes: writing link.text when you actually wanted the URL, or vice versa.
Common Misconception: "Scraping" and "Using an API" Are the Same Thing
Many students hear "getting data from a website with code" and assume it's always the same activity. It isn't, and the difference matters:
- An API (Application Programming Interface) is data a website's owner deliberately publishes for programs to consume — usually as structured JSON, with documentation, at a stable address, meant to stay the same over time.
- Web scraping extracts data from HTML that was built for human eyes in a browser, not for programs. Nobody promised you that structure would stay stable.
This is not a minor technicality — it has a real consequence you can predict: if a website redesigns its page and renames the CSS class fare to price-value, every line of our first scraper that says class_="fare" instantly stops working, usually returning an empty list rather than an error, which is often more dangerous because your program keeps running silently with zero data instead of crashing loudly. An API, by contrast, is a published contract — if it changes, the provider typically warns developers in advance and versions the change. Always prefer an official API over scraping when one exists; scraping is the fallback tool for when no API is offered.
Why Scrapers Break: The Fragility Problem
Let's make the fragility concrete. Suppose the fare table's HTML changes ever so slightly — the site's developers decide fare cells no longer need a class:
<td>2500</td> <!-- class="fare" removed -->Now trace what happens: soup.find_all("td", class_="fare") searches for a class attribute that no longer exists on that tag. It returns an empty list, []. The for t, f in zip(trains, fares) loop then simply does not execute at all — zip stops at the shorter list, and an empty list makes the whole pairing empty. Your program finishes without printing anything and without any error message. This "silent failure" is precisely why professional scrapers always add checks — for example, printing a warning if len(fares) == 0 — rather than trusting that a selector which worked yesterday will still work today.
There is a second, subtler fragility worth knowing at this stage: some modern websites don't send their data as HTML at all in the first response. Instead, the initial HTML is nearly empty, and JavaScript running in your browser fills it in afterward by making its own separate requests. Since requests.get() only fetches that first, mostly-empty HTML and does not run JavaScript, a scraper built the way we've described here would find nothing — even though your eyes, looking at the rendered page, see the data perfectly well. This is why "the data isn't in the HTML I downloaded" is one of the very first things experienced scraper-writers check when their code returns nothing.
Scraping Responsibly: robots.txt and Rate Limits
Because scraping reads pages that were built for browsers, not robots, most well-run websites publish a file at a predictable address — example.com/robots.txt — listing which parts of the site automated programs are asked not to visit. It looks something like this:
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /This is a voluntary convention, not a lock — nothing stops a program from ignoring it technically — but respecting it is the accepted standard of good behaviour among programmers, similar to not walking into a shop's "staff only" room even though the door isn't locked. Separately, a site's Terms of Service may explicitly permit or forbid automated data collection, and that is a stronger, contractual boundary. Finally, sending requests too fast — hundreds per second — can overload a server the same way thousands of people rushing through one narrow gate at once causes a jam; responsible scrapers deliberately pause between requests (for example, using Python's time.sleep()) so they behave like one polite visitor rather than a flood.
From Extracted Data to Structured Storage
Extraction is only useful if the result ends up somewhere you can reuse it — a spreadsheet, a database, a chart. The final, natural step after our first scraper is writing the extracted pairs into a CSV file, which any spreadsheet program (including one you might open in a computer lab) can read directly:
import csv
with open("fares.csv", "w", newline="") as file:
writer = csv.writer(file)
writer.writerow(["Train", "Fare"])
for t, f in zip(trains, fares):
writer.writerow([t.text, f.text])Tracing this: writer.writerow([...]) writes one line per call, with commas separating the values. After the header row and the loop runs twice (once per pair from our earlier trains/fares lists), the file fares.csv contains exactly:
Train,Fare
Rajdhani Express,2500
Shatabdi Express,1800This is the complete arc of web scraping in one chapter: fetch raw HTML over a request-response cycle, parse it into a navigable tree, select nodes precisely using tags and attributes, extract either text or attribute values, and store the result in a structured file — while staying aware of exactly where and why the process can silently fail.
Check Yourself
- Given
<li class="item" data-price="450">Notebook</li>parsed into a variabletag, what doestag.textreturn, and what doestag["data-price"]return? Are these the same kind of value? - A classmate writes a scraper for a college's exam-results page using
soup.find_all("td", class_="marks"). It works perfectly in January but returns an empty list in June. List two realistic, different explanations, based on what you learned about fragility. - Why does
requests.get()sometimes fail to retrieve data that you can clearly see when you open the same URL in a browser? Name the specific mechanism responsible. - Explain, in one or two sentences, why "I can see it in my browser, so I can scrape it" is not automatically true — mention both a technical and a non-technical (permission-based) reason.
Summary
Web scraping is the practice of programmatically retrieving a web page's raw HTML and extracting specific data from it, as an alternative to manual copy-pasting when no official API is available. The process has four concrete stages: an HTTP request-response cycle fetches the raw HTML text from a server; a parsing library like BeautifulSoup converts that text into a tree-shaped DOM; tag names and attributes (especially class) are used to precisely select the nodes you want, via methods like find_all(); and the extracted values — read either as .text for inner content or ["attribute"] for attribute values — are stored in a structured format such as CSV. Scraping is inherently more fragile than an API because it depends on a page's presentation structure, which its owners are free to change without warning, and it may miss data that is added to a page only after JavaScript runs in the browser. Responsible scraping respects a site's robots.txt guidance and Terms of Service, and paces its requests so it never behaves like an overload attack on the server it is visiting.