Say you want to buy a phone during a Flipkart or Amazon sale, and you want to know whether today's price is actually the lowest it has been in the last month, or just a "sale" sticker on the same price as last week. To find out, you would need to open the product page every single day, note the price, and build a table by hand. Do that for thirty days and you have wasted an afternoon on something a ten-line program could do while you sleep: visit the page, read the price off the screen, save it, and repeat tomorrow. That program is a web scraper, and the whole process — pulling structured information out of a page that was built for human eyes, not for machines — is called web scraping. This chapter builds the technique from the ground up, and then spends equal time on a question the technique itself cannot answer: just because you can extract data from a page, should you?
What a Webpage Actually Is
When your browser shows you a product page, what it received from the server was never a picture — it was plain text written in HTML (HyperText Markup Language). The browser is the one doing the work of turning that text into boxes, fonts, and colours. A scraper skips the "turn it into a pretty picture" step entirely and works directly on the text the server sent, which looks something like this:
<div class="catalog">
<div class="book">
<h3 class="title">Introduction to AI</h3>
<span class="price">₹499</span>
</div>
<div class="book">
<h3 class="title">Python for Beginners</h3>
<span class="price">₹350</span>
</div>
</div>
Notice the structure: an outer <div> labelled "catalog" contains two inner <div>s labelled "book", and each of those contains a title tag and a price tag. Tags nest inside other tags the way folders nest inside other folders. This nesting is not incidental — it is the entire reason scraping is possible at all. If every page were just a flat wall of text with no tags, there would be no reliable way to tell "this particular number is a price" from "this number is a page count" or a phone number. The tags, and the class names attached to them, are the labels a program can grab onto.
Turning Tags into a Tree: the DOM
Before any extraction happens, a scraping program converts that flat block of HTML text into a tree structure called the DOM (Document Object Model), where each tag becomes a node and each nested tag becomes a child node. This is exactly what your browser does before it paints the page, and it is exactly what a scraping library does before it lets you search for anything. For the catalog HTML above, the tree looks like this:
Once the HTML is a tree instead of a string, "find the price" becomes a precise, mechanical instruction: "starting at the root, go to a div.book child, then to its span.price child, then read the text inside it." That instruction is exactly what a parsing library executes for you.
Why "Just Use Regex" Is a Trap
A natural first idea, before you know about DOM trees, is to skip all this and just search the raw text for patterns — for instance, "find the text between <span class="price"> and </span>" using a regular expression. This works on the tidy four-line example above. It quietly breaks on real websites, for reasons that have nothing to do with your regex skill: a real price tag might read <span class='price' data-currency="INR"> (single quotes, extra attributes, different attribute order), or the price might be split across nested tags for styling, like <span class="price">₹<b>499</b></span>. HTML tags can nest to arbitrary, unpredictable depth, and matching "nested things that can contain more of themselves" is precisely the kind of pattern that regular expressions — which describe flat, fixed patterns — are mathematically the wrong tool for. This is such a well-known trap in programming that it has its own running joke among developers: "you cannot parse HTML with regex." The fix is not a cleverer regex; it is using an actual HTML parser that builds the real DOM tree, exactly as a browser would, so that nesting, quote style, and attribute order stop mattering.
Parsing With a Real Library: a Full Trace
In Python, the standard tool for this is BeautifulSoup, paired with the requests library for fetching pages. Let's trace it line by line on the catalog HTML from before, stored in a variable called html:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
books = soup.find_all("div", class_="book")
for book in books:
title = book.find("h3", class_="title").text
price = book.find("span", class_="price").text
print(title, "->", price)
Tracing what happens: BeautifulSoup(html, "html.parser") reads the text and builds the DOM tree from the diagram above, storing it in soup. soup.find_all("div", class_="book") walks that tree and returns a list of the two div.book nodes — note the trailing underscore in class_, needed only because class is already a reserved Python keyword. The loop then visits each book node in turn: on the first pass, book.find("h3", class_="title") returns the <h3> node containing "Introduction to AI", and .text pulls out just the visible characters, discarding the tags. The program prints:
Introduction to AI -> ₹499
Python for Beginners -> ₹350
Two structured rows, extracted from four lines of tag soup, with zero regular expressions and zero assumptions about quote style or attribute order.
Scraping Across Many Pages, Politely
Real catalogues span many pages, and a real scraper fetches the live page over the network instead of using a hardcoded string:
import requests
from bs4 import BeautifulSoup
import time
BASE_URL = "https://example-shop.in/products"
for page_number in range(1, 4): # pages 1, 2, 3
response = requests.get(f"{BASE_URL}?page={page_number}")
soup = BeautifulSoup(response.text, "html.parser")
for item in soup.find_all("div", class_="product"):
name = item.find("h2").text.strip()
price = item.find("span", class_="price").text.strip()
print(page_number, name, price)
time.sleep(2) # wait 2 seconds before requesting the next page
range(1, 4) produces 1, 2, and 3 — three pages, not four, since range's upper bound is exclusive. Each iteration fetches one page, parses it, prints every product on it, and then calls time.sleep(2) before moving on. That last line is doing more work than it looks like: without it, this loop would fire three HTTP requests back-to-back in a few milliseconds, hammering someone else's server. With a two-second gap, three pages take at least 3 × 2 = 6 seconds of deliberate delay — for three pages that costs you nothing, but scaled up to 3,000 pages it is the difference between a scraper that behaves like one extra polite visitor and one that behaves like a denial-of-service attack. This single line, time.sleep(2), is where scraping technique and scraping ethics physically meet inside the code.
The Website's Posted Rules: robots.txt
Most websites publish a plain-text file at a fixed address — example.com/robots.txt — that states which parts of the site automated programs are and are not permitted to visit. It might read:
User-agent: *
Disallow: /private/
Disallow: /search
Allow: /search/public
Crawl-delay: 10
Read line by line: User-agent: * means these rules apply to every automated crawler, not just one named company's bot. Disallow: /private/ means nothing under that path should be visited by a bot at all. Disallow: /search blocks the entire /search section — except that the next line, Allow: /search/public, carves out a specific exception, because most crawlers apply whichever rule matches the longest, most specific part of the path. So a request for /search/public/results is allowed (the more specific /search/public rule wins), while a request for /search/internal is blocked (only the broader /search rule applies there). Crawl-delay: 10 is the site operator directly telling you the polite gap between requests: ten seconds, not two. This file is not a technical lock — nothing stops your code from ignoring it — which is exactly why respecting it matters: it is the site owner's explicit, written permission boundary, and deliberately crossing it is the scraping equivalent of walking past a "staff only" sign because the door happened to be unlocked.
Scraping vs. Calling an API
If you have already worked with a JSON API, you have seen a much friendlier version of "getting data from a website": you send a request to an address the provider built specifically for programs, and you get back clean, labelled data — no tags to parse, no DOM tree to walk. Scraping is what you resort to when no such API exists, or when the API does not expose the specific field you need. The trade-offs matter: an API is a contract — the provider promises the shape of the data won't change without warning, and usually documents exactly how much you're allowed to request and how often. A scraped page is not a contract with you at all; it was built for a human with a browser, and if the site redesigns its layout tomorrow — renaming class="price" to class="cost" — every scraper built against the old structure breaks silently. Scraping is a fallback technique for when the front door (the API) is locked, not a first choice.
When "Public" Doesn't Mean "Permitted"
It is tempting to reason that if a page is visible to anyone with a browser and no password, then reading it with a program must also be fine. That reasoning misses three separate boundaries that operate independently of whether a page happens to require a login:
- Terms of Service. By using a website, you typically agree to its terms, and many sites explicitly forbid automated access even to public pages. IRCTC's terms, for example, forbid booking tickets through unauthorised software or "bots" rather than the official interface — a rule aimed squarely at scraping-style automation, and using such software to book tickets has been treated in India as an offence comparable to ticket touting, not merely a broken promise. Violating a site's terms is generally a contract or civil matter rather than "hacking," but it is real and enforceable, and it exists independently of whether the underlying page was public.
- Copyright. The text, images, and original data on a page are usually someone's copyrighted work the moment they are published, public or not. Extracting facts (like a price, which is not copyrightable) is very different from copying and republishing whole articles, reviews, or photographs — the second is a copyright question even if scraping the page itself was technically trivial.
- Privacy. A page being visible does not make the personal details on it fair game to harvest in bulk. In India, the Digital Personal Data Protection Act, 2023 governs how personal data — names, phone numbers, photos, addresses — can be collected and processed, and scraping such details off public profiles at scale can fall squarely within its scope, even though no single piece of data was hidden.
A well-known illustration of this tension played out over several years in the United States, between a data-analytics company, hiQ Labs, and LinkedIn, over whether scraping publicly visible LinkedIn profiles was permitted. Courts ended up drawing a distinction between two different questions that are easy to conflate: whether scraping public data violates strict computer-hacking law (aimed at breaking into systems you have no authorisation to access at all), and whether it violates a website's own posted terms (a separate, contract-style claim). The two questions can have different answers for the exact same scraping script — which is precisely the point: "not hacking" and "fully permitted" are not the same thing, and a scraper can be legal on one axis and still get you blocked, sued, or banned on the other.
A Checklist Before You Scrape
Put together, the practical questions to ask before running a scraper — every time, not just once for a project — form a short sequence:
Notice that only two of the four checks are technical (robots.txt, request speed); the other two — Terms of Service and personal data — are legal and ethical judgments that no library or piece of code can make for you. That is the core idea to take away from this chapter: writing the parsing code is the easy 20%; deciding whether you should run it is the harder 80%, and it does not get easier just because the "Disallow" list happened to leave your target path unmentioned.
A Heads-Up: Pages That Build Themselves After Loading
One more wrinkle worth knowing about, even at overview level: the technique in this chapter — requests.get() followed by BeautifulSoup — only sees the HTML the server sends immediately. Many modern sites send a nearly empty page and then run JavaScript in your browser afterward to fill in the actual product list or comments, fetching that data separately. If you request such a page with requests, you will get back the empty shell, not the data you can see when you open it in an actual browser. Scraping those sites needs a tool that runs a real browser behind the scenes — such as Selenium or Playwright — so the JavaScript actually executes before anything is extracted. The DOM-tree idea and the ethics checklist above apply identically either way; only the fetching step changes.
Check Your Understanding
- A page's
robots.txtcontainsDisallow: /reviewsfollowed byAllow: /reviews/summary. Is a request to/reviews/summary/2026allowed? Why? - Explain, in your own words, why "just search the HTML text with a pattern-matching regex" breaks on real pages but a DOM-tree parser like BeautifulSoup does not.
- A classmate says, "The prices on this shop's site are visible to anyone, so scraping and reselling the exact list is totally fine." Name two separate reasons this could still cause a problem, beyond whether the scraping itself worked technically.
- A loop scrapes 500 pages with
time.sleep(3)between each request. Roughly how long does the delay alone add to the total run time, and why does that number matter for the target server? - What specifically does an API promise you that a scraped webpage does not?
Summary
A webpage is text before it is a picture, and that text is structured as nested tags — which a parser turns into a DOM tree so that "find the price" becomes a precise walk from parent node to child node, rather than a fragile text search. BeautifulSoup (with requests for fetching and time.sleep() for pacing) is the standard toolkit for this in Python, and it is what you reach for only when a proper API is unavailable. But the code that fetches and parses a page is the smaller half of "doing web scraping correctly." The larger half is a discipline: check robots.txt before you request anything, check whether the site's Terms of Service permit automated access at all, treat any personal data you encounter as governed by privacy law regardless of visibility, keep your request rate slow enough that you are indistinguishable from one polite human visitor, and never assume that "the page was public" settles every legal or ethical question by itself. Scraping is a genuinely useful data-engineering skill — and, like any tool that lets a program act on someone else's server at machine speed, it is one where writing correct code is necessary but nowhere near sufficient.
Think About It
Think about this: How would you explain web scraping and data extraction: ethics and practice to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.