The problem: watching a price change, one click at a time
Suppose you want to know the best day to buy a coding book that an online store sells at a changing price — sometimes ₹399, sometimes discounted to ₹349 during a sale. To find the cheapest day, you decide to check the page every evening at 6 pm for a month and write the price in a notebook. Day 1: open the browser, find the price, write it down. Day 2: repeat. Day 30: repeat again. You never miss a step, but you also never enjoy the process — it is thirty identical, boring actions, and one tired evening where you forget to check breaks the whole record.
A computer does not get tired of repeating a boring, exact set of steps — that is precisely what computers are good at. If "open the page, find the price, write it down" can be written as a sequence of instructions, Python can run that sequence every evening without you touching a keyboard. Writing a program that automatically visits a webpage, reads its content, and pulls out just the pieces of information you care about is called web scraping. This chapter builds that program from the ground up — no scraping library will feel like magic by the end, because you will know exactly what each line asks the computer to do.
What a webpage actually is: text with labels, not a picture
Before writing any code, you need one correct mental model: a webpage is not a photograph. When your browser shows you a price in a box, what actually arrived over the internet was plain text — a long string of characters written in a language called HTML (HyperText Markup Language). The browser reads that text and draws it as a page. Web scraping skips the drawing step and works directly on the text.
HTML text is built from tags — words wrapped in angle brackets that label a piece of content. A tag almost always comes in an opening and closing pair, and whatever sits between them is what the tag describes:
<h3>AI for Class 9</h3>Here, h3 is the tag name (it means "a level-3 heading"), and "AI for Class 9" is the content it wraps. Tags can also carry extra information inside the opening bracket, called attributes. The most common one you will use for scraping is class, a label the page's designer attached so that similar-looking pieces of the page can be styled — and, conveniently for us, found — together:
<span class="price">₹399</span>Tags nest inside other tags, the way a folder can sit inside another folder. A whole product listing might look like this:
<div class="book">
<h3 class="title">Python for Beginners</h3>
<span class="price">₹399</span>
</div>Read this as a small family tree: one div (a generic container) is the parent of two children, an h3 and a span. This nested, tree-shaped structure is the single most important fact about HTML for a scraper to exploit — because "find the price" really means "find the span tag with class price that lives inside this particular div," and a tree is exactly the kind of structure a program can search through systematically.
The two-step recipe: fetch, then parse
Every web scraping program, no matter how complicated, is built from exactly two operations performed one after the other:
- Fetch — ask a web server for a page and receive its raw HTML text back, the same way your browser does when you type a URL.
- Parse — read that HTML text, understand its tag structure as a tree, and pull specific values out of specific branches.
In Python, one library does each job well: requests fetches pages, and BeautifulSoup (from the package beautifulsoup4) parses HTML. Neither is part of core Python, so before using them you install them once from the terminal:
pip install requests beautifulsoup4Step 1 — Parsing HTML you already have (no internet needed yet)
It is easier to learn the parsing half first, on HTML you can see completely, before adding the extra moving part of a live internet connection. Imagine an online bookstore's page has this structure for three books:
html_doc = """
<html>
<body>
<div class="book" data-stock="yes">
<h3 class="title">Python for Beginners</h3>
<span class="price">₹399</span>
</div>
<div class="book" data-stock="no">
<h3 class="title">Data Structures Made Easy</h3>
<span class="price">₹499</span>
</div>
<div class="book" data-stock="yes">
<h3 class="title">AI for Class 9</h3>
<span class="price">₹350</span>
</div>
</body>
</html>
"""Now parse it and pull out each title and price:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, "html.parser")
books = soup.find_all("div", class_="book")
for book in books:
title = book.find("h3", class_="title").text
price = book.find("span", class_="price").text
in_stock = book["data-stock"]
print(title, "-", price, "- In stock:", in_stock)Trace this line by line, because tracing is exactly the skill CBSE Python questions test. BeautifulSoup(html_doc, "html.parser") reads the whole string and builds the tag tree in memory, storing it in soup. soup.find_all("div", class_="book") walks that tree and collects every div tag whose class is book into a list — here, a list of three tags. The for loop then visits each one in the order they appeared in the HTML. On the first pass, book is the first div; book.find("h3", class_="title") searches only inside that div (not the whole page) for one matching h3, and .text pulls out just the human-readable content, stripping the tags away. book["data-stock"] reads the div's own attribute the way you would read a value from a dictionary by key. Running this prints:
Python for Beginners - ₹399 - In stock: yes
Data Structures Made Easy - ₹499 - In stock: no
AI for Class 9 - ₹350 - In stock: yesNotice what just happened: three lines of near-identical, tedious-to-copy-by-hand text were extracted in one loop, and the loop would do exactly the same work whether there were 3 books on the page or 3,000.
Step 2 — From loose tags to usable data
Printed text is fine to look at, but a real program usually wants the extracted values stored somewhere it can compute with — sorted by price, filtered to only in-stock items, or saved to a file. The natural container is a list of dictionaries, one dictionary per book:
book_list = []
for book in books:
data = {
"title": book.find("h3", class_="title").text,
"price": book.find("span", class_="price").text,
"in_stock": book["data-stock"] == "yes"
}
book_list.append(data)
print(book_list)Tracing again: book_list starts empty. Each loop pass builds one dictionary — note "in_stock": book["data-stock"] == "yes" compares the attribute's text to the string "yes" and stores the resulting Boolean, not the raw text, so later code can write if item["in_stock"]: instead of comparing strings again. After three passes, book_list holds three dictionaries, and the printed result is:
[{'title': 'Python for Beginners', 'price': '₹399', 'in_stock': True}, {'title': 'Data Structures Made Easy', 'price': '₹499', 'in_stock': False}, {'title': 'AI for Class 9', 'price': '₹350', 'in_stock': True}]This is the real destination of most scraping code: turning scattered HTML tags into a clean Python list your program can sort, filter, or write to a CSV file with the tools you already know from earlier chapters — loops, conditionals, and dictionaries doing the same jobs they always do, just on freshly harvested data.
Step 3 — Fetching a real page instead of a hardcoded string
Everything above worked on an HTML string you typed yourself. To scrape an actual website, you replace that string with one fetched live over the internet, using requests:
import requests
from bs4 import BeautifulSoup
url = "https://example-bookstore.test/catalogue"
headers = {"User-Agent": "SchoolProject-Bot/1.0 (contact: student@example.com)"}
response = requests.get(url, headers=headers)
if response.status_code == 200:
soup = BeautifulSoup(response.text, "html.parser")
books = soup.find_all("div", class_="book")
print(f"Found {len(books)} books")
else:
print("Request failed with status code:", response.status_code)requests.get(url, headers=headers) sends an HTTP request to the server and waits for a reply, storing it as a Response object. Two parts of that object matter here: response.status_code, a three-digit number the server sends back to report what happened — 200 means "here is your page, successfully," 404 means "no page exists at that address," and 403 often means "you are being blocked" — and response.text, the raw HTML, which is exactly the kind of string you built by hand in Step 1. The headers dictionary sends a User-Agent, a short line identifying who is making the request; sending an honest one that names your project, rather than pretending to be a regular browser, is considered good scraping etiquette. From this point forward, every technique from Steps 1 and 2 — find_all, reading attributes, building dictionaries — works identically, because both a hand-typed string and a freshly fetched page are, to BeautifulSoup, just HTML text to build a tree from.
Selecting tags the CSS way
Real pages often have deeply nested, messier HTML than the tidy example above, and repeatedly calling .find() inside .find() gets clumsy. BeautifulSoup also understands CSS selectors — the same short-hand pattern language web designers use to target tags — through the .select() method:
titles = soup.select("div.book > h3.title")
for t in titles:
print(t.text)Read "div.book > h3.title" as: "a div with class book, and then, as a direct child (that is what > means), an h3 with class title." Running this on the same three-book HTML prints the three titles, one per line, in document order — Python for Beginners, then Data Structures Made Easy, then AI for Class 9. A single selector string here replaces a nested, multi-line block of .find() calls, which is why most working scraping code favors .select() once the target tags are more than one or two levels deep.
Looping over pages, politely
Product catalogues and result lists are rarely one page — they are usually split across pages numbered in the URL, like ...catalogue?page=1, ...catalogue?page=2, and so on. Scraping several pages just wraps the fetch-and-parse steps in an outer loop:
import time
all_books = []
for page_number in range(1, 4):
url = f"https://example-bookstore.test/catalogue?page={page_number}"
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
for book in soup.find_all("div", class_="book"):
all_books.append(book.find("h3", class_="title").text)
time.sleep(2)
print(f"Collected {len(all_books)} titles from 3 pages")The line time.sleep(2) pauses the program for two seconds before requesting the next page. This is not a performance bug — it is deliberate. A real website's server has to do real work to answer every request; a script firing hundreds of requests per second looks, from the server's side, indistinguishable from an attack, and can slow the site down for genuine visitors. A short pause between requests is standard, responsible scraping practice, not an optional extra.
Misconception 1: "Whatever I see in the browser is what my script will get"
This is false for a large and growing share of modern websites, and it is the single most common reason a scraping script "finds nothing" even though the data is clearly visible on screen. Many sites send the browser a nearly empty HTML shell along with a JavaScript program; the browser then runs that JavaScript, which fetches the real data separately and builds the visible page piece by piece. When requests.get() fetches such a page, it downloads only that initial shell — it does not run JavaScript at all — so response.text may contain almost none of the content you can see when you open the same URL in an actual browser. The fix is not more BeautifulSoup; it requires a different tool that can drive a real browser engine (such as Selenium or Playwright) to let the JavaScript run first, or, better, finding whether the site offers the data through a documented API that returns it directly. Before writing a single line of scraping code for an unfamiliar site, it is worth checking the raw HTML — for instance by viewing the page source, not the rendered page — to see whether the data you want is actually present in the text requests would receive.
Misconception 2: "If I can see it in my browser for free, I'm free to scrape and reuse it"
Being technically able to fetch a page's HTML is not the same as having permission to copy, store, or republish its data. Two things are worth knowing before scraping any real site. First, most sites publish a file at /robots.txt (for example, example.com/robots.txt) listing which parts of the site automated programs are asked not to visit — it is a courtesy convention, not a lock, but ignoring it is a well-recognized sign of bad-faith scraping. Second, a site's Terms of Service may separately restrict automated data collection or reuse of its content even for paths robots.txt does not mention, and copyright still applies to the underlying text, images, and data regardless of how easily it was copied. Responsible scraping practice, in order of preference, is: check if the site offers an official API first (an API returns clean, structured data — usually JSON — exactly meant for programs to consume, and is far more stable than parsing HTML that might change its tag names next month); if there is no API, check robots.txt and the Terms of Service; and if scraping is appropriate, identify your script honestly via the User-Agent header, request no faster than a human clicking would, and never attempt to scrape personal data about individuals.
Where this connects in your CS toolkit
Nothing in this chapter introduced a new way of thinking about control flow — the for loop, the dictionary, the conditional, and the function are exactly the tools from earlier Python chapters. What is new is a source of data: instead of typing values into a list yourself, or reading them from a file you already had, you now have a way to acquire data from the outside world programmatically. This acquisition step is the entry point of almost every real-world data project — it is why, in the CBSE Artificial Intelligence curriculum's AI Project Cycle, "Data Acquisition" appears as its own stage before data can be cleaned, explored, or modeled, and web scraping is one of the standard methods listed for gathering data that is not already sitting in a neat spreadsheet.
Check your understanding
- Given
<p class="score">87</p>stored as the stringhtml, what doesBeautifulSoup(html, "html.parser").find("p", class_="score").textevaluate to, and what Python type is it — an integer or a string? (Trace it:.textalways returns a string, so it is"87", not the number 87 — code that wants to compare or add it must first convert it withint().) - In the three-book example, if you changed
soup.find_all("div", class_="book")tosoup.find_all("div")with no class filter, and the page's<body>also contained one unrelated<div class="ad">...</div>, how would the loop's output change, and why? - Why does
book.find("h3", class_="title")inside the loop search only within one book's tags, whilesoup.find_all("div", class_="book")before the loop searches the whole page? What is different about whatbookandsoupeach refer to? - A classmate's scraper returns an empty list from a page that clearly shows twenty product cards in the browser. List two different, specific explanations from this chapter for why that could happen, and how you would check which one it is.
- Rewrite the CSS selector
"div.book > h3.title"to instead select everyspanwith classpricethat is a direct child of adivwith classbook.
Summary
Web scraping automates the manual, repetitive work of visiting a webpage and copying out specific values — something computers do without fatigue or error once the steps are exact. Every scraper is built from two operations: fetching a page's raw HTML text with requests.get(), which succeeds when response.status_code is 200, and parsing that text into a searchable tag tree with BeautifulSoup, which you query with .find(), .find_all(), and CSS-style .select() to reach specific tags by name, class, and attribute, then read with .text or dictionary-style attribute access. Extracted values are normally collected into familiar structures — lists, dictionaries — using the same loops and conditionals from earlier chapters. Two limits matter as much as the technique itself: HTML fetched by requests does not include content a page builds later with JavaScript, so some sites need a browser-driving tool instead; and permission to view a page is not the same as permission to scrape or reuse its data, so checking robots.txt, the site's Terms of Service, and preferring an official API when one exists are part of doing this correctly, not optional extras.