Suppose your school library has shortlisted ten reference books for the Class 8 CBSE Computer Science exam, and you want the cheapest one before Diwali sale prices disappear. You open an online bookstore, find the price of book one, write it in a notebook, go back, search for book two, write its price, go back again... After doing this for ten books across three different stores, you have thirty numbers copied by hand — and if the store updates a price tomorrow, you have to repeat the entire thing. This is exactly the kind of repetitive, mechanical, rule-following task a computer should be doing for you instead of a human doing it with a mouse and a notebook. The technique of writing a program that automatically visits a webpage and pulls out specific pieces of information from it — instead of a person reading the page and copying by hand — is called web scraping. In this chapter you will learn to do it properly, using a Python library called BeautifulSoup, and you will also learn where scraping breaks down and where it becomes unethical or illegal to use it.
A webpage is not a picture — it is a tree of tags
Before you can extract anything from a webpage, you need to understand what a webpage actually is underneath the pretty rendering you see in a browser. It is not an image. It is a plain text file written in HTML (HyperText Markup Language), where every piece of content sits inside a pair of tags — an opening tag like <li> and a matching closing tag like </li>. Tags can sit inside other tags, and this nesting is not decorative — it defines a strict hierarchy, the same way a folder on your computer can contain sub-folders, which can contain more sub-folders. A tag that sits directly inside another tag is called that tag's child, and the containing tag is the child's parent. Two or more tags that share the same immediate parent are called siblings of each other — they live at the same level of nesting, side by side.
Here is a small, realistic webpage for an online bookstore. Read it carefully — you will use this exact page for every example in this chapter.
<!DOCTYPE html>
<html>
<head>
<title>AICI Book Corner</title>
</head>
<body>
<h1 id="store-name">Welcome to AICI Book Corner</h1>
<ul class="catalogue">
<li class="book" data-price="399">
<span class="title">Discovering Python</span>
<span class="author">R. Sharma</span>
</li>
<li class="book" data-price="549">
<span class="title">AI for Everyone</span>
<span class="author">A. Verma</span>
</li>
<li class="book" data-price="299">
<span class="title">CBSE Computer Science X</span>
<span class="author">NCERT</span>
</li>
</ul>
</body>
</html>
Trace the nesting by hand before moving on. The <ul class="catalogue"> tag contains three <li> tags — so all three <li> tags are children of that one <ul>, and they are siblings of one another, because they share the same parent. Look inside just the first <li>: it contains two <span> tags — title and author. Those two spans are children of that particular <li>, and siblings of each other. Notice also that each <li> carries an attribute, data-price="399" — attributes are extra name-value pairs written inside the opening tag itself, separate from the text the tag wraps. The diagram below draws this exact page as a tree, with the parent-child lines showing which tag contains which.
This tree structure is the single most important idea in this whole chapter. Every tool BeautifulSoup gives you — finding a tag, reading its text, walking to its sibling, reading its attributes — is really just a way of moving around this tree. If you can point at any tag in the diagram above and correctly name its parent, its children, and its siblings, you already understand the data structure that BeautifulSoup is built on.
Why not just search the HTML text with string methods?
A common instinct for a beginner is to skip a proper HTML parser entirely and try to pull out the price using plain string operations — for example, splitting the raw HTML text on data-price=" and reading the characters that follow, or writing a regular expression to hunt for digits after that phrase. This looks like it works on your one test page, but it is fragile in a way that is not obvious until it breaks: real HTML from a real website is rarely as neatly formatted as the example above. Attributes can appear in a different order (class="book" data-price="399" versus data-price="399" class="book"), tags can be self-closing or omit optional closing tags, extra whitespace and line breaks can appear anywhere, and attribute values can use single quotes instead of double quotes. A plain string search that happens to work today can silently return the wrong text — or no text at all — the moment the page's formatting changes even slightly, and it will not warn you when it does. This is exactly the problem a proper HTML parser solves: instead of treating the page as a flat string of characters, it reads the tag structure and builds the actual tree you saw in the diagram, so you can ask "give me the span with class title inside this specific li" instead of guessing where that text sits in a string.
Meet BeautifulSoup
BeautifulSoup is a Python library (its package name is bs4) that takes a string containing raw HTML and turns it into a navigable tree object called a soup. Once you have a soup, you can search it by tag name, by attribute, or by CSS-style selector, and BeautifulSoup hands back tag objects you can read text and attributes from. You install it with pip install beautifulsoup4, and you build a soup by handing it your HTML text plus the name of a parser to use — "html.parser" is Python's own built-in parser and needs no extra installation, which is what every example in this chapter uses.
from bs4 import BeautifulSoup
# html_doc holds the exact page HTML shown earlier in this chapter
soup = BeautifulSoup(html_doc, "html.parser")
print(soup.title)
print(soup.title.string)
print(soup.h1.text)Trace this line by line. soup.title is a shortcut for "find the first <title> tag anywhere in the tree" and returns the whole tag object, markup included, so it prints exactly <title>AICI Book Corner</title>. soup.title.string goes one step further and returns just the text sitting inside that tag, as a plain string: AICI Book Corner. soup.h1.text uses the same idea for the <h1> tag and returns its inner text, Welcome to AICI Book Corner. So the three lines print:
<title>AICI Book Corner</title>
AICI Book Corner
Welcome to AICI Book Cornerfind() versus find_all() — a mix-up that produces a real error
BeautifulSoup gives you two closely related search methods, and confusing them is the single most common beginner mistake. soup.find(tag_name) searches the whole tree and hands back exactly one tag object — the first match it encounters, even if five more matches exist further down the page. soup.find_all(tag_name) searches the whole tree and hands back a list of every matching tag — a list of one item if only one tag matches, and an empty list if none match, but always a list, never a single tag on its own.
first_book = soup.find("li")
print(first_book.find("span", class_="title").text)
all_books = soup.find_all("li")
print(len(all_books))The first line finds only the very first <li> in the whole document — the "Discovering Python" one — and inside it, finds the span whose class is title, printing Discovering Python. The second line collects every <li> tag on the page into a list of three items, so len(all_books) prints 3.
Now watch what happens when you forget this distinction and try to treat the output of find_all() as if it were a single tag:
books = soup.find_all("li")
print(books.text)This does not silently give you the wrong answer — it crashes, and BeautifulSoup's error message is unusually direct about the exact mistake you made:
AttributeError: ResultSet object has no attribute 'text'. You're
probably treating a list of elements like a single element. Did you
call find_all() when you meant to call find()?A list does not have a .text property — only an individual tag does — so you must loop over the list and read .text from each item inside it:
for book in books:
print(book.find("span", class_="title").text)Discovering Python
AI for Everyone
CBSE Computer Science XAttribute values are always strings — even when they look like numbers
Reading an attribute off a tag uses square-bracket notation, the same way you would read a value out of a Python dictionary: book["data-price"]. Every attribute value BeautifulSoup gives you back is a Python string, regardless of what the value looks like — "399" is the three characters 3, 9, 9, not the number 399. This causes a genuinely subtle bug, because unlike most mistakes, it does not crash — it just silently gives you the wrong answer. Suppose you want the total price of all three books:
books = soup.find_all("li", class_="book")
total = ""
for book in books:
total = total + book["data-price"]
print(total)Trace it carefully. total starts as an empty string. On each pass through the loop, + between two strings does not add numbers — it glues characters together. So total goes "" → "399" → "399549" → "399549299". The program prints 399549299: not a price, just three price-tags mashed into one long digit string, and Python never complains because string concatenation with + is perfectly legal code — it is simply not the calculation you meant to write. The fix is to explicitly convert each value to an integer with int() before adding it, and to start the running total as the number 0 instead of an empty string:
total = 0
for book in books:
total = total + int(book["data-price"])
print(total)Now trace the arithmetic: 0 + 399 = 399, then 399 + 549 = 948, then 948 + 299 = 1247. The program correctly prints 1247, the true total price in rupees. The lesson generalises far beyond price tags: any number you extract from a webpage arrives as text and must be deliberately converted before you do arithmetic with it — BeautifulSoup has no way of knowing that "399" was "meant" to be a number rather than, say, a pin code or a roll number.
Walking the tree: parents, siblings, and multi-valued attributes
Because a soup is a tree, not just a flat list of tags, you can move relative to a tag you already have, instead of always searching from the top of the document. Two useful moves are asking a tag for its parent, and asking it for its next sibling:
first_li = soup.find("li")
second_title = first_li.find_next_sibling("li").find("span", class_="title")
print(second_title.text)
print(first_li.parent["class"])find_next_sibling("li") starting from the first <li> walks forward to the next <li> that shares its parent — the "AI for Everyone" one — so the first line prints AI for Everyone. The second line asks the first <li> for its parent's class attribute, and prints ['catalogue'] — as a one-item list, not the plain string 'catalogue'. This is not a mistake in the example; BeautifulSoup deliberately treats a handful of HTML attributes that are allowed to hold multiple space-separated values — class is the most common one, since a real tag can legally be class="book featured sale" — as Python lists, even when only one value is present.
There is a related trap worth naming explicitly: BeautifulSoup also exposes a plain .next_sibling property (no find_ prefix), and beginners often reach for it expecting the same result as find_next_sibling(). It is not the same. .next_sibling returns the very next node in the tree exactly as parsed — and because the HTML source has a line break and spaces between </li> and the next <li> (indentation you cannot see once rendered in a browser, but which is still text sitting in the tree), that next node is usually just a short whitespace string, not a tag at all. find_next_sibling() is the safe choice because it explicitly skips over any text nodes and keeps looking until it finds a tag matching your criteria.
Selecting with CSS-style selectors
Everything so far has used find/find_all with a tag name plus keyword arguments. BeautifulSoup also offers a second, equally valid API — .select() — which accepts the same selector syntax you would write in a CSS stylesheet:
titles = soup.select("ul.catalogue > li > span.title")
for t in titles:
print(t.text)The > here means "direct child of," so this selector reads as "every span with class title, that is a direct child of an li, that is a direct child of a ul with class catalogue." It prints the same three titles as before. Neither API is "more correct" than the other — find/find_all tends to read more naturally for simple single-condition lookups, while .select() tends to be faster to write once a search involves several levels of nesting, since you already know CSS selector syntax from styling webpages.
From a live webpage to a soup
Every example so far has parsed a Python string you already had. To scrape a real page on the internet, you first need to download its HTML, which is the job of a separate library called requests (pip install requests):
import requests
from bs4 import BeautifulSoup
response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)requests.get(url) sends an HTTP request to the server and waits for the page to come back; response.text is the raw HTML the server sent, as one long string — the exact same kind of string as the html_doc variable you have been parsing all chapter, just fetched over the network instead of typed by hand. example.com is a domain the Internet Assigned Numbers Authority reserves specifically for use in documentation and examples like this one, so it is a safe, stable target to practice on; running this code prints whatever its live <title> tag currently contains.
There is an important limit to what this technique can see, and it is worth naming precisely rather than glossing over: requests.get() only downloads the HTML the server sends in its very first response. Many modern websites — particularly ones with live-updating listings, infinite scroll, or content that "loads in" a moment after the page appears — build part of their content afterwards, in the browser, using JavaScript. That JavaScript-generated content never appears in response.text, because requests does not run JavaScript at all; it only fetches the initial document. If you print soup after scraping such a page and the data you wanted is simply missing, this is almost always the reason. Scraping that kind of page needs a different tool that can actually drive a browser (such as Selenium or Playwright), which is beyond what BeautifulSoup alone can do — BeautifulSoup only ever parses the HTML text you hand it.
Scraping responsibly
Being able to write a scraper does not mean every page is fair game to scrape. Most well-run websites publish a file called robots.txt at their root (for example, a site's own /robots.txt path) that states which parts of the site automated programs are and are not permitted to crawl — checking it before scraping a new site is standard practice, not optional. Separately, a site's Terms of Service may restrict automated data collection even on pages robots.txt does not block, and the text, images, and prices on a page are typically the site owner's copyrighted or proprietary content, not free data for you to republish. Practically, this means: read a site's robots.txt and terms before scraping it for anything beyond personal learning, never send requests in a tight loop that hammers a server (space repeated requests apart using time.sleep()), and prefer storing what you scrape locally rather than re-downloading the same page repeatedly. None of the examples in this chapter used a real external website for exactly this reason — the sample bookstore page lives entirely in a Python string you control.
From scraped tags to a dataset
The real reason scraping matters for data science is that it turns messy, human-readable HTML into clean, structured data you can actually work with — the kind of structure a spreadsheet or a database table expects, one record per row:
records = []
for book in books:
records.append({
"title": book.find("span", class_="title").text,
"author": book.find("span", class_="author").text,
"price": int(book["data-price"])
})
print(records)[{'title': 'Discovering Python', 'author': 'R. Sharma', 'price': 399}, {'title': 'AI for Everyone', 'author': 'A. Verma', 'price': 549}, {'title': 'CBSE Computer Science X', 'author': 'NCERT', 'price': 299}]Each dictionary in that list is one row of data — its keys (title, author, price) are column names, and its values are that row's data, with price already converted to a genuine integer so it is ready for arithmetic, sorting, or being written straight into a CSV file or a database table. This is the actual bridge between "a webpage a human reads" and "a dataset a program can analyse" — and it is the point of everything this chapter has built up to.
Check your understanding
- In the bookstore page, is
<title>a sibling of<h1>? Explain using the parent of each tag. - What does
soup.find_all("span")return: how many items, and of what type is each item'sclassattribute value? - Predict the exact output of
soup.select("li")[1]["data-price"], and state whether it behaves like a number or like text in Python. - A classmate writes
price_total = soup.find("li")["data-price"] + 100and gets aTypeError. Explain exactly why, and fix the line. - Why does
first_li.next_siblingoften not give the result a beginner expects, whilefirst_li.find_next_sibling("li")does?
Answers: (1) No — <title>'s parent is <head>, while <h1>'s parent is <body>; tags are only siblings if they share the same parent, and these two do not. (2) Six items — two span tags per li, three li tags; each span's class attribute comes back as a one-item list, e.g. ['title'], because class is treated as a multi-valued attribute. (3) It prints the string "549" (the second li in document order, index 1), and it behaves like text — it cannot be added to a number without wrapping it in int() first. (4) soup.find("li")["data-price"] is the string "399", and Python does not allow + between a string and an integer; the fix is int(soup.find("li")["data-price"]) + 100. (5) Because the HTML source has whitespace (a newline and indentation) sitting between one <li>'s closing tag and the next <li>'s opening tag, and .next_sibling returns the very next node in the tree exactly as parsed — which is usually that whitespace text, not a tag; find_next_sibling() explicitly skips text nodes and searches for a matching tag instead.
Summary
- A webpage is a tree of nested tags: a tag directly inside another is its child; the containing tag is its parent; tags sharing a parent are siblings.
- BeautifulSoup (
bs4) turns raw HTML text into a searchable tree object (a "soup") using a parser such as"html.parser", and is far more reliable than searching HTML with plain string operations. find()returns one tag (the first match);find_all()always returns a list, even for zero or one matches — calling.textdirectly on afind_all()result raises anAttributeError.- Attribute values, read with
tag["attr"], are always strings — convert withint()orfloat()before doing arithmetic, or additions will silently concatenate text instead of adding numbers. tag["class"]returns a list, since HTML allows multiple space-separated values for that attribute.find_next_sibling()safely skips whitespace text nodes between tags; the plain.next_siblingproperty does not..select()offers the same searches using CSS selector syntax, including the direct-child combinator>.requests.get(url).textfetches a live page's raw HTML for BeautifulSoup to parse — but it never executes JavaScript, so content a page builds after loading is invisible to it.- Always check a site's
robots.txtand Terms of Service, and avoid rapid repeated requests, before scraping anything beyond a page you built yourself for practice. - Scraped data becomes genuinely useful once it is reshaped into structured records — a list of dictionaries, one per item, ready for a CSV file or a database table.
Think About It
Think about this: How would you explain web scraping with beautifulsoup to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.