Python Web Scraping Guide: Extract Data from Websites

What is Web Scraping?

Web scraping is automating the extraction of data from websites. It powers price comparison sites, research tools, news aggregators, and data analysis projects. Python is the dominant language for scraping due to its excellent libraries and readable syntax.

Legal and Ethical Considerations

Before scraping any site: check the robots.txt file (site.com/robots.txt) for scraping policies, review terms of service, respect rate limits (don't overwhelm servers), and only scrape public data. Never scrape personal data or behind-login content without authorization.

Method 1: BeautifulSoup (Static Sites)

import requests
from bs4 import BeautifulSoup

# Fetch a page
url = "https://example.com/products"
headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get(url, headers=headers)

# Parse HTML
soup = BeautifulSoup(response.text, "html.parser")

# Extract elements
titles = soup.find_all("h2", class_="product-title")
for title in titles:
    print(title.text.strip())

# Extract with CSS selector
prices = soup.select(".price span")
for price in prices:
    print(price.text)

Practical Example: Scraping News Headlines

import requests
from bs4 import BeautifulSoup

def get_headlines(url):
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(response.text, "html.parser")
    headlines = []
    for h2 in soup.find_all("h2"):
        text = h2.text.strip()
        if len(text) > 20:  # Filter short strings
            headlines.append(text)
    return headlines[:10]

headlines = get_headlines("https://timesofindia.indiatimes.com")
for i, h in enumerate(headlines, 1):
    print(f"{i}. {h}")

Method 2: Selenium (Dynamic/JavaScript Sites)

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")

# Wait for element to load
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "price")))
print(element.text)
driver.quit()

Saving Scraped Data

import pandas as pd

data = [{"title": "Product 1", "price": "₹999"},
        {"title": "Product 2", "price": "₹1499"}]

df = pd.DataFrame(data)
df.to_csv("products.csv", index=False)
print(df)