Python Web Scraping in 2026 — Playwright vs Selenium vs BeautifulSoup

I've shipped over thirty scrapers in the last few years — from a tiny price tracker for a friend's shop to a system pulling millions of pages a day. This is the playbook I actually use to pick between Playwright, Selenium and BeautifulSoup. No theory, just what works.

Short answer if you're in a hurry

• Static HTML, no JavaScript, no login → requests + BeautifulSoup
• JS-rendered page, no aggressive anti-bot → Playwright
• Tough anti-bot (Cloudflare, PerimeterX, DataDome) or complex auth flow → Playwright with playwright-stealth (and residential proxies)
• Old enterprise app that thinks it's still 2012 → Selenium, sadly

BeautifulSoup — still my fastest tool

First thing I do on any new target: right click, View Page Source. If the data I want is in there, I don't need anything fancy. requests to pull the HTML, BeautifulSoup to parse it, done. A hundred lines of code, fifty megs of RAM, fifty thousand pages an hour on a single box.

import httpx
from selectolax.parser import HTMLParser

with httpx.Client(timeout=10, headers={"User-Agent": "Mozilla/5.0"}) as client:
    r = client.get("https://example.com/listings")
    tree = HTMLParser(r.text)
    for item in tree.css("article.listing"):
        title = item.css_first("h3").text(strip=True)
        price = item.css_first(".price").text(strip=True)
        print(title, "—", price)

Tiny note — I usually skip BeautifulSoup these days in favour of selectolax. It's a C-backed parser, five to ten times faster, with the same CSS selector API. At a million pages an hour, the difference matters. BeautifulSoup is still great for learning, but selectolax wins in production.

Playwright — my default in 2026

Most modern sites are React or Vue or Next.js — the initial HTML is mostly empty and data shows up after JavaScript runs. Plain requests is useless there. Playwright spins up a real browser (Chromium, Firefox or WebKit), waits for the page to settle, then reads the DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as pw:
    browser = pw.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/dashboard", wait_until="networkidle")
    page.wait_for_selector("[data-loaded='true']")
    rows = page.eval_on_selector_all(".row", "els => els.map(e => e.innerText)")
    browser.close()

Where Playwright pulls ahead of Selenium: auto-waiting (you write wait_for_selector, the runtime handles the rest), network interception (block ads, trackers, anything heavy), and clean access to the browser's underlying APIs. On one scrape job, the target page fired 47 analytics requests per pageview. I blocked them with route.abort() and tripled my throughput overnight.

Selenium — only for one reason

I've still reached for Selenium a few times — every time for the same reason. Some legacy enterprise ERP or banking app that only works through ActiveX and old Java applets, and the QA team only has Selenium-Java drivers. For a greenfield project, there is no reason to pick Selenium over Playwright in 2026. Playwright is newer, faster, and needs less boilerplate.

Anti-bot — where things get spicy

Cloudflare, DataDome, PerimeterX, Akamai — they spot a vanilla headless browser in milliseconds. Open Playwright the default way and the JS fingerprint screams 'automated'. I usually layer a few things.

• playwright-stealth plugin — patches obvious tells like navigator.webdriver
• Residential proxies (Bright Data, Oxylabs) — datacenter IPs are basically worthless
• Real user-agent rotation — mobile/desktop, Chrome/Safari, actual current versions
• Human-like timing — page.wait_for_timeout(random.randint(800, 2400)) between actions
• Mouse movement on the page — just moving the cursor without clicking helps surprisingly often

My default stack today

When I start a fresh scraping project, the default setup is: try httpx + selectolax first. If it doesn't work → Playwright with Chromium. Hit anti-bot → add playwright-stealth, residential proxies, randomised timing. Data goes into Postgres, parallelism handled by a Redis queue, errors tracked in Sentry. Everything runs in Docker and I keep a Grafana board open during big runs.

Wrapping up

Web scraping isn't a tool, it's a process. You don't need a single favourite, you need the right one for each job. My rule of thumb: start with the simplest thing that could work, escalate only when forced. If you're working on something specific and want a second pair of eyes, get in touch — I've shipped 30-plus scrapers and I'm happy to take a look.