Bypassing Cloudflare and Captchas — An Ethical Scraper's Playbook

Before we start: bypassing anti-bot is a technical question, an ethical question, and sometimes a legal one. This post is for sites you own, sites that have given you permission, or genuinely public data — not for stealing personal info, scraping behind logins, or anything that violates a site's terms of service.

How Cloudflare detects you

Cloudflare's bot protection has multiple layers. First layer — IP reputation. Datacenter IPs (AWS, Hetzner, DigitalOcean) automatically get low scores. Second layer — TLS fingerprint (JA3/JA4 hash). Python's requests library produces a TLS handshake that looks nothing like a real browser, and they spot it instantly.

Third layer — JavaScript challenge. Cloudflare injects JS that checks how your browser renders canvas, what plugins exist, which fonts are installed. Headless Chrome forgets most of these by default. Fourth layer — behavior. If the mouse moves in perfect straight lines, never pauses before clicks, scrolls at constant speed — they know it's a bot.

Layer 1: IP — residential proxies are mandatory

You cannot beat Cloudflare from a datacenter IP. Bright Data, Oxylabs, Smartproxy — these route through residential and mobile devices. Starts at $200-300/month for 10 GB. Expensive but unavoidable. For smaller projects I've been testing $50/month providers (IPRoyal, ProxyEmpire) — quality drops a bit, but they work.

Layer 2: TLS fingerprint

Python's httpx and requests use the default OpenSSL, fingerprint screams 'Python'. To fix this I use curl_cffi — it impersonates the real curl/Chrome TLS stack. Same API as httpx, but the fingerprint looks like Chrome 120.

from curl_cffi import requests

# Chrome 120 fingerprint imitasiyası
r = requests.get(
    "https://protected-site.com",
    impersonate="chrome120",
    proxies={"https": "http://user:[email protected]:8080"},
)
print(r.status_code, len(r.text))

Layer 3: Playwright + stealth

If the page renders JS (most modern sites do), you need Playwright. Vanilla Playwright gets caught in milliseconds. Use playwright-stealth or playwright-extra — hides the navigator.webdriver flag, fixes the plugin/font list to match real Chrome, randomises WebGL/Canvas fingerprints.

import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';

chromium.use(stealth());

const browser = await chromium.launch({ headless: true });
const ctx = await browser.newContext({
  proxy: { server: 'http://residential:port', username: 'u', password: 'p' },
  viewport: { width: 1366, height: 768 },
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...',
});
const page = await ctx.newPage();
await page.goto('https://protected-site.com');

Layer 4: act human

Simplest and most effective way to hide bot behaviour: slow down. Wait 800-2400ms before clicks, move the mouse with a slight curve to the target, scroll at human speed, take a 30-second break every 5-10 pages. A scraper that imitates 10,000 user-actions per hour gets banned. One that does 200 pages per hour never does.

Captcha — last resort CAPSOLVER

If you've stacked all the above layers and still get captchas, your last resort is captcha-solving services. 2captcha, CAPSOLVER, Anti-Captcha — about $1.50 per 1,000 hCaptcha solves, $2-3 for reCAPTCHA v3. You send the captcha to their API, get a token back in 10-20 seconds, inject it into the page. Ethics note: this is human labour (often in low-wage countries) and it directly subverts the site's protection. Think twice.

The ethical line — where to stop

• The site's `robots.txt` disallows your path → STOP
• Personal data (email, phone, address) sits behind a login → STOP
• You're adding more than 1% load to the site → THROTTLE, don't stop
• You're going to resell the data → STOP
• The Terms of Service explicitly forbid scraping → STOP (or talk to a lawyer)