Is Web Scraping Legal? Ethics, Law and Best Practices

I hear this question at least 5 times a month: 'Is web scraping legal?'. Short answer: usually yes, but how you do it and what you do with the data is what matters. This isn't legal advice — it's a summary of my 3-year experience plus the cases I've followed. For real legal questions, talk to a lawyer.

Core principle — publicly available data

Data visible on the open internet, not behind a login, that anyone with a browser can see — scraping it is legal in most jurisdictions. The classic case here is hiQ Labs v. LinkedIn (US, 2019, reaffirmed 2022). The court held that LinkedIn couldn't block hiQ from scraping publicly accessible profile pages.

Where the red line starts

• Data behind a login — entering credentials and scraping usually violates ToS
• PII under GDPR/CCPA — collecting emails, phones, addresses is risky, especially in Europe
• Copyrighted content — republishing someone else's writing on your own site
• Bypassing paid APIs — they sell an API and you scrape instead
• US CFAA — the 'unauthorized access' definition gets read broadly

Robots.txt — ethical, not legal

Robots.txt is voluntary — it doesn't legally bind your scraper. But ignoring it shows up against you in court — 'they knew the published rules and ignored them'. I always respect robots.txt, even when the law doesn't require it.

Real cases — quick tour

• hiQ v. LinkedIn (2019, 2022) — scraping public data is allowed
• Meta v. Bright Data (2024) — Meta lost, Bright Data continues scraping
• Ryanair v. PR Aviation (2015, EU) — ToS-based scraping bans can be enforced
• Van Buren v. United States (2021) — narrowed CFAA's 'authorized access'
• Clearview AI — multiple GDPR fines ($30M+) for face scraping

GDPR — European personal data is touchy

If your scraped data includes personal info on EU citizens (email, name, phone), GDPR applies. The location of your server doesn't matter — the location of the data subject does. Clearview AI is paying millions for this. Scraping emails for cold outreach is a GDPR violation and enforcement is increasing.

Practical guidance

• Read and respect the site's robots.txt
• Rate-limit yourself — don't hit a site more than 1-2 req/sec
• Identify yourself in your User-Agent (don't anonymise)
• Don't resell scraped data
• Don't collect PII you don't need
• Read the site's ToS — if it forbids scraping, think hard
• Scraping commercially? Talk to a lawyer once

Ethical vs legal — different things

Legal doesn't mean ethical. Illegal doesn't always mean unethical. They're different scales. My personal principle: if the site owner called me directly and asked 'are you scraping us?', could I answer in a way both sides would accept? Yes → continue. No → stop.

Have a scraping project and want to clarify the legal/ethical side? Send the specific use case and we'll work through the risk profile together.