Web Scraping Guides & Field Notes
Guides on web scraping, anti-bot detection, browser engines, GDPR compliance, and change monitoring. From the ScrapeNest team.
Introducing recipes: ready-made scrapers with a JSON contract
30 ready-made scrapers for Google, YouTube, Amazon, GitHub, Bluesky, Mastodon, Hacker News, news publishers and more. Send a small input, get schema-validated JSON, pay only for definitive answers. Here's how they work, and the DSL behind them.
How to collect public LinkedIn job listings with ScrapeNest
A practical, login-free workflow for collecting public LinkedIn job-search results, extracting aligned job records, checking completeness, and monitoring a query for new listings.
How to monitor public app stores for brand impersonation signals
Build a defensible public-surface watchlist for lookalike apps and brand abuse: collect candidates, preserve evidence, score signals, monitor change, and send humans only the cases worth reviewing.
Scrape the web from any MCP agent with ScrapeNest
Give any MCP-capable CLI or web agent 20 ScrapeNest tools for live-web research, extraction, comparison, job management, and change monitoring. Add one URL, sign in, and get to work.
The SerpApi ruling, and why the robots.txt answer flips in Europe
A US court just gutted Google's DMCA case against SerpApi, and the coverage is getting the reasoning wrong. Here is what the order actually holds, and why the EDPB is moving in the opposite direction on the exact same question.
Monitoring, without the cron-job graveyard
Recurring scrapes are easy to start and hard to run well: missed windows, overlapping runs, silent failures when you run out of budget. Here's how we built monitoring so none of that is your problem - and so you get told when a page actually changes.
TLS fingerprints are the new User-Agent
Spoofing the User-Agent header stopped working years ago. Here's why your HTTP client gets blocked before it sends a single byte of HTTP, and what to do about it.
When to reach for a real browser (and when not to)
A headless browser is the most expensive way to fetch a page. Here's a practical decision tree for picking the lightest engine that actually works on your target.
Scraping and the GDPR: what a French SaaS actually has to do
Public doesn't mean free-for-all. A grounded look at where web scraping sits under the GDPR, written by a company that has to answer the questionnaire.