Scrape the web from any MCP agent with ScrapeNest

There's a gap between what AI assistants know and what's on the web right now. Your assistant can write code, plan a migration, draft a proposal. But if you ask it what's on a page today, it guesses from training data or tells you it can't browse.

The Model Context Protocol closes that gap. MCP is an open standard that lets AI clients call external tools. ScrapeNest works with MCP-capable CLI agents and web agents, including Claude, Codex, OpenCode, Cursor, Windsurf, Cline, Mistral Le Chat, DeepSeek, and Z.ai.

We built an MCP server for ScrapeNest. It ships 20 tools that give your assistant the full scraping platform: fetch pages as clean Markdown, extract structured data with CSS selectors, compare two URLs, check your usage, manage monitors, retry failed jobs. Same engines, same credit billing, same EU-only data residency as the API.

We run it. The hosted connection needs no package to install and no key to paste - you add one URL to your client and sign in with the ScrapeNest account you already have.

https://mcp.scrapenest.com/mcp

What it actually does

The headline tool is scrape. Give it a URL and an engine, and it submits a job, waits for the result, downloads the HTML artifact, and converts it to clean Markdown. Not raw HTML with all its nav, scripts, ads, and tracking pixels, but readable content that fits in a context window without burning half your tokens on noise. Hosted results are capped at 40,000 characters to protect the conversation.

That alone is useful, but a scraper that can only fetch one page and hand back text is a party trick. The interesting part is what happens when you give the assistant more verbs.

extract takes CSS selectors or XPath and returns structured JSON. Instead of scraping a product page and asking the model to find the price in the Markdown, you pass {"price": ".price", "title": "h1"} and get {"price": ["29.99"], "title": ["Widget Pro"]} back. The extraction runs server-side, inside the scrape job, so the model never has to parse HTML.

crawl scrapes many pages concurrently and returns Markdown for each. Point it at five documentation pages and you get five clean documents ready for a RAG pipeline or a summary. It's the bulk path.

screenshot captures a PNG. Viewport-only by default (cheaper, faster), full-page if you ask. The image comes back as a content block the model can see. Useful for visual QA, design audits, or "show me what this page looks like."

compare is new and the one we're most excited about. Give it two URLs and it scrapes both in parallel, converts both to Markdown, and runs a unified diff. The output looks like a git diff of the two pages. Compare your staging deploy against production. Compare your pricing page against a competitor's. Compare two versions of a docs page after a rewrite. The assistant sees the diff inline and can reason over what changed.

Not just scraping

An assistant that can scrape but can't check whether the scrape worked, or see its history, is an assistant that flies blind. So the MCP server exposes the same account-level tools your dashboard does.

list_jobs filters your job history by status, engine, tag, URL, or date range. "Show me all failed stealth jobs from this week" is a one-liner. usage pulls your throughput stats: total jobs, success rate, average duration, broken out by hour. resubmit retries a previous job by ID, optionally upgrading the engine. A failed light scrape that might work on stealth is one tool call away.

get_job and get_artifact let the assistant inspect any past job or re-download its artifacts. Every scrape the model runs is a normal ScrapeNest job, which means it shows up in your console, counts toward your usage, and produces the same artifacts. There is no shadow system.

Monitoring from a chat window

If you've used our monitoring feature, the MCP server gives your assistant the full set of controls: create monitors, pause and resume them, edit their cron and detection config, check run history, list detected changes, and pull the actual unified diff for any change.

The workflow this enables is genuinely useful. In one session, you tell your assistant: "Watch this page every hour, and flag me if the pricing table changes." It calls create_monitor with the right cron, engine, URL, and a detection block targeting your selector. In the next session, you ask "Did anything change on competitor pricing?" and it calls monitor_changes, sees a hit, calls get_change_diff, and shows you exactly what moved.

monitor_runs is the operational layer. It shows every fire of a monitor: minted (a job was created), skipped because of quota, or errored. When a monitor is behaving strangely, the run history tells you why before you have to dig.

update_monitor is a full replacement of a monitor's configuration. Change the cron, the URL, the engine, or the detection block without deleting and recreating. The assistant should call get_monitor first to read the current config, then apply the change. That's the pattern we'd use ourselves.

Three engines, same tool call

Every scraping tool takes an engine parameter. Light (1 credit) is an HTTP client with TLS fingerprint impersonation, no browser. Standard (5 credits) is headless Chromium with full JavaScript rendering. Stealth (30 credits) is a fingerprint-hardened browser for targets with serious anti-bot defenses.

The assistant picks the engine per call from the engines available to your plan. Ask it to scrape a static pricing page and it'll use Light. Ask it to screenshot a dashboard that renders client-side and it'll reach for Standard. Tell it a target is protected by Cloudflare and it'll go Stealth. The same decision tree applies whether you're calling the API or talking to your assistant.

Browser engines also accept dismiss_cookies (best-effort banner dismissal that never fails the job), wait_until, navigation_timeout_ms, locale, and actions for driving interactions before capture. Stealth locks down viewport and locale to its fingerprint, and the server validates that before the job leaves, so you get a clear error message instead of a mysterious 422.

Auth: you sign in, that's it

For the hosted service, we chose not to ask you to install a package and feed it an API key. Long-lived keys in config files on laptops are the thing that leaks, and a shared key would have meant everybody's jobs billing to one org.

Instead mcp.scrapenest.com is an OAuth 2.1 resource server. Your client discovers where to authenticate, opens a browser, and you sign in with the same ScrapeNest account you use for the console. The token comes back scoped to you, we verify it before anything is queued, and every tool call runs under your own org with your own scopes and your own credits.

That has some consequences we like. No long-lived ScrapeNest API key sits in a config file. Access follows the member's role and organization, so separate customer organizations stay isolated while teammates retain the access their organization gives them. And because access is tied to the account rather than to a string somebody pasted once, there is no copied API key on a laptop to chase down later.

Getting started

Add the endpoint to your agent or web app, sign in with your ScrapeNest account, and ask it to scrape something. The connection is the same everywhere:

https://mcp.scrapenest.com/mcp

For example, in Claude Code, one command:

claude mcp add --transport http scrapenest https://mcp.scrapenest.com/mcp

Then run /mcp and sign in when it prompts you.

In Claude Desktop or Claude on the web, open Settings, go to Connectors, add a custom connector, and paste the same URL. On Team and Enterprise plans, an organization owner adds the custom connector first. In another CLI agent or web app, add the same URL through its MCP or connector settings. Your browser opens, you sign in, you approve the connection, done.

Either way the next thing you do is ask your assistant to scrape something. It discovers the 20 tools, picks the right one, and you're live.

The full tool reference, engine comparison, browser options, and troubleshooting are in the integration guide.

If you don't have a ScrapeNest account yet, the free tier gives you 1,000 credits for Light and Standard scraping. Monitoring and Stealth depend on your plan. And if you're already a customer, you have everything you need already - your assistant just learned to use it.