Skip to content
ScrapeNest
All posts
·7 min read·by ScrapeNest Engineering

Introducing recipes: ready-made scrapers with a JSON contract

30 ready-made scrapers for Google, YouTube, Amazon, GitHub, Bluesky, Mastodon, Hacker News, news publishers and more. Send a small input, get schema-validated JSON, pay only for definitive answers. Here's how they work, and the DSL behind them.

recipesproductextractiondslmcp

Most scraping jobs aren't original. Somebody, somewhere, is pulling Google results for a keyword, the latest uploads of a YouTube channel, the open issues of a GitHub repository, or today's headlines from Le Monde. They write the same selectors and the same pagination loop, and their scraper breaks the same Tuesday morning when the site ships a redesign.

ScrapeNest has always handled the fetch: proxies, rendering, retries, anti-bot. Today we're taking the next step and handling the scraper itself. Meet recipes, now in beta.

A recipe in one paragraph

A recipe is a ready-made scraper for a well-known site. You send a small input, such as a search query, a repository, or a channel, and get back clean JSON with a stable schema. We maintain the scraper. When the site changes, we fix the recipe, and your integration doesn't change.

curl -X POST https://api.scrapenest.com/v1/recipes/github-issues/runs \
  -H "X-API-Key: $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"owner": "cli", "repo": "cli", "state": "open"}}'

The result arrives like any other job's: by polling GET /v1/jobs/{job_id}/result, or inline in the signed job.completed webhook. You get a list of issues with numbers, titles, labels, authors and dates, already shaped and already validated.

Contracts, not promises

Every recipe publishes one or more contracts: the JSON Schema of the input you send and of the output you receive. Behind a contract runs an implementation, the scraper itself.

The two move at different speeds, on purpose:

  • Implementations change whenever the site does. A new class name, a moved API field or a fresh consent banner gets a new implementation, deployed silently, as long as its output still validates against the contract.
  • Contracts change only for a breaking output change. That ships as a new contract, and the old one keeps working until an announced sunset date. Pin a contract and you're insulated.

Before any output reaches you, the control plane re-validates it against the contract's schema. If a site change slips past us, you get a clear layout_changed outcome. You never get a half-empty record that looks fine and quietly poisons your pipeline.

You pay for answers, not attempts

A recipe run ends with an outcome, and the outcome decides the bill:

  • Billed: ok (results found), ok_empty (the site genuinely has no results), not_found (the repository, video or account doesn't exist).
  • Free: blocked, rate_limited, upstream_error, layout_changed, schema_invalid, timeout, error.

"Nothing matches your search" is a real answer, so it's billed. "We couldn't get through", "the site changed" and "we produced something wrong" are our problems, so they're free. Recipes that paginate reserve credits for the maximum pages you ask for and charge only for the pages actually fetched.

The catalog: 30 recipes in eight categories

  • Search engines: google-serp, google-autocomplete
  • Video: youtube-search, youtube-video, youtube-subtitles, youtube-channel-feed
  • E-commerce: amazon-search, amazon-product
  • Social & communities: Bluesky posts, profiles and threads; Mastodon hashtags and trends; Lemmy communities; public Telegram channels; Hacker News
  • Code hosting: GitHub issues, repositories, repository search and releases
  • Developer tools: npm and PyPI packages, Stack Overflow search
  • News: headlines from Le Monde, Le Figaro, L'Humanité, BBC News, The Guardian and the New York Post, read from the publishers' own RSS feeds
  • Reference: Wikipedia summaries

A couple of deliberate lines. Social recipes return only what anyone can see without an account, so networks that require a login to read are not in the catalog. News recipes return headlines, links and summaries from the publisher's own feed, never the article text.

Every recipe carries a health status in the console and in GET /v1/recipes. Canary runs hit the live site every 15 minutes. If they start failing, the recipe shows as Degraded before you have to file a ticket, and failed runs keep costing nothing while we fix it.

How we build them: a DSL for scrapers

We didn't want 30 hand-written Python scrapers, each with its own retry logic and its own idea of what "empty" means. So we built a small declarative language for describing a scraper, and every recipe in the catalog is written in it.

Here is most of the Hacker News recipe:

# Light engine: the official HN Search API.
slug: hackernews-search
contract: 1
tier: light
steps:
  - request:
      method: GET
      url: "https://hn.algolia.com/api/v1/search?query={{ input.q | urlencode }}&tags={{ input.type | urlencode }}"
assertions:
  page_ok: [{ status: 200, jsonpath_exists: "$.hits" }]
  empty: [{ status: 200, body_matches: '"nbHits":\s*0\b' }]
  rate_limited: [{ status: 429 }]
extract:
  items:
    type: records
    container_jsonpath: ["$.hits[*]"]
    fields:
      id: { jsonpaths: ["$.objectID"], required: true }
      title: { jsonpaths: ["$.title"], required: true, transform: [html_text] }
      hn_url: { jsonpaths: ["$.objectID"], format: "https://news.ycombinator.com/item?id={}" }
      points: { jsonpaths: ["$.points"], default: 0 }
output:
  results: { from: items }
  captured_at: { meta: captured_at }
invariants:
  items:
    min_records: 1
    distinct: [id]
    absolute_url: [hn_url]

Read it top to bottom and you have the whole scraper:

  • steps says how to fetch. On the Light engine that's an HTTP request, optionally followed by paginate (link headers, next-page URLs, cursors or page numbers) or follow, which picks one item from a list and fetches it. Picking a video's caption track in your language is a follow. On the browser engines the steps are navigate, click, fill, scroll, wait_for_selector, dismiss_consent and friends.
  • assertions say what the response means: a real answer, an honest empty result, a 404, a bot wall, a rate limit. This is the part most scrapers skip, and it's what makes outcome-based billing possible.
  • extract pulls fields with CSS, XPath or JSONPath. Every locator is a fallback chain, so a recipe can survive a site that's halfway through a migration.
  • output maps extractions to the published contract.
  • invariants guard against drift: at least one record, distinct ids, absolute URLs. They never reach you; they're how we notice a site moved before you do.

Inputs are templated ({{ input.q | urlencode }}), but only into URLs, request bodies and form values, never into selectors, scripts or hostnames. A recipe input can change what you search for, not what the scraper does.

Notice the tier: light. The DSL makes the cheapest path the natural one. Official APIs, JSON embedded in the HTML, and publisher feeds come first. A recipe that drives a real browser has to justify, in its README, why none of those work. That's why most of the catalog runs on the 1-credit Light engine, and only a handful need the Stealth browser.

Tested like code, because it is code

Every recipe ships with 3 to 10 golden cases and captured fixtures. In CI we validate each definition against the DSL's JSON Schema and replay it offline against the captured pages, so a change that breaks extraction never merges. In production, the same golden inputs run as live canaries every 15 minutes, and those canaries feed the health status you see.

Built for agents too

Recipes are exposed through the ScrapeNest MCP server as four tools: list_recipes, get_recipe, run_recipe and get_recipe_result. An agent can browse the catalog, read a recipe's input schema, run it and get validated JSON back, without scraping a page into Markdown and guessing where the numbers are.

List the open bug issues on cli/cli with their labels.

That's one run_recipe call on github-issues. The answer comes with a contract, and the agent never writes a selector.

Coming soon: write your own

The DSL started as our internal tool for building the catalog fast. It's already the part we're proudest of, and we've designed it from day one so it doesn't have to stay internal.

Soon you'll be able to write your own recipes for the sites you care about, in the same language, validated by the same JSON Schema and replayed by the same offline test runner we use, then share them. Your recipes will run under your organization, your quota and your plan, with their own health checks.

If there's a site you'd like as a recipe, ours or yours, tell us. It directly shapes what we build next.

Try it

Recipes are in beta and available on every plan. The few that need the Stealth browser (Google search, Amazon) require a plan with Stealth. Open Recipes in the console to run one from a form generated from its input schema, or start with the recipes documentation. The full catalog is on the recipes page.