Recipes · beta
Ready-made scrapers.Clean JSON, every time.
Pick a site and send a small input, such as a search query, a repository, or a channel. You get back JSON that matches a published schema. We maintain the scraper: when the site changes, we fix the recipe and your integration doesn't change.
30 recipes/8 categories/pay for answers only
$ curl -X POST https://api.scrapenest.com/v1/recipes/hackernews-search/runs \
-H "X-API-Key: $SCRAPENEST_API_KEY" \
-d '{"input": {"q": "rust", "type": "show_hn"}}'
# GET /v1/jobs/{job_id}/result
{
"recipe": { "slug": "hackernews-search", "contract": 1, "outcome": "ok" },
"output": {
"total_hits": 214,
"results": [
{ "title": "Show HN: …", "points": 412, "comments": 96,
"hn_url": "https://news.ycombinator.com/item?id=…" },
…
]
}
}A scraper you never have to fix
Every recipe comes with a contract, a test suite and a billing rule that only charges for real answers.
Versioned contracts
Each recipe publishes input and output JSON Schemas. Pin a contract and it stays stable. A breaking change ships as a new contract, and the old one keeps working until its announced sunset date.
Validated before delivery
Every output is checked against its schema before you see it. When a site changes under us, you get a clear layout_changed outcome. You never get silently wrong data.
Tested every 15 minutes
Canary runs hit the live site around the clock. Each recipe's health, Operational or Degraded, is shown in the console and returned by the API, so you know before you build on it.
Pay for answers only
Results, an empty result, or 'not found' are billed. Blocked, rate-limited, broken or timed-out runs cost nothing. Paginated recipes charge only for the pages actually fetched.
Cheapest path first
Recipes use official APIs, embedded JSON and publisher feeds before reaching for a browser. Most run on the 1-credit Light engine, and a full browser runs only when the site requires one.
Just another job
A recipe run is an ordinary job: idempotency tokens, polling, signed webhooks with the result inline, job history, EU-only storage. Nothing new to wire up.
The catalog
30 recipes in beta, and more every week. Public content only: networks that require a login to read are not in the catalog.
Search engines
google-serp
Organic results for a query, language and country
google-autocomplete
The suggestions Google offers for a partial query
Video
youtube-search
Videos for a query: channel, views, duration, date
youtube-video
One video's metadata, keywords and caption languages
youtube-subtitles
Existing captions as timed segments and full text
youtube-channel-feed
A channel's latest uploads with dates and views
E-commerce
amazon-search
Search results: ASIN, price, rating, reviews, sponsored
amazon-product
One product: price, availability, features, categories
Social & communities
Public content only
bluesky-posts
Recent public posts of an account, with engagement
bluesky-profile
Bio, avatar, follower counts and pinned post
bluesky-thread
A post and its direct replies
mastodon-hashtag
Recent public posts with a hashtag
mastodon-trends
What's trending right now, in a chosen language
lemmy-community
A community's posts: score, votes, comments
telegram-channel
Latest posts of a public channel, with views
hackernews-search
Stories, Ask HN, Show HN or the front page
Code hosting
github-issues
Issues of a public repository, with labels and dates
github-repo
Stars, forks, topics, license and activity
github-search-repos
Repositories matching a search, qualifiers included
github-releases
Releases: tag, date, notes and assets
Developer tools
npm-package
Latest version, license, dependencies and size
pypi-package
Version, requirements, links, known vulnerabilities
stackoverflow-search
Questions with score, answers and accepted answer
News
Publishers' own RSS: headlines, never article text
lemonde-headlines
Le Monde, by section
lefigaro-headlines
Le Figaro, by section
humanite-headlines
L'Humanité, latest headlines
bbc-news-headlines
BBC News, by section
guardian-headlines
The Guardian, by section
nypost-headlines
New York Post, by section
Knowledge & reference
wikipedia-summary
An article's description, lead, image and Wikidata id
One call
Send an input. Get the record.
No selectors, no rendering decisions, no parsing. The recipe knows how to fetch the site and what the answer looks like. You pick the recipe and fill in its input schema.
- Same API key, same job endpoints, same webhooks as every other job
- Python SDK support (client.recipes.run_and_wait) in the next SDK release
- Console form generated from the input schema, with results as a table
Under the hood
A recipe is forty lines of YAML
We built a small declarative language to describe a scraper: how to fetch, how to tell a real answer from a wall, what to extract, and what the output looks like. It's validated by a published JSON Schema, replayed offline in CI against captured pages, and run against the live site every 15 minutes.
- Steps
- HTTP request, paginate and follow on the Light engine. Navigate, click, fill, scroll, wait and dismiss consent on the browser engines.
- Assertions
- Decide what the page means: a real answer, an empty result, not found, blocked, or rate-limited. That outcome is what decides billing.
- Extraction
- CSS, XPath or JSONPath, each as a fallback chain, with typed fields, defaults and transforms.
- Invariants
- Minimum records, distinct ids, absolute URLs. These catch drift before a customer does.
- Sandboxed inputs
- Templated inputs go only into URLs, bodies and form values, never into selectors, scripts or hosts.
# Light engine: the official HN Search API.
slug: hackernews-search
contract: 1
tier: light
steps:
- request:
method: GET
url: "https://hn.algolia.com/api/v1/search?query={{ input.q | urlencode }}&tags={{ input.type | urlencode }}"
assertions:
page_ok: [{ status: 200, jsonpath_exists: "$.hits" }]
empty: [{ status: 200, body_matches: '"nbHits":\s*0\b' }]
rate_limited: [{ status: 429 }]
extract:
items:
type: records
container_jsonpath: ["$.hits[*]"]
fields:
id: { jsonpaths: ["$.objectID"], required: true }
title: { jsonpaths: ["$.title"], required: true, transform: [html_text] }
url: { jsonpaths: ["$.url"] }
hn_url: { jsonpaths: ["$.objectID"], format: "https://news.ycombinator.com/item?id={}" }
points: { jsonpaths: ["$.points"], default: 0 }
comments: { jsonpaths: ["$.num_comments"], default: 0 }
output:
results: { from: items }
captured_at: { meta: captured_at }
invariants:
items:
min_records: 1
distinct: [id]
absolute_url: [hn_url]For agents
Your agent stops guessing selectors
Through the ScrapeNest MCP server, an agent can list the catalog, read a recipe's input schema, run it and get validated JSON back. It's faster and cheaper than scraping the page and reading the Markdown, and the answer comes with a contract.
get_recipe("hackernews-search")
run_recipe("hackernews-search", {q: "rust", type: "show_hn"})
outcome ok · contract 1 · schema valid
Here are this week's top Show HN posts about Rust, ranked by points, with links to each discussion.
Coming soon
Write your own recipes
The DSL that powers our catalog is the same one you'll write. Soon you'll be able to turn the sites you care about into recipes, with your own schemas, tests and health checks, and share them.
- The same validator and offline replay we use in our own CI
- Runs under your organization, your quota and your plan
- Publish to your team, or share with everyone