Monitoring, without the cron-job graveyard

Most scraping that matters is recurring. You don't check a competitor's prices once; you check them every hour. You don't pull a catalog on a whim; you pull it every night. And almost everyone starts the same way: a cron line on a box somewhere that curls an endpoint and hopes.

That works right up until it doesn't. The box reboots during your window and the run just never happens. Two runs overlap because last night's scrape was slow, and now you're paying twice and racing yourself. You run out of budget mid-month and every run fails silently, so you find out three days later when the dashboard is empty. Cron is a timer, not a system, and recurring scraping needs a system.

So we built monitoring into ScrapeNest. A monitor is a scrape that runs on a cron cadence you define - and, when you want it, watches the page and tells you the moment it changes. You give us a cron expression and a timezone; we run the job on that cadence, hand you a full history of what happened, and (optionally) a before/after diff when the content moves. Here's what we decided, and why.

A monitor is just a job that repeats

The most important design decision is the one you can't see: a monitored run is a normal job. When your monitor fires, it creates exactly the same job you would have submitted by hand - same engine, same options, same artifacts, same webhooks. It bills the same way too: credits by engine weight (Light 1, Standard 5, Stealth 30), charged only when the job actually delivers content.

That means there is nothing new to learn and nothing new to reconcile. Your existing job.completed webhooks fire for monitored runs. Your usage analytics already count them. The artifact you get from a 3 a.m. monitor run is byte-for-byte the artifact you'd get from a manual one. A monitor is a trigger with a memory, not a separate product.

Tell me when it changes, not every time it runs

Running on a cadence is table stakes. The reason you set up a monitor is usually a question: did this page change? So detection is built in. Turn it on and every run is fingerprinted over normalized content - whitespace collapsed, scripts and the noisy regions you name stripped out - and compared to the previous version. Rotating ads, timestamps, and session tokens don't trip it. A real change does.

You choose what to watch: the whole page, a single CSS/XPath selector (a price, a stock badge), the structured fields your extraction rules already produce (a field-level diff, price 49 to 39), or the canonical JSON of an API response. When something clears your thresholds, you get a webhook and an email with the change ratio and a link to the diff. When nothing changes, you get silence - which is the point.

Exactly once per window, even when things go wrong

Under the hood, monitors run on a durable scheduling engine, not a loop that sleeps and prays. That buys us the guarantees that are annoying to build yourself:

• Missed windows are handled deliberately. If our scheduler is briefly unavailable when your window arrives, the run isn't silently dropped; it's reconciled according to well-defined catch-up semantics.
• Overlap is a policy, not an accident. If a run is still going when the next one is due, you choose: skip it, queue exactly one, or allow the overlap. The default is skip, because that's what you almost always want and never remember to enforce.
• Every fire is recorded. Not just the successes. Each run shows up in the history as it happened.

Running out of budget shouldn't mean failing loudly

Here's the failure mode we cared about most. You've set an hourly monitor, and halfway through the month you exhaust your credits. What should happen?

The wrong answer is what cron gives you: the job runs, fails, and maybe pages someone. The run cost you nothing useful and the noise costs you attention.

Our answer: the run is skipped, not failed. It costs nothing, it's recorded in the history as skipped_quota, and if you want to know about it, there's a dedicated schedule.run_skipped webhook. A skipped run is a normal, expected thing - a monitor that quietly waits for next month's credits instead of throwing errors at you is behaving correctly. The same applies if you downgrade to a plan that no longer includes the engine a monitor uses: it pauses itself and tells you, rather than firing into a wall.

What it looks like

Creating a monitor is one call. Add a detection block and it also watches for changes:

from scrapenest import ScrapeNestClient

client = ScrapeNestClient(api_key="sn_live_...")

client.monitors.create(
    name="hourly-prices",
    cron="0 * * * *",
    timezone="Europe/Paris",
    job_type="light",
    target_url="https://example.com/pricing",
    detection={
        "enabled": True,
        "mode": "selector",
        "selector": ".price",
        "notify": {"email": ["alerts@acme.eu"]},
    },
)

From there you can pause and resume without losing the definition, edit the cadence, and page through the run history and the list of detected changes. The same lives in the console under Monitoring, with one-click pause and a run log and change history per monitor.

Monitoring is available on Starter and up. Frequency and how many monitors you can keep scale with the plan, mostly as a guardrail: a per-minute monitor on the most expensive engine adds up fast, and we'd rather make that a deliberate choice than a surprise invoice.

The monitoring docs have the full cron reference, overlap policies, detection modes, and plan limits. If you've been running a scraping cron job you don't trust, this is the part you get to delete.