Skip to content
ScrapeNest
All posts
·8 min read·by ScrapeNest Engineering

How to collect public LinkedIn job listings with ScrapeNest

A practical, login-free workflow for collecting public LinkedIn job-search results, extracting aligned job records, checking completeness, and monitoring a query for new listings.

tutorialjobsextractionmonitoring

You do not need a LinkedIn account to see every job result. LinkedIn exposes a public search route that returns a useful subset of job-search results to logged-out visitors. That distinction matters. A URL under /jobs/search-results may redirect an anonymous visitor to sign-in; a URL under /jobs/search can return the public result list.

That is enough for a legitimate, useful workflow: collect public listings for a role and geography, turn them into rows, and alert when the result set changes. Think job-market research, a recruitment team's watchlist, or an internal tool that spots when named companies are hiring for a capability your business sells.

This guide uses a real public LinkedIn search for “web scraping remote” in France:

https://www.linkedin.com/jobs/search?keywords=web%20Scraping%20remote&location=France&geoId=105015875&position=1&pageNum=0

When we tested it, the page returned a result count and public cards with a title, company, location, posted time, and a link to the listing. It did not require a login.

The boundary: public search results, not an automated LinkedIn account

This tutorial is deliberately about the public route. Do not put a LinkedIn username, password, session cookie, or MFA code into a scraping job. ScrapeNest is not an authenticated LinkedIn connector, and a login-gated page is a useful stop signal, not an invitation to automate around it.

Check the target's applicable terms and robots policy, keep request volume modest, retain only the data you have a legitimate reason to use, and link users back to the original listing. If you need data that only an account or a partner product can access, use LinkedIn's authorized integration or another licensed source.

What we are building

The first useful output is not a blob of HTML. It is a list of records like this:

{
  "title": "Ruby Engineer - Web Scraping (Remote)",
  "company": "SearchApi",
  "location": "France",
  "posted": "2 months ago",
  "url": "https://fr.linkedin.com/jobs/view/..."
}

Once records are aligned, the rest becomes straightforward:

  1. Store a normalized copy and a stable key such as the job URL.
  2. Compare it with the last successful run.
  3. Mark disappeared jobs as no longer seen, rather than immediately deleting them.
  4. Send only new or materially changed records to a person, a CRM, or your own job board.

The scraping is one step. Freshness, deduplication, and a clear response to a failed run are what make the result useful.

Start with the lightest engine

LinkedIn renders the public result cards in the initial response for this search. We verified that ScrapeNest's Light engine can read the titles and companies, so it is the right place to start: it is faster and costs fewer credits than a browser. Do not start with Stealth just because the target is LinkedIn.

If your specific query begins returning an empty page or client-rendered content, retry one sample with Standard and inspect the result before increasing the engine for the whole batch. A target changing is a reason to investigate, not a reason to make every request expensive.

Install the Python SDK and set the API key in your environment:

pip install scrapenest-sdk
export SCRAPENEST_API_KEY="sn_live_..."

Extract rows, not parallel lists

It is tempting to extract all titles, then all companies, then all locations. Do not. Any card missing a location shifts the later lists out of alignment and assigns the wrong company to the wrong job.

Use a records hook instead. It starts at one result-card container and resolves every field relative to that card. A missing field becomes null on that record; it cannot corrupt the records after it.

import json
import os

from scrapenest import ScrapeNestClient

SEARCH_URL = (
    "https://www.linkedin.com/jobs/search"
    "?keywords=web%20Scraping%20remote"
    "&location=France"
    "&geoId=105015875"
    "&position=1"
    "&pageNum=0"
)

client = ScrapeNestClient(api_key=os.environ["SCRAPENEST_API_KEY"])

result = client.scrape(
    job_type="light",
    target_url=SEARCH_URL,
    artifact_options={"include_extraction": True},
    extraction={
        "hooks": [
            {
                "hook_id": "jobs",
                "type": "records",
                "container": "ul.jobs-search__results-list > li",
                "max_results": 100,
                "fields": {
                    "title": {
                        "selector": "h3.base-search-card__title",
                        "transform": ["collapse_whitespace"],
                        "required": True,
                    },
                    "company": {
                        "selector": "h4.base-search-card__subtitle",
                        "transform": ["collapse_whitespace"],
                    },
                    "location": {
                        "selector": "span.job-search-card__location",
                        "transform": ["collapse_whitespace"],
                    },
                    "posted": {"selector": "time"},
                    "url": {
                        "selector": "a.base-card__full-link",
                        "attribute": "href",
                        "required": True,
                    },
                },
            }
        ]
    },
)

# Read the structured extraction artifact, not the page HTML.
job = client.jobs.get(result.job_id)
artifact = next(
    a for a in job.artifacts if a.artifact_type in {"extraction", "json"}
)
payload = json.loads(client.artifacts.download_text(artifact.artifact_id))

hook = payload["hooks"][0]
if hook["status"] != "succeeded":
    raise RuntimeError(f"Extraction failed: {hook.get('error')}")
if hook["truncated"]:
    raise RuntimeError(
        f"Only {len(hook['result'])} of {hook['total_matched']} results were returned"
    )

jobs = hook["result"]
for job in jobs[:5]:
    print(job)

The selectors above are intentionally attached to public result-card semantics rather than their position in the page. They matched the live target when this guide was written. Still, every selector is a contract with a page you do not own: keep a small canary run and treat selector_missing, an empty result, or an unexpected page title as a reason to review it.

Check completeness before you write anything downstream

The default extraction cap is 50 results. That is why the example explicitly asks for up to 100 and then checks truncated and total_matched.

Skipping that check is how a collector silently becomes a partial collector. If the search has 136 public matches and you only received 50, you do not have 50 new jobs; you have the first 50 jobs on an incomplete page. Persist the run metadata alongside the rows:

  • query URL and the time collected;
  • total_matched, truncated, and skipped_records from the hook;
  • ScrapeNest job ID and final outcome;
  • a content fingerprint or source URL for every record.

That data lets you explain why a job appeared, prove that a run was incomplete, and avoid treating a target outage as a mass deletion.

Normalize and deduplicate the result set

Job titles are not identifiers. The same role can appear once for France and once for Paris; a company can also repost it with a slightly different title. Use the canonical listing URL when it is available. If it is absent, use a conservative fallback such as a lowercased tuple of company, title, location, and the collection date.

from urllib.parse import urlsplit, urlunsplit

def canonical_url(url: str | None) -> str | None:
    if not url:
        return None
    parts = urlsplit(url)
    # Tracking parameters identify a click, not the job itself.
    return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))

def job_key(job: dict[str, str | None]) -> tuple[str, ...]:
    if url := canonical_url(job.get("url")):
        return ("url", url)
    return (
        "fallback",
        (job.get("company") or "").strip().lower(),
        (job.get("title") or "").strip().lower(),
        (job.get("location") or "").strip().lower(),
    )

current = {job_key(job): job for job in jobs}

Compare current with the last successful collection, not merely the last scheduled attempt. A blocked or failed run contains no evidence that a job has vanished.

Also expect false matches. A query for “web scraping remote” can return adjacent data-engineering, SEO, and data-acquisition roles. Relevance is a product decision: define your inclusion rules after collection and preserve the source title so a human can audit the filter.

Pagination: discover it; do not invent it

Search platforms change how they expose additional results. Do not assume that adding a page or start parameter will work forever. First extract the actual next-page links or pagination controls from the public page. Then submit one ScrapeNest job per discovered URL, with an idempotency token per run and page.

For a known and verified page sequence, the batch pattern looks like this:

from datetime import date

batch = f"linkedin-public-jobs-{date.today().isoformat()}"

for page_num, page_url in enumerate(verified_page_urls):
    client.submit(
        job_type="light",
        target_url=page_url,
        tags=["source:linkedin-public", f"batch:{batch}"],
        idempotency_token=f"{batch}-page-{page_num}",
        artifact_options={"include_extraction": True},
        extraction={"hooks": [jobs_records_hook]},
    )

For more than a handful of pages, collect job.completed webhooks rather than holding open a process and polling each result. Keep the batch tag so you can reconcile the run later.

Watch the search, not every card

If your question is “what is new this morning?”, a monitor is cleaner than reimplementing a scheduler. Watch the public search-result list once or twice a day, depending on your permitted rate and how time-sensitive the workflow really is.

Use selector-scoped change detection so cookie copy, footer links, and sign-in prompts do not generate alerts. This creates the monitor; it does not send a request until its first scheduled run.

monitor = client.monitors.create(
    name="public-linkedin-web-scraping-jobs-france",
    cron="0 7 * * 1-5",
    timezone="Europe/Paris",
    job_type="light",
    target_url=SEARCH_URL,
    detection={
        "enabled": True,
        "mode": "selector",
        "selector": "ul.jobs-search__results-list",
        "ignore_selectors": ["time"],
        "min_change_ratio": 0.02,
        "cooldown_seconds": 86400,
        "notify": {"webhook": True},
    },
)

The first run establishes a baseline. Later changed runs create a diff and can emit schedule.change_detected. Treat that event as a prompt to run the records extraction and calculate the actual added, changed, and no-longer-seen jobs. A page diff is evidence that the search changed; it is not your final dataset.

The production checklist

A small collection script becomes reliable when it has answers to the boring questions:

  • Did the public page still load? Store the final URL, response outcome, and a page-title or selector canary.
  • Was the result complete? Reject or flag a truncated extraction rather than publishing partial data.
  • Did the schema move? Alert on a missing selector; do not replace yesterday's dataset with empty rows.
  • What changed? Add new records, mark missing records as no longer seen after a grace period, and retain the source link.
  • What is the request budget? Use Light, run only as often as the use case needs, and spread batches politely.
  • Are you still in scope? Keep the workflow on public pages and reassess if the route becomes login-gated or the site's terms change.

Where to go from here

This pattern is useful beyond one LinkedIn query. Use it to maintain a focused public-job watchlist, then add company career pages and public ATS boards as separate sources. Normalize every source to the same record shape, deduplicate by canonical source URL, and send people back to the original job to apply.

The important discipline is simple: collect public data, make incomplete runs visible, and build decisions from structured records rather than a pile of scraped pages. ScrapeNest handles the collection, extraction, artifacts, and scheduling; your product can spend its energy on the job matches that are actually worth someone’s attention.