Job Pipeline

A containerised, end-to-end job-search automation pipeline. Scrapes job boards, deduplicates listings, persists them to Google Sheets, and scores each role against a structured candidate profile using a local LLM. Runs entirely on a developer laptop — no API keys, no cloud dependencies, no per-token cost.

💡 New here? If you want to run this on your own laptop, follow the step-by-step guide in docs/SETUP.md. Expect about 60–90 minutes including Google OAuth setup.

Built as a portfolio project for a Dublin-based BA / Data Analyst job search across Ireland and the UK.

---

Performance

Real numbers from a complete production run:

• ~190 unique jobs per scrape run across 4 role families × 2 geos
• ~5 minutes for the scrape phase (8 parallel HTTP calls, deduped, upserted)
• ~6 seconds average LLM scoring latency per job (Llama 3.2 3B, local)
• ~48 minutes for full scoring of 335 jobs at concurrency = 3
• 336 cached scored jobs persisted in 86 KB of JSON on disk
• 0 API cost — all inference runs on the developer laptop
• 0 errors in the most recent full scoring pass (335/335 success)
• Re-run latency for already-scored URLs: <10 milliseconds (disk cache hit)

---

How it's wired

The n8n workflow stitches the whole pipeline together. One trigger fires the entire flow:

Three containers, one shared volume for CSV persistence, OAuth2 to Google Sheets, local-only LLM inference. n8n orchestrates; jobspy scrapes and scores; Ollama runs the model.

---

Phases

• Phase 1 — Containerised scraping service (FastAPI + python-jobspy)
• Phase 2 — Google Sheets integration via OAuth2 with idempotent schema
• Phase 3 — Multi-search loops (4 role families × 2 geos) with deduplication and upsert
• Phase 4 — Local LLM scoring (Ollama + llama3.2:3b) with disk-backed cache

---

What it does

A single click in n8n triggers the full pipeline:

1. Scrape — 8 parallel searches across Indeed and LinkedIn (4 role families × 2 geos)
2. Aggregate — collect all responses into a single bundle
3. Deduplicate — remove URL collisions across searches (same job in BA + Systems searches counts once)
4. Upsert — append new rows or update existing ones in Google Sheets, matched on job_url
5. Score — read unscored rows, batch-send to local LLM via the /score-batch endpoint, get back a 0-10 fit score plus one-sentence reason for each
6. Persist scores — write fit_score and fit_reason back to the sheet

Re-running the workflow is safe: existing jobs are upserted (not duplicated), and already-scored jobs come from the disk cache in milliseconds instead of 6 seconds of inference.

---

The candidate profile

The scorer is fed a structured JSON profile covering:

• Current location, willingness to relocate, right to work
• Experience (years, current and previous employers, domains)
• Education and certifications
• Core skills grouped by capability cluster
• Target roles (preferred / acceptable / avoid)
• Salary expectations and industry preferences

Stored in jobspy-service/profile.json and loaded fresh on every scoring call so iterations don't require a rebuild.

---

Phase 1 — Containerised scraping service

Stack: Python 3.11, FastAPI, python-jobspy, pandas, Pydantic.

A POST to /scrape accepts:

• search_term (e.g. "Senior Business Analyst")
• location (e.g. "Dublin, Ireland")
• sites (Indeed, LinkedIn, Glassdoor, ZipRecruiter, Google, Naukri, Bayt — defaults to Indeed + LinkedIn)
• results_wanted, hours_old (recency window)
• country_indeed (auto-inferred from location if omitted)
• role_family (used as a CSV filename tag)
• save_csv flag

Returns a structured response with per-job metadata and writes a CSV to a shared Docker volume (/data/shared/jobs_<family>_<timestamp>.csv).

The country inference handles Ireland, UK, Netherlands, Germany, France and Indian cities (Bangalore, Mumbai, Delhi).

One subtle decision: the /scrape endpoint returns plain dicts, not Pydantic-validated response models. python-jobspy returns pandas DataFrames containing NaN values, which Pydantic rejects. Plain dicts pass through cleanly.

---

Phase 2 — Google Sheets integration

OAuth2 setup in Google Cloud (own project, OAuth client ID, redirect URI, consent screen), then n8n's Google Sheets credential authenticates against it. The sheet has 16 columns, A-P:

scraped_at | role_family | site | title | company | location | date_posted
| is_remote | job_url | min_amount | max_amount | currency | search_term
| description | fit_score | fit_reason

A debugging note worth preserving: column headers are whitespace-sensitive in n8n's auto-mapping. Trailing or leading spaces in the sheet (invisible to the eye) silently cause schema mismatches at insert time. Diagnostic: =LEN(E1) in any empty cell — should match the column name length exactly.

---

Phase 3 — Multi-search loop with idempotent upsert

A "Build search configs" Code node emits 8 items per click — one per (role_family × geo) combination — and feeds them into the HTTP Request node which loops automatically. The 8 responses are aggregated into a single bundle, flattened, deduplicated by job_url, then upserted to the sheet.

The idempotent upsert pattern means the workflow can run any number of times per day. Existing rows update their scraped_at timestamp; new jobs append. The sheet always reflects the current state of the market, not the cumulative scrape history.

---

Phase 4 — Local LLM scoring with disk-backed cache

Why local LLM: No API costs, no rate limits, no data leaving the developer's machine. Production-relevant for any company handling sensitive candidate or job data.

Groq alternative: For machines with limited RAM, the service also supports Groq as a drop-in cloud backend. Set SCORING_BACKEND=groq and GROQ_API_KEY in .env — no GPU, no model download, no Docker memory requirement. The free tier is sufficient for batches under ~100 jobs. See Checkpoint 9b in SETUP.md for setup steps.

Why a disk cache: Each scoring call takes ~6 seconds of inference. A full run of 335 jobs takes ~48 minutes. Without a cache, any downstream failure (n8n timeout, network blip, container restart) means re-scoring from scratch. With the cache, scored URLs are persisted to /data/shared/score_cache.json mid-run (every 25 items) and survive process restarts. Re-runs return cached results in milliseconds.

After the first full pass: 336 cached scores in 86 KB of JSON. A re-run of the same data completes in seconds, not minutes.

The batch scoring service logs progress in real time:

The architectural pivot worth describing

The first attempt put the scoring loop inside n8n: read 312 unscored rows, call /score once per row in a Loop Over Items node. Every iteration of this design failed at scale.

Diagnosis revealed three independent schedulers with conflicting assumptions about concurrency, timeouts, and backpressure: n8n's loop semantics, FastAPI's worker model, and Ollama's inference queue. The orchestrator (n8n) couldn't safely manage long-duration concurrent state across hundreds of items. PowerShell calls to the same endpoint succeeded perfectly with identical concurrency — proving the bottleneck wasn't compute but orchestration.

The fix was an ownership boundary change: move iteration, retries, concurrency control, and partial-failure handling out of n8n and into the Python service.

The new /score-batch endpoint accepts an array of jobs, uses asyncio.Semaphore(3) to bound Ollama concurrency, persists progress to the cache every 25 items, captures per-item errors without aborting the batch, and returns a single structured response with success/error counts and elapsed time. n8n now makes one HTTP call (with a 90-minute timeout) instead of looping over 300+ items.

This is the architectural pattern that distinguishes workflow orchestrators from compute schedulers: orchestrators route work across systems and trigger pipelines; compute schedulers manage queues, retries, concurrency, and durable state. Each is good at one and bad at the other.

---

Local setup

Prerequisites: Docker Desktop, Git, PowerShell (Windows) or bash, ~8 GB free disk for the Llama 3.2 3B model.

git clone https://github.com/Ayushman-Raghav/job-pipeline.git
cd job-pipeline
cp .env.example .env       # No secrets needed for local LLM, but creates the file
docker compose up -d
docker exec -it ollama ollama pull llama3.2:3b

No GPU / low RAM? Skip the ollama pull step and use Groq instead: get a free key at console.groq.com, then set SCORING_BACKEND=groq and GROQ_API_KEY=gsk_... in .env before starting. See Checkpoint 9b for the full walkthrough.

Wait for all containers to report Up, then verify:

(Invoke-WebRequest http://localhost:8000/health).Content
docker exec ollama ollama list

The full container stack at idle:

Then open n8n at http://localhost:5678, import workflows/job-pipeline-phase-4.json, configure your Google Sheets OAuth2 credential, and click Execute workflow.

First run takes ~5 min for scraping plus ~50 min for scoring 300+ jobs. Subsequent runs take ~5 min for scraping and a few minutes for sheet updates because scoring comes from the cache.

---

Endpoints

Method	Path	Purpose
GET	/health	Liveness probe with version
GET	/sources	List supported job boards
GET	/role-families	Return canonical role taxonomy
GET	/files	List CSVs in shared volume
GET	/profile	Inspect the loaded candidate profile
GET	/score-cache/stats	Inspect the cache (count, path, size)
DELETE	/score-cache	Clear the cache (e.g. after profile changes)
POST	/scrape	Scrape one (search × location), optionally save CSV
POST	/score	Score one job synchronously
POST	/score-batch	Score many jobs concurrently with caching

Example scrape request

curl -X POST http://localhost:8000/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "search_term": "Senior Business Analyst",
    "location": "Dublin, Ireland",
    "sites": ["indeed", "linkedin"],
    "results_wanted": 15,
    "hours_old": 168,
    "role_family": "BA",
    "save_csv": true
  }'

Returns:

{
  "search_term": "Senior Business Analyst",
  "location": "Dublin, Ireland",
  "sites_queried": ["indeed", "linkedin"],
  "role_family": "BA",
  "scraped_at": "2026-05-10T11:43:01.540123",
  "count": 27,
  "csv_path": "/data/shared/jobs_ba_20260510_114301.csv",
  "jobs": [ /* per-job records */ ]
}

Example batch scoring request

curl -X POST http://localhost:8000/score-batch \
  -H "Content-Type: application/json" \
  -d '{
    "jobs": [
      {
        "title": "Senior Data Analyst",
        "company": "Stripe",
        "location": "Dublin, Ireland",
        "description": "Build dashboards in Looker, write SQL across Snowflake, partner with finance and product teams. 5+ years experience required.",
        "job_url": "https://example.com/job/1"
      }
    ]
  }'

Returns:

{
  "results": [
    {
      "job_url": "https://example.com/job/1",
      "score": 9,
      "reason": "Excellent fit on role, level, location, skills, and industry preference for a senior data analyst.",
      "error": null
    }
  ],
  "model": "llama3.2:3b",
  "total_seconds": 6.1,
  "success_count": 1,
  "error_count": 0
}

---

Repository layout

job-pipeline/
├── docker-compose.yml              ← 3 services + named volumes + bridge network