Scraping a government procurement portal behind bot protection — legitimately

Let me put the framing up front, because it matters. The data in this post is public procurement information — tenders that a government publishes specifically so that suppliers can find and bid on them. Access is rate-limited and polite: a few requests every few hundred milliseconds, well inside what a couple of human analysts refreshing the page would generate. I'm not going to write a bot-check bypass guide, and there isn't one here. What's worth writing about is the architecture — an extraction ladder that keeps cost and hallucination down by trying the boring, reliable methods before the clever, expensive one.

Why a ladder at all

The naive way to build a general scraper today is to point a large language model at the HTML and ask for JSON. It works, and it's a trap. An LLM extraction costs seconds of latency and real money per call, and — worse — it occasionally invents fields that look plausible and are wrong. A cost bucket of "200k - 400k" becomes an exact value of 200400. A missing deadline becomes a confidently hallucinated date.

So instead of leading with the model, we lead with everything cheaper and more truthful, and only climb to the model when the rungs below it fail. Concretely, extraction runs through eight stages, 0 through 6:

• Step 0 — Official API. If we know the site has a real feed (OCDS, TED),

call it. ~100 ms, can't hallucinate, highest confidence.

• Steps 1–3 — Learned and configured selectors. CSS selectors from a

knowledge graph, previously-validated selectors, and admin-configured rules. ~50 ms, no model.

• Step 3.5 — Site-specific parser. A hand-written deterministic extractor

for this exact site. If it returns rows, we short-circuit and skip the LLM entirely.

• Step 4 — LLM extraction. The general-purpose safety net, for arbitrary

layouts nothing above could handle. Seconds and cents per call. The only paid rung.

• Steps 5–6 — Reuse and generic fallback. Reuse the Step-3.5 result at

lower confidence, or a last-ditch generic DOM heuristic.

The dispatch is a plain escalation — return at the first rung that yields tenders, and never bother with the rest:

// Step 0: official API feed (cheapest, most trustworthy)
if (strategy === 'api') {
  const r = await apiExtractor.extractFromApi(config);
  if (r.tenders.length > 0) return finalize(r, 'api');
}
// Steps 1-3: learned / validated / configured CSS selectors
for (const selectors of [kgSelectors, validatedSelectors, backendRules]) {
  const r = ruleBasedExtraction({ html, selectors });
  if (quality(r) > 0.5 && r.tenders.length > 0) return finalize(r, 'selectors');
}
// Step 3.5: site-specific parser -- short-circuits BEFORE the LLM
const early = await extractSiteSpecific(siteId, siteUrl, html);
if (early && early.tenders.length > 0) return finalize(early, 'site-specific');
// Step 4: LLM -- only reached if everything above failed
if (kimiAvailable) {
  const r = await kimi.runExtractionWithFallback(prompt);
  if (r.success) return finalize(r, 'kimi-ai');
}
// Steps 5-6: reuse / generic DOM fallback
return finalize(structuredExtraction({ html }), 'structured-jsdom');

Steps 0, 4 and 6 are never skipped; the ones in between can be turned off per site once we've learned they never fire. In practice 80–90% of sites resolve before the LLM rung — meaning the expensive, fallible path runs on a small minority of hard cases, which is exactly where you want to spend it.

A concrete rung: the bot-defended portal

This particular portal is a national procurement system carrying about 9,700 active tenders over a rolling 90-day window, served as a paginated JSON feed. In front of that feed sits an enterprise bot-check that issues a session cookie before it'll serve data.

At the architecture level — which is as far as I'll go — the legitimate move is the obvious one: behave like a browser that a real visitor drives. Load the public landing page first, receive the same session cookie any visitor's browser would receive, then read the JSON feed with that cookie, politely and rate-limited. No headless-browser farm, no fingerprint games; just don't skip the step a normal client wouldn't skip. That's the whole philosophy, and I'm deliberately not publishing the operational specifics, because "how to look like a legitimate client to this vendor's product" is exactly the knowledge that shouldn't be in a blog post.

The engineering worth publishing is what happened next.

The bug worth the whole post: trusting the page size you asked for

The first version of this extractor requested PageSize=50 and stopped paginating when a response came back with fewer than 50 rows — the textbook "last page is short, so we're done" heuristic. It ran green. It reported success. It ingested 24 tenders.

Twenty-four. Out of ~9,700.

The server silently caps every response at 24 rows regardless of the requested page size — ask for 25, 50, or 100 and you get 24 every time. So the extractor fetched page one, saw 24 < 50, concluded it had reached the last page, and stopped. 0.2% of the corpus, extracted with total confidence. This is the worst kind of failure: not a crash, not an error log, just a wrong number wearing a success message.

The fix is to never trust the parameter you sent as a proxy for what the server did. Compare against actual rows returned and the known server cap, and keep going until a page is genuinely short or empty:

const SERVER_PAGE_CAP = 24;   // server caps responses at 24 regardless of ask
// ...
if (rows.length === 0) {
  if (++emptyStreak >= 2) break;               // tolerate a transient blank page
  await sleep(PAGE_DELAY_MS * 4);              // back off, then retry
  continue;
}
emptyStreak = 0;
// stop only when the server truly returned a short/last page:
if (addedThisPage === 0) break;                       // no new unique rows
if (rows.length < Math.min(pageSize, SERVER_PAGE_CAP)) break;

Same idea, general lesson: validate pagination against the data you got back, not the request you made. Servers lie about honouring your page size all the time, and a stop-condition built on the request parameter will silently truncate you.

Politeness is a completion strategy, not just courtesy

The delays between requests aren't only good manners — they're what lets a long run actually finish. Hammer a rate-limited endpoint and you trip the very throttling and blocking you're trying to stay under, and the run dies half-done. So: 250 ms between pages, tolerate up to two consecutive empty pages (rate-limit hiccups) with a 4× backoff before giving up, and a hard cap of 400 pages as a safety belt. Walking the full 90-day window takes about 100 seconds and completes reliably. A cycle costs a fraction of a cent because no LLM is involved — versus the seconds and cents per call the model rung would add.

Dedup that survives a re-run

Because runs overlap and retry, extraction has to be idempotent. Two layers do it: an in-memory Set of seen IDs within a single cycle, and a canonical (siteId, sourceId) upsert at the database so re-ingesting the same tender updates the existing row instead of duplicating it. Re-run the same page ten times and the corpus is identical.

Why the ladder short-circuits the AI

Back to Step 3.5. For a site with an authoritative parser — this portal, or the CERN Shiny app I wrote about separately — we return the deterministic result before Step 4 ever runs, so the model never gets a chance to "improve" on ground truth by inventing a field. It costs essentially nothing (we already parsed the page) and it eliminates an entire class of hallucination. The LLM is a safety net for the long tail of sites we haven't hand-tuned, not the default tool for the ones we have.

The takeaways

• Escalate, don't lead, with LLMs. The cheapest, most truthful extraction

is the API call you make before you reach for a model.

• Validate pagination against returned data. The page size you asked for is

not evidence of what the server did. Silent truncation reports success.

• Politeness keeps runs completing. Rate-limiting yourself is how you avoid

the blocking that kills a job mid-corpus.

• Public data deserves respectful access. Behave like a legitimate client,

stay well inside human-scale request rates, and the transparency mandate the data was published under is the whole point.

---

We build production extraction pipelines for public and enterprise data — resilient to messy HTML, respectful of the sources, and cheap to run at scale. That's the data-extraction service we offer if that's a problem you have.