Extracting structured data from an R Shiny app at CERN

We monitor public procurement portals for a living. Every few weeks a new one lands on my desk, and the first hour is always the same: figure out what the site actually is before writing a line of extraction code. CERN's "Forthcoming Tendering Procedures" page at forthcoming-ms.app.cern.ch punished me for skipping that hour.

The wrong first assumption

The URL carried an Angular-style #!/ hashbang. I've scraped enough single-page apps to have a reflex: hashbang means client-side routing, means the data arrives via XHR, means I should wait for network to settle and read the DOM. So my first probe did exactly that — navigate, wait for networkidle, read the table.

The table was empty.

Not "missing a few rows" empty. The rendered HTML was about 18 KB and contained no tender data at all — just an application shell. networkidle had fired, my selector had timed out, and I had nothing. The reflex was wrong, and the URL had lied to me.

What the site actually is

A few minutes with the network panel corrected the record. This isn't Angular. It's an R Shiny app — shiny.router for routing, the DT / datatables-binding for the grid. Shiny keeps a WebSocket open to its server and pushes rendered content into the DOM after the initial page load. The tender table lands roughly 2–3 seconds after load, well after networkidle has already resolved on the empty shell.

That single fact reframes the whole job. networkidle is meaningless against a UI that hydrates over a persistent WebSocket — the network is idle, because the socket is just sitting there waiting to deliver the next frame. You cannot wait for the network. You have to wait for the content.

Waiting for the content, not the network

The fix is to wait on the thing you actually want — a row in the data table — and give the WebSocket push a moment to complete:

// Shiny renders via WS, so networkidle can fire before the table is filled.
await page.goto(URL, { waitUntil: 'networkidle', timeout: 45000 });
try {
  await page.waitForSelector('table.dataTable tbody tr', { timeout: 25000 });
} catch {
  console.log('[wait] no DataTable rows appeared within 25s');
}
await page.waitForTimeout(2000);

I also added a defensive size gate in the parser itself. The shell is ~18 KB; a populated page is comfortably larger. If the parser is handed less than 20 KB of rendered HTML, the push hasn't landed yet, so it returns an empty result and lets the caller retry rather than emit half-baked garbage:

function extractCernForthcoming(html, url) {
  // Sanity: the Shiny shell is ~18 KB and has no data. If we get less than
  // ~20 KB of rendered HTML, the WS push hasn't landed yet -> return [] and
  // let the caller fall through.
  if (!html || html.length < 20000) return [];
  // ... JSDOM parse of table.dataTable ...
}

With that, listing extraction was solid: 64 forthcoming procedures, parsed by a pure JSDOM function with no AI anywhere in the hot path. Good enough to ship — except each row hides its detail (description, requirements, contacts) behind a modal you have to click open, and that is where I lost the day.

The bug worth the whole post: synthetic click ≠ real click

To enrich each tender I needed to open its row's modal, read the .modal-body, and parse the detail. Simple: find the row, click it, wait for the modal. I wrote exactly that, ran it, and… nothing happened. No modal, no error, no clue. The click "succeeded" — the element was there, the call returned — but the app did not react.

Here's the root cause, and it's a genuinely useful thing to carry around. Shiny wires up its interactions with jQuery delegated event handlers — handlers bound to a stable ancestor that fire when an event bubbles up from a matching descendant. Playwright's elementHandle.click() in its DOM-dispatch form fires a synthetic event that does not reproduce the full trusted mousedown → mouseup → click sequence a real user generates. jQuery's delegation never sees it. The modal never opens.

The fix is to stop synthesising the event and drive a real mouse instead — Playwright's Locator/ElementHandle.click() moving the actual cursor and issuing real mousedown/mouseup:

// Find the row by reference text, then dispatch a *real* mouse click through
// Playwright -- synthetic element.click() does NOT trigger Shiny's jQuery
// delegated handler; only a full mousedown/mouseup sequence opens the modal.
const rowHandle = await page.evaluateHandle((ref) => {
  const rows = Array.from(document.querySelectorAll('table.dataTable tbody tr'));
  for (const row of rows) {
    const cells = row.querySelectorAll('td');
    if (cells.length >= 3 && (cells[2].textContent || '').trim() === ref) {
      return cells[3] || cells[2] || row;   // the .underline click target
    }
  }
  return null;
}, t.sourceId);

const element = rowHandle.asElement();
await element.scrollIntoViewIfNeeded({ timeout: 2000 });
await element.click({ timeout: 4000 });      // real mouse event -> modal opens
await page.waitForSelector('.modal-body', { timeout: 8000 });

That was the entire fix. One line — element.click() through a real handle instead of a synthetic dispatch — and enrichment went from 0/64 to working. If you take one thing from this post: synthetic events skip delegated handlers. Any framework that leans on $(document).on('click', selector, …) — which is most of the jQuery-era web, and everything built on top of it — will ignore a click you didn't route through a real input device.

Two more Shiny-isms

Once the modal opened, two smaller quirks remained.

The table paginates client-side. All 64 rows are in the DOM, but DataTables only shows a page at a time, so a naive row loop only ever sees the first ten. To touch every row I flip the page-size selector to "all" (DataTables' convention is the value -1):

// DataTables paginates client-side. Flip the page-size <select> to -1 ("all")
// so every row is clickable. Falls back gracefully if the menu is hidden.
await page.evaluate(() => {
  const sel = document.querySelector('select[name$="_length"]');
  if (sel) {
    sel.value = '-1';
    sel.dispatchEvent(new Event('change', { bubbles: true }));
  }
});

The modal itself hydrates progressively. Even after .modal-body appears, the description <ul> arrives a beat later via another WebSocket push, so I pad 400 ms before reading. And the whole enrichment loop is strictly sequential — Shiny reuses a single modal slot, so two concurrent clicks corrupt each other. At ~1–2 seconds per row that's 75–150 seconds for a full pass, which is fine for a feed that updates a few times a day.

Letting the parser refuse to guess

One last decision that matters more than it looks. CERN publishes cost buckets — "200k - 400k" — never exact figures. A general-purpose LLM extractor, handed that string, will cheerfully report the value as 200400. That's not extraction; that's fabrication. So this site is deterministic on purpose: the parser forces value: null, parses the bucket into valueRangeMin/valueRangeMax, and never lets a model re-derive a number that was never there. Knowing when your extractor should decline to produce a field is as important as knowing how to produce it. (This is one rung of a larger extraction ladder that keeps a language model out of the hot path for sites we've hand-tuned.)

Where it landed

64 of 64 rows extracted, 64 of 64 enriched with description, requirements, timeline, and contact leads in the live trial. No AI in the extraction path, no flaky "wait a random number of seconds and hope" — just a parser that knows the site is Shiny, waits for content instead of network, drives a real mouse, and refuses to invent numbers.

The meta-lesson is the cheap one: identify the framework before you pick a strategy. The hour I skipped at the start would have told me it was Shiny, and I'd have reached for a real click on day one instead of day two.

---

We build resilient extractors for messy, JavaScript-heavy, bot-defended public data sources — the ones that lie about what they are. If you've got a portal that fights back, here's how we handle sites like this.