Status & progress

2026-07-11Yalçın Yörük

The EU AI Act's high-risk obligations land in August — a builder's readiness checklist

On 2 August 2026 the EU AI Act's obligations for high-risk AI systems become applicable — and they reach UK companies whose systems or outputs touch the EU market. This is an engineer's translation of what the Act actually asks for: the provider/deployer role split, how risk classification works in practice, and what inventory, model documentation, human oversight and logging look like when you build them as software rather than shelf-ware policy. Not legal advice; a working checklist.

2026-07-11Yalçın Yörük

The reference architecture we run in production — and deliver to clients

The seven-layer architecture behind our tender-intelligence platform — channels, experience, domain services, integration, data & AI, platform, and a cross-cutting evidence layer — and the TypeScript end-to-end stack that implements it: NestJS, Next.js, Postgres with pgvector, queue-based pipelines, and a gateway-mediated LLM layer. Why we deliver client platforms by adapting this architecture rather than designing from a blank page, with the real production details — including the parts that only exist because something broke.

2026-07-05Yalçın Yörük

Tender week in numbers — 2,743 notices, France out front (w/e 5 July 2026)

2,743 tender notices in the week to 5 July 2026. France led with 1,988. Largest disclosed value: 940m NOK. Markets, sectors, buyers and closing deadlines — all computed from the live corpus.

2026-07-03DataMesh Consulting

Extracting structured data from an R Shiny app at CERN

A CERN procurement portal with an Angular-style hashbang URL turned out to be an R Shiny app that pushes its data table over a WebSocket seconds after load — and the bug that cost me a day was that a synthetic element.click() is invisible to Shiny's jQuery event delegation. Here is the whole story, with the code.

2026-07-03DataMesh Consulting

How Google deindexed our 9,000-page site — and the recovery

Four weeks after launch, Google Search Console showed 9,189 URLs submitted, 21 crawled, and 0 indexed. A minority of junk auto-generated titles had triggered a domain-wide quality cascade. Here's the honest post-mortem — the two-layer title filter, the sitemap that dropped from 8,548 to 2,186 URLs, the lastmod-trust bug, and the outage where an empty sitemap served with 200 OK was worse than a 5xx.

2026-07-03DataMesh Consulting

Scraping a government procurement portal behind bot protection — legitimately

A national procurement portal with ~9,700 public tenders sits behind an enterprise bot-check. The interesting engineering isn't the bot-check — it's the eight-stage extraction ladder that tries the cheapest, most truthful method first and only escalates to an LLM when it must. Plus the one-line pagination assumption that once capped us at 24 of 9,700 tenders while reporting success.

2026-06-02DataMesh Consulting

2 June — platform migrated to Fly.io + Supabase: faster, leaner, single-region London

We moved the entire platform — public site, API, admin dashboard, scraping agent, queue and object storage — off our previous cloud onto Fly.io (London) with a Supabase Postgres database (also London). Same features, lower latency, ~50% lower run cost, and a simpler stack we can reason about end to end.

2026-05-26DataMesh Consulting

26 May — CEJN Phase-2 detail enrichment + five more extractors land in WIP staging

CEJN (Catalonia's contracting platform) gained opt-in detail enrichment today — `enrichDetails:true` runs a per-row fetch against `/api/cadocuments/GetTenderById` to pick up description text, estimated budget, pipe-delimited CPVs, procurement-officer email/phone, and eight category/procedure flags that don't appear in the listing JSON. Concurrency 4, circuit-broken after 6 consecutive failures so a bad afternoon at the source doesn't take down the whole ingest. Alongside that, five new extractors landed in the repo but haven't been wired into the URL/API dispatch tables yet — buy-nsw (NSW state procurement), dgmarket (development-bank multi-source aggregator), koneps-kr (Korea g2b), philgeps-ph (Philippines), and sam-gov (US federal GSA). Each ships with a smoke-test harness; the wiring step is intentionally deferred so we can verify capture quality side-by-side against the legacy paths before they go live. RTA Dubai and Qatar Rail Procurement deactivated in the site list — neither has produced a valid notice in three weeks.

2026-05-24DataMesh Consulting

24 May — Late-Saturday extractor trio: CAPT Kuwait, Oman Tender Board, CERN R-Shiny (our first Step-5)

Three new portals went live late Saturday evening, all written as JSDOM parsers but with different orchestration shapes. CAPT (Kuwait) replaces the legacy SharePoint-table parser with a Cloudflare-gated WordPress listing where each card pre-renders its own detail block. Oman Tender Board is a server-rendered J2EE app behind an F5 BIG-IP TSPD WAF — the encrypted-POST pagination is Phase-2, but the first-page GET is unencrypted and ships ~50 fresh rows, which is what daily ingest needs. CERN forthcoming-ms is our first Step-5 — the page is an R Shiny app that streams its DataTable into the DOM via WebSocket only after a 2–3 s handshake. The browser pool's existing `table tr td` + 2 s wait turned out to be enough margin, so the parser ended up as the same pure JSDOM walk as the others.

2026-05-23DataMesh Consulting

23 May — MERX Canada and AusTender extractors land, iOS Matches refresh-cancel hotfix

Two more national portals come online — MERX (Canada's federal + provincial procurement aggregator) and AusTender (Australian federal government). Both as Step-0 extractors with full detail enrichment. On the iOS side, a hotfix went out as version 1.2 for a stubborn "Couldn't refresh" banner that some users were seeing on cold launch even though the server was returning fresh data fine. Root cause turned out to be a SwiftUI lifecycle quirk cancelling the in-flight URLSession task during AppState state churn; the fix bounded-retries on URLError.cancelled so the next attempt completes cleanly. Diagnostic instrumentation stays in this build to surface any remaining edge cases.

2026-05-22DataMesh Consulting

22 May — Four new Step-0 extractors (ADB, UNGM, GeBIZ, GETS NZ) and a data-quality sweep

A focused coverage day. Four new Step-0 extractors landed — Asian Development Bank (via its public SearchStax Solr index), UNGM (the UN procurement portal), a full rewrite of GeBIZ Singapore, and GETS NZ. All four include detail-page enrichment, not just listing-level coverage. Alongside the extractor work, three correctness fixes in the data pipeline: the junk-text title filter was dropping legitimate "Selection of …" titles, the immediate-submit path was nulling out tenderType / procedureType / contact fields, and the home-feed sort by deadlineAt had a column-alias collision that nulled every deadline. TED site consolidation script also ran in prod.

2026-05-21DataMesh Consulting

21 May — iOS v1.1 to the App Store, AfDB detail-page extractor, dashboard site reorder

A heavy ship day across three surfaces. iOS v1.1 went to App Store review — version bump, full AppIcon asset catalog, IAP compliance, plus three behavioural fixes (honest offline banner, rich SwiftData cache, detail-screen refetch after deep-link). The African Development Bank extractor was rewritten as a URL-dispatching Step-0 that handles both listing pages and per-project detail pages, with terminal-status pruning so closed/cancelled tenders don't keep getting re-extracted. And the operator dashboard's Sites tab got drag-and-drop reorder + enable/disable controls, replacing the JSON-edit-and-redeploy workflow.

2026-05-21Yalçın Yörük

Tender week in numbers — 1,857 notices, United Kingdom out front (w/e 21 May 2026)

1,857 tender notices in the week to 21 May 2026. United Kingdom led with 522. Largest disclosed value: £120bn (framework ceiling). Markets, sectors, buyers and closing deadlines — all computed from the live corpus.

2026-05-20DataMesh Consulting

20 May — Functional-report sweep: country counts, CPV dedup, sitemap priority

A scheduled QA-feedback day. Two waves of fixes from the latest functional test report — first the medium-severity items (country i18n, stale "65+ portals" claims, World Bank country tagging, source list dedup, /api → /v1 redirect), then the low-severity polish (sticky-header wrap on narrow widths, country card counts off by one, CPV chip duplication, insights typography). And one structural change: the sitemap.xml is now ordered hubs-first, tender-details-last so the most important URLs get crawled in every fetch cycle.

2026-05-19DataMesh Consulting

19 May — Kimi model migrated to k2.6, Doffin extractor lands, tender-page hang fix

Two pieces of forced-march work and one nice piece of new coverage. Moonshot is deprecating the k2 series on 2026-05-25, so every Kimi call now points at kimi-k2.6 (with kimi-k2-thinking kept around just long enough to drain in-flight cached reasoning). A Doffin (Norway) Step-0 extractor went live via the public search API — no Playwright needed. And the public tender pages had been hanging up to 60 seconds waiting on a synchronous AI-summary call during SSR; that's now capped at 3 seconds with the summary backfilling client-side afterwards.

2026-05-18DataMesh Consulting

18 May — Kimi HTTP access unblocked, GSC indexing remediation, infra spring-clean

A consolidation day. The Moonshot HTTP fallback that had been silently 403'ing for two weeks is now mitigated via coding-agent impersonation headers, restoring the parallel-to-CLI capacity we'd been running without. Search Console indexing got a proper remediation pass — sitemap lastmod hygiene, noindex on faceted/admin URLs, a 5xx retry policy — and the /team page is back live with Person JSON-LD. On the data plane, the scraping-memory migration was applied to prod and a handful of Hermes correctness bugs (effectiveSiteId, junk DETAIL jobs, queue bloat) were fixed.

2026-05-15DataMesh Consulting

15 May — Intelligence dashboard gets weekly quotas, cached-vs-billable split, and live provider rate-limit panel

Three small but operationally important additions to the operator dashboard. Weekly budget cards stop us silently overspending across a week of normal-looking days. The 24-hour LLM call count now separates cached hits from real billable calls, so the headline number reflects actual spend instead of being padded by cache returns. And a new provider rate-limit panel captures Moonshot's `x-ratelimit-*` response headers in real time so we can see how close we are to throttling before it bites.

2026-05-14DataMesh Consulting

14 May — V3 self-healing scraping lands end-to-end, junk filter closed, six new extractors

A long sprint day. The V3 self-healing refactor (S1 → S6) merged in full — three dedup layers, an LLM-provider seam, nightly site-health audit, and auto-relearn/auto-disable on regression. Alongside that, the junk-title filter was bypassed at two ingest paths (now closed), description quality got a residue-stripping pass, six new portal extractors went live, and the operator dashboard finally has an Intelligence tab tracking Kimi spend per-site.

2026-05-13DataMesh Consulting

13 May — world-map heatmap is real now, portal-stats endpoint ships, two stealth UX bugs caught

The world map on the homepage and country pages had been a flat dark base layer with no per-country shading for weeks. We traced it to a missing backend endpoint, built it, then upgraded the choropleth itself with log-scale intensity, a hover tooltip, and a colour-ramp legend. Also caught and fixed two stealth UX bugs along the way — CPV codes that read as tender counts, and an empty sector grid on countries with zero corpus.

2026-05-12DataMesh Consulting

12 May — SEO recovery sprint, sitemap drops 8 548 → 2 186 URLs, freshness pipeline live

A Google Search Console audit reported 9 189 URLs in our sitemap but only 21 crawled and 0 indexed. We traced it back to junk-titled tenders inflating the corpus with auto-generated-looking pages, then shipped a five-commit fix that cleans them out at the source, tightens the sitemap to high-confidence URLs only, and wires up an end-to-end freshness pipeline so future tenders reach Google within minutes.

2026-05-09DataMesh Consulting

9 May — public portal hardening day, junk filter removes 48% noise, country-first homepage

A long deploy day. Three threads landed in production — backend hardening (throttler + relation filter + junk hide), Phase 3 SEO surfaces, and full DataMesh Consulting Ltd legal-entity disclosure. Plus the homepage now prioritises tenders from the visitor's own country.

2026-05-09DataMesh Consulting Editorial

Launch-week snapshot — 4 966 active tenders, 127 portals, the geography of what's in flight

A first look at the tender pipeline as DataMesh Consulting opens to the public — which countries dominate this week, which sectors are over-indexed against historical norms, and the contracts worth flagging.