How Google deindexed our 9,000-page site — and the recovery

This is the mistake I'd least like to write up, which is exactly why it's worth writing up. We put a young site online, let it auto-generate a page per scraped tender, and Google responded by refusing to index essentially all of it. Here is what happened and how we dug out — numbers included, because a post-mortem without numbers is just an apology.

There's a same-week status note on the recovery sprint if you want the terse changelog version. This is the long, honest one.

The number that ruined a morning

Four weeks after launch, this was Google Search Console for datameshconsulting.co.uk:

• Sitemap: 9,189 URLs submitted
• Crawled: 21
• Indexed: 0

Zero. Over a month online we'd asked Google to discover nine thousand URLs, it had crawled twenty-one of them, and indexed none. The bulk sat in "Crawled — currently not indexed" and "Discovered — currently not indexed," which are the buckets you least want to be in: Google either looked and decided the page wasn't worth keeping, or found the URL and didn't even bother fetching it.

The instinct is to treat that as 9,000 individual page problems. It isn't. It's one domain problem. Google assigns a young site a small crawl budget and a provisional quality estimate, and if the pages it samples look thin or auto-generated, that estimate drops for the whole domain — including the good pages it hasn't crawled yet. A minority of junk doesn't get individually filtered; it poisons the site-wide signal. We were on track for permanent invisibility.

What we'd done wrong

We generate a page per tender from scraped procurement notices. Scrapers being scrapers, a meaningful fraction of those notices came in with titles that were never meant to be titles:

• Pure numbers — 134022360.00 — where a portal's KPI field leaked into the

title slot.

• Bare reference codes — 746-26-835-PS.
• Portal chrome captured as content — Plataforma de Contratación del Estado,

Öffentliche Ausschreibungen, TED - Tenders Electronic Daily.

• Anti-bot and error pages that got scraped as if they were tenders — "Request

Rejected", "Just a moment…", "Service Unavailable".

• On one memorable occasion, our own site <title> captured as a tender.

Individually, each is a shrug. In aggregate, thousands of near-duplicate, machine-looking pages are a textbook low-quality-domain signal. We'd built a junk factory and pointed Google's sitemap straight at it.

Fix layer 1 — stop minting junk at the source

The first fix is the least glamorous and the most important: reject junk at insert time, in the backend, so it never reaches the database or the sitemap in the first place. That's a growing list of patterns — form-label leaks, camelCase machine identifiers, error and anti-bot page titles — checked before a tender is written:

// A representative slice of GARBAGE_TITLE_PATTERNS (backend, insert-time gate)
/^request\s+rejected\b/i,               // F5 BIG-IP ASM block page
/^just\s+a\s+moment/i,                  // Cloudflare interstitial
/^(?:40[0-9]|50[0-9])(?:\s*[-:]\s*|\s+)\S/,  // "404 Not Found", "503 - ..."
/^service\s+unavailable\s*$/i,
/select[A-Z]/,                          // camelCase machine identifier leak
/\b0\/\d+\b/,                           // placeholder counters like "0/4000"
/screen\s*reader\s*access/i,            // portal chrome scraped as a title

There was a nasty twist here. The filter existed but two ingestion paths passed null where the quality checker should have been, so those paths wrote straight past the gate. Closing them took fresh-ingest junk quality from 92% to 99%. A one-off archive pass then flipped 187 historical junk tenders from active to archived — 99 pure-numeric, 11 bare reference codes, 9 portal homepages, and the rest. The factory was shut off and the back-catalogue swept.

Fix layer 2 — a sitemap that only advertises pages worth indexing

Cleaning the corpus is necessary but not sufficient. The sitemap is a promise to Google — "these URLs are worth your crawl budget" — and ours was writing cheques the content couldn't cash. So eligibility got strict: a URL only enters the sitemap if its title is junk-free, its description is at least 50 characters, it has a named contracting authority, and it was published within the last 90 days.

export function isSitemapEligible(t): boolean {
  if (isJunkTitle(t.title ?? null)) return false;
  // Content gate -- a tender with no description or a one-liner is exactly
  // the "thin content" signal Google downgrades on.
  if ((t.description ?? '').trim().length < 50) return false;
  // Org gate -- without a contracting authority, the page is largely
  // unanchorable to entity-graph signals.
  if ((t.organization ?? '').trim().length === 0) return false;
  // Recency gate -- Google's first crawls should land on fresh content.
  if (t.publishedAt) {
    const ageDays = (Date.now() - Date.parse(t.publishedAt)) / 86_400_000;
    if (Number.isFinite(ageDays) && ageDays > 90) return false;
  } else {
    return false;   // no publishedAt at all -- usually a thin extraction
  }
  return true;
}

The sitemap dropped from 8,548 URLs to 2,186 — a 4× cut. That feels wrong the first time you do it; you're voluntarily hiding two-thirds of your pages from Google. But it's the core counterintuitive lesson of the whole episode: on a young domain, a smaller sitemap of high-confidence URLs gets crawled and indexed faster than a big sitemap padded with thin ones. Crawl budget is the scarce resource. Don't spend it advertising pages that will land in "not indexed" and drag the domain average down with them.

The subtle one — lastmod you can actually trust

With the corpus clean and the sitemap tight, recrawl was still sluggish, and the cause was embarrassing. Every ISR regeneration and every webhook rebuild stamped roughly 3,700 aggregate pages with <lastmod> = the moment they rendered. To Google that reads as "this URL changes every few minutes, and yet every time I crawl it nothing's different" — so it learns to distrust your freshness signals and slows down.

The fix is to make lastmod true: bucket it to UTC-midnight so it only changes once a day, and for aggregate pages derive it from the newest tender they actually contain, rather than from render time.

// Freshness signals have to be TRUE to be useful. Bucket to UTC-midnight...
const dayStamp = new Date(`${now.toISOString().slice(0, 10)}T00:00:00.000Z`);
// ...and derive each aggregate's lastmod from its newest constituent tender:
const stamp = (epoch) => (epoch && epoch > 0 ? new Date(epoch) : dayStamp);

The outage where an empty 200 was worse than a 5xx

Then, during a June backend outage, we nearly undid all of it in a single request. The tender feed the sitemap builder reads returned empty. Without a guard, the builder would have happily serialised an empty <urlset> and served it with 200 OK — which tells Google, authoritatively, "this site has zero indexable pages." An empty, successful sitemap is a deindex instruction.

The defense is to treat "the corpus came back empty" as a failure to serve, not a valid state. The builder flags the model degraded when the scan returns nothing, and the route refuses to render it:

function assertServableModel(model) {
  if (model.degraded) {
    // Throw rather than serve an empty 200. Next's stale-while-error keeps the
    // last good ISR copy; a cold instance serves 5xx, which Google treats as
    // transient. Both are strictly better than the empty-200 that deindexes.
    throw new Error('sitemap model degraded — refusing to serve empty urlset');
  }
  return model;
}

A 5xx costs you nothing — Google shrugs and keeps the sitemap it already has. An empty 200 costs you the whole index. The one exception is build time: if the backend is down during a deploy we do bake a thin sitemap, so shipping isn't blocked, and the first healthy regeneration replaces it.

What recovered, honestly

I'm not going to hand you a triumphant "and then we hit N indexed pages" number, because the clean before/after that makes a tidy story isn't one I can stand behind. What I can tell you is concrete: the corpus is swept and the insert-time gate holds fresh junk at 99%; the sitemap is curated to ~2,186 high-confidence URLs and later trimmed further, to a 2,000-tender cap with relevance tiering, while Google worked through a backlog of ~4,919 "Discovered — currently not indexed" — the crawl-budget starvation you'd expect on a young domain that asked for too much too early. The trajectory is right and the signals that caused the cascade are gone. Recovery on a young domain is measured in weeks of patient recrawling, not a single deploy.

The takeaways

• Junk is a domain-level tax, not a per-page one. A 10–15% junk rate in

your sitemap can collapse crawl budget for every page, including the good ones.

• Curate the sitemap; don't dump it. Fewer high-confidence URLs beat a big

thin list on a young domain.

• lastmod must be true. Stamping render-time on unchanged pages destroys

the freshness trust it's supposed to build.

• An empty 200 is worse than a 5xx. Never serve a successful-but-empty

sitemap; throw and let stale-while-error or a transient error protect you.

• *Filter at the source and at render. Two independent gates mean a miss in

one is caught by the other.

---

We build the extraction and data-quality pipelines behind sites like this — including the title-quality and sitemap-eligibility logic that keeps a programmatic site on the right side of Google's quality signals. If you're scaling a content or data site and want it done right, here's what we do.*