A 59% Signup Jump Right After Shipping Structured Data

The alert said signups were up 59% week over week. Against the prior two years’ seasonal baseline for the same week, that looked unusual enough to be a real signal, and the obvious suspect was the SEO work. We had just added structured data to the individual FAQ article pages and shipped a new glossary page with its own signup link. Something shipped, and the spike came right after.

The first wrong turn: crediting the glossary link

The glossary link has its own tracking, so we pulled sign_up events attributed to it and compared them against total tracked signups for the trailing three weeks. It came back at 0.49%, under one signup in two hundred. That was not the cause, and not close.

That should have been the point where we widened the investigation. The timing still lined up, though, so the SEO angle felt live, and we kept pulling on it before turning to signup composition directly.

What the referral fields were actually doing

The SEO theory died when we looked at where the referral data came from. No second attribution query was involved.

• One tracking field is only populated from an explicit ?src= or ?source= query param, and today only paid ads set that param.
• A second field is a long-lived cookie, but it is only written if a visitor’s very first pageview ever on the site had an external referrer.
• Direct visits, bookmarks, and anyone who clicked through mid-visit (glossary to signup, help topic to signup) never populate it.

The same capture logic was copy-pasted near-identically across four separate site templates.

When we ran the numbers, 55% of that week’s signups had both fields blank. They were not attributed to search or to anything else. More than half the funnel was invisible to the tracking, all the time, not just during the spike. The 59% jump was ordinary seasonal growth arriving on top of a measurement system that was already blind to most of its own traffic.

The fix

That gap became the ticket for this fix, filed as a hotfix and cut from the production branch into its own working branch.

• Accept a UTM source param as an alias for the legacy param, so tagged campaigns stop getting dropped.
• Capture UTM medium, campaign, term, and content into a new long-lived cookie, deliberately not wired to any database write yet.
• When the referrer cookie exists (a confirmed external referrer) but the source field is blank, derive a coarse source label from the referrer’s domain server-side, instead of leaving it hard-blank.
• Tag internal links with their own source going forward, starting with the glossary link.

The one bug review caught

It was in that last piece. The derivation had to fire only when the referrer was confirmed external, using the existing cookie as the gate, and not on every blank source field. The first pass had that gate loose enough that an internal click-through, glossary to signup with no tag, was at risk of being labeled as an external referral instead of left alone.

That gate mattered more than the rest of the ticket combined. The failure mode was mislabeling rows inside the 55% that had no attribution at all, turning an honest blank into a false external referral.

Before anything touched production, we ran a review pass with a second agent, explicitly told not to trust the implementing agent’s summary. It had to:

• read the actual diff, not the description of it;
• confirm the external-referrer gate holds on every changed path;
• confirm the legacy param behavior is unchanged;
• confirm the new UTM cookie really is cookie-only, with no database write;
• reproduce the claimed test scenarios independently rather than re-running the existing harness blindly.

That pass confirmed the gate fix closed the mislabeling path across all four files, not just the one where it was first caught.

Shipped, and then a second gap

It shipped as a hotfix, verified live on production. A teammate then found the same kind of gap one layer up. The structured data fix credited for part of the spike story had gone out on the individual FAQ article pages, but not on the FAQ index pages that link to them. That went out as a second hotfix on the same ticket.

So there were two real fixes, and neither was the explanation for the number that started the investigation. The 59% was just the season. The blank referral data was the thing worth finding, and it had been sitting there under working-looking numbers the whole time.

If a metric jumps right after a release, check how much of the underlying data carries a source before you credit the release. Here 55% of that week’s signups had both referral fields blank, so the tracking could not attribute most of them to anything. The share of conversions with no source at all is a number worth having on hand before the next spike.

dxdev LESSON · paste into your AI coding agent
LESSON: A Coincidence in Timing Is Not a Causal Link
SOURCE: dxdev.com/blog/signup-spike-was-seasonal-not-the-seo-fix
WHAT HAPPENED: A metric jumped right after a change shipped, and the change got credit by default. The real test, a direct attribution query against the specific mechanism, ruled it out at under one percent. The actual driver was a seasonal pattern visible in the same week the prior two years running. Chasing the wrong explanation anyway surfaced a real gap: the tracking that would have settled the question on the first look was blind to more than half of all signups, not just the ones in the spike.
THE RULE: When a metric moves right after a change ships, test the specific mechanism directly before crediting the change. If the attribution data itself has a large blind spot, that gap is usually the more valuable finding than whichever story explains the spike.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any dashboard or alert that reports a metric change without also reporting what share of the underlying events have no attribution or source data at all.
  2. Any referral, UTM, or source-capture logic duplicated across multiple entry points that could silently drift out of sync with each other.
  3. Any code path that would label internal, same-site navigation as an external referral if a upstream flag or cookie is empty when it shouldn't be trusted.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.