Why ScheduleWizard accounted for 7 of 24 timeouts in one window

524s on the main site, measured over five complete days: 6 on the 24th, 9 on the 26th, 1 the day after. The Dev Notes on the ticket had called it one account’s data hitting a slow path. That reading died the moment the flood that supposedly caused it was gone and the timeouts kept showing up on separate days anyway.

Two codes, one fuse

Cloudflare gives an origin 100 seconds before it gives up and returns a 524. Over that window the main site logged 40 failed requests total. 24 were 522s on / and /accounts/default, and 16 were 524s. Same family, different half of the same fuse. A 522 means the connection to the origin died before the 100 second ceiling. A 524 means the origin was still working when the ceiling hit. Both point at the same thing: something on the other end is slow, and Cloudflare doesn’t care why.

The baseline that was about to move

A companion ticket had already measured why. /accounts/default -> home on an org site was pulling miniScheduleWide at 11,682ms, versus 330ms on a plain site, a 35x gap for what’s nominally the same widget. Two other sections of the page were 3.9x and 4.5x slower on the org path. The query count on that page dropped from 255 to 123 once the fix landed, and a helper called InactiveExludeSQLGet that had been firing 34 times for 1,479ms got memoized down to one call.

That fix was real and measured, and it was still on develop, not master. So every number on the timeout ticket was a baseline that was about to move under it. Re-running the Cloudflare counts against that ticket before the fix shipped would have meant tuning against a floor already scheduled to drop.

Pages that were never going to finish

When we went through the pages on the timeout list, they split into two categories, and query tuning touches them very differently.

Four of the worst offenders had their script timeout set higher than Cloudflare’s edge ceiling:

• ScheduleWizard: 180s
• Registration_FileUpload: 600s
• MonitorReport: 2400s
• MailerOrderSend: 2700s

ScheduleWizard alone accounted for 7 of the 24 measured timeouts in the window. Across that whole set, the server was configured to let the script run for anywhere from 3 minutes to 45 minutes past the point where Cloudflare had already decided to hang up and return an error. It doesn’t matter if the underlying query gets 4x faster. Shave MailerOrderSend from 2700 seconds down to 900 and it is still 9x past the 100 second ceiling it will die against on the edge.

That is an async/chunking problem. The work has to move off the request/response cycle entirely, either queued and polled or broken into pages, because no amount of speeding up a synchronous script gets it under a ceiling it was never configured to respect.

The pages where tuning might work

The other bucket is /admin/home, pageedit, and accountinfo, which came in through a separate ticket reporting IIS killing scripts at the origin rather than Cloudflare timing out at the edge. Those pages don’t set an inflated script timeout. They pull AppHQ, Prefs, and Members through a shared Common include, the same include chain the org-site fix touched. Nobody had measured those three pages directly yet, but they sit on the exact code path that was already getting rewritten. That bucket is a real candidate for query optimization to just work.

One ticket, two ends of a connection

We merged the two tickets into one because they were the same failure measured from opposite ends. IIS killing the script at the origin is one ticket’s view, and Cloudflare giving up waiting for that same script is the other’s. When you’re staring at an error rate dashboard, keep this straight: a 522 and a 524 in the same week are one slow origin observed from both sides of a 100 second wall.

So the check I’d run first is which pages have a script timeout configured above that wall. Those pages need their work queued and polled or broken into pages, since faster queries behind them leave the ceiling where it is. Take any measurement for the rest only after the pending fix has shipped, so you’re measuring the floor as it will actually be.

dxdev LESSON · paste into your AI coding agent
LESSON: A page whose own configured timeout exceeds the edge's timeout ceiling can never succeed, no matter how much the query is optimized
SOURCE: dxdev.com/blog/timeout-ceiling-blocks-optimization
WHAT HAPPENED: A cluster of edge timeout errors was first attributed to one account's data hitting a slow query path, a theory that died once the errors kept recurring on separate days after that account's load had cleared. The affected pages split into two genuinely different categories. Four pages had their own server-side script timeout explicitly configured well above the fixed timeout ceiling enforced by the edge in front of them, meaning even a substantial query speedup on the worst offender would still leave it many times past the ceiling it dies against, since the real fix required moving that work off the synchronous request cycle entirely, queued and polled, or broken into smaller chunks, rather than making the same synchronous work faster. A separate set of pages hit a timeout through a different mechanism, the origin server itself killing the script rather than the edge giving up waiting, and sat on a shared code path that an unrelated optimization effort was already rewriting, making those genuine candidates for a query-tuning fix to actually work. Two separate tickets describing what turned out to be the same underlying slow-origin condition, observed from opposite ends of the same timeout boundary, were merged into one investigation once that was recognized.
THE RULE: Before treating a timeout error as a performance problem to optimize away, check whether the affected page's own configured timeout already exceeds the timeout ceiling of whatever sits in front of it, such as a CDN or a load balancer; if it does, no amount of query optimization can succeed, because the page was architecturally never going to finish inside the window that actually decides its fate, and the real fix is moving the work off the synchronous request cycle entirely.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any page or endpoint experiencing edge or gateway timeouts that has not been checked for whether its own configured request timeout already exceeds the timeout ceiling of any proxy or load balancer in front of it.
      2. Two separate timeout-related error codes being investigated as different bugs, when they could be the same slow-origin condition observed from opposite ends of the same time boundary.
      3. A query or performance optimization being planned for a timing-out page without first confirming that speeding up the query could actually bring it under the relevant timeout ceiling, rather than just making an already-doomed request marginally less doomed.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.