A routine JIRA ticket-create came back 403 Basic Authentication Failure - AUTHENTICATION_DENIED with credentials we’d been using all session. The natural first reaction is “the password rotated” or “the token’s revoked.” Both were wrong. The create call wasn’t the problem at all. Our earlier JIRA search calls had been failing against the wrong endpoint, and that is what tripped a CAPTCHA challenge on the account. By the time we got to the create, even correct credentials were refused.

Nothing about our creds changed. We attacked our own account with a wrong endpoint, and the server did what it’s supposed to do against something that looks like a brute-force attempt.

One client, two backends, one hardcoded path

We have a shared Jira class that talks to two different instances:

  • Our HQ project on a separate site, which is JIRA Cloud on REST v3.
  • The SaaS platform on <jira-server-host>, which is JIRA Server on REST v2.

Same Python class, same auth shape, different REST API versions under the hood. That difference is invisible at the call site, which is the trap.

The search() method hardcoded the Cloud path:

/rest/api/3/search/jql

Against the Cloud instance, fine. Against the Server instance, that path does not exist and it 404s. Server JIRA wants:

/rest/api/2/search

So any code that reached for the obvious .search() against the SaaS platform’s Server instance got a 404. The code that “worked” only worked because someone had hand-split it to call search_v2() explicitly. The natural method name was a landmine, and only the people who already knew about it had learned to step around it.

How a 404 becomes a 403

A 404 from a single bad path is harmless on its own. Run it in a loop, though, and from the server’s side you are a client hammering an endpoint with auth headers and getting rejected over and over. That pattern looks the same as a credential-stuffing attempt.

JIRA answers with a CAPTCHA challenge. The tell is in the response header:

X-Authentication-Denied-Reason: CAPTCHA_CHALLENGE

Once that challenge is armed, the account is gated. Correct username, correct password, correct token, it makes no difference. The server answers 403 AUTHENTICATION_DENIED until a human completes the CAPTCHA through a web login. The API has no way to clear it. You have to go to the browser.

Retrying makes it worse. Every additional failed request while the challenge is live counts as one more failed login, which keeps the lock fresh. The instinct under a 403 is to retry with slightly different auth handling, and that instinct deepens the hole.

Getting out of the lock

Once you recognize a CAPTCHA lockout, stop hitting the API. Each retry is another tally against you.

The unlock is a human-only step by design. So you drive a browser you control, fill in the username, and hand the actual CAPTCHA to a person to solve. The lockout response even gives you the login URL: login-url=https://<jira-server-host>/login.jsp. Once the web login clears the challenge, the account is good again and the API calls work with the same creds that were “failing” a minute earlier. The lock was the problem, not the creds.

The fix: auto-route instead of hand-splitting

Clearing the lock gets you working again. It doesn’t stop the next person, or whoever comes next a month from now, from writing the obvious .search() against the SaaS platform and re-triggering the whole thing.

So the fix went into the client. First we confirmed the failure empirically instead of trusting our own read of it: .search() does 404 against the SaaS platform’s Server instance, and .search_v2() works there.

Then search() learned to route itself. The constructor now derives is_cloud from the base URL, and search() picks v3 for Cloud and v2 for Server on its own. search_v2() stays as an explicit escape hatch for anyone who wants to force the Server path. The commit message states the goal bluntly:

jira_api: search() auto-routes Cloud-v3 vs Server-v2 (fixes 404 footgun)

This changes nothing for callers who were already correctly hand-split. They keep working exactly as before. It only rescues the people who reached for the natural method name and would otherwise have broken. The fix is invisible to anyone doing it right and a safety net for anyone doing it the obvious way.

Verified live after the change: .search() on <jira-server-host> returns issues where it used to 404, and the Cloud detection lands correctly.

Two shell traps on the way

While untangling this we hit two adjacent self-inflicted wounds, both in the same family of an error message lying to you about the cause.

curl -u choked on a password containing special characters. It came back as a 401 that read like a genuine auth failure, but shell quoting was mangling the password before it ever left the machine. The Python client, which never round-trips the password through a shell, authenticated fine with the same value. A 401 from curl -u is no proof your credentials are wrong. It might just be your shell.

Then the password got overwritten when we meant to source it. We ran something like:

export JIRA_PASSWORD="$JIRA_PASS"

The variable that actually existed was JIRA_PASSWORD. There was no JIRA_PASS, so $JIRA_PASS expanded to empty, and that export overwrote a correctly sourced password with an empty string. “The creds are empty” was true, but the cause wasn’t a missing secret. We had stomped a good value with a typo’d variable name. Same shape as a payment-gateway bug from the same day, where an empty AGENT_USERNAME= in a later-loaded .env-local silently overrode the real value from .env-dev. A blank value overwriting a good one is a recurring way to manufacture an auth failure out of nothing.

What to carry out of it

A shared client over two backends should route itself or avoid a backend-agnostic method name. Here search() worked on Cloud and returned a 404 on Server, and only the people who already knew to avoid it were safe. When authentication suddenly returns a 403 with credentials you know are good, stop retrying and rotating secrets and look for an earlier call failing in a loop, because that can trigger a CAPTCHA lockout on your own account. Clearing it takes a person completing the CAPTCHA through a web login, and the lasting fix is making the wrong call impossible to write by accident.