504 Gateway Timeout nginx: meaning and how to fix it

A 504 Gateway Timeout means a proxy (nginx, an ingress controller, a load balancer) stopped waiting for the server behind it and answered the client itself; the upstream may be slow or unreachable, but it sent nothing in time. In nginx the error log says which wait ran out: upstream timed out (110: ...) while reading response header from upstream means the upstream accepted the connection and did not answer within proxy_read_timeout (60s by default), and while connecting to upstream means it could not be reached within proxy_connect_timeout. Confirm which by timing one request straight to the upstream, bypassing the proxy. The usual fix is in the upstream (most often a dependency call with no timeout); raising the proxy timeout only makes users wait longer for the same slow answer.

At a glance

What does 504 Gateway Timeout mean?

RFC 9110 defines it in one sentence: the server, "while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request." Two things follow. The 504 is generated by the proxy, never by your application, so the application's own logs will not contain it. And "timely" is whatever the proxy was configured to wait, so the same slow upstream can produce a 504 at one hop and a client-side timeout at another.

The body tells you which hop gave up. Stock nginx (our run, nginx:1.27-alpine):

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.27.5</center>
</body>
</html>

If the footer names a different server (a cloud load balancer, a CDN), the timeout that fired is theirs, and the nginx below it may never have seen the request finish either way.

504 vs 502: how do the nginx logs differ?

We put one nginx in front of a small test upstream and produced each failure on purpose (Docker, nginx:1.27-alpine, proxy_connect_timeout 2s, proxy_read_timeout 3s, a log format that adds $request_time, $upstream_connect_time and $upstream_header_time). What the client got and what nginx wrote:

The access log lines for the failing and slow cases, as written (trimmed: the 1 second 200 and the interleaved error log lines are left out):

192.168.156.1 [29/Oct/2026:15:56:53 +0000] "GET /app/sleep/5 HTTP/1.1" 504 rt=3.004 urt=3.004 uct=0.001 uht=- us=504 ua=192.168.156.2:8080
192.168.156.1 [29/Oct/2026:15:56:55 +0000] "GET /dead/ HTTP/1.1" 504 rt=2.002 urt=2.003 uct=- uht=- us=504 ua=192.0.2.10:8080
192.168.156.1 [29/Oct/2026:15:56:55 +0000] "GET /closed/ HTTP/1.1" 502 rt=0.000 urt=0.000 uct=- uht=- us=502 ua=192.168.156.2:9999
192.168.156.1 [29/Oct/2026:15:56:55 +0000] "GET /app/close HTTP/1.1" 502 rt=0.000 urt=0.001 uct=0.000 uht=- us=502 ua=192.168.156.2:8080
192.168.156.1 [29/Oct/2026:15:57:05 +0000] "GET /app/drip/5/2 HTTP/1.1" 200 rt=10.005 urt=10.004 uct=0.000 uht=0.001 us=200 ua=192.168.156.2:8080
192.168.156.1 [29/Oct/2026:15:57:06 +0000] "GET /app/sleep/5 HTTP/1.1" 499 rt=1.006 urt=1.006 uct=0.001 uht=- us=- ua=192.168.156.2:8080

Three rules come out of this table:

• A 504 is exactly as slow as a timeout. In this matrix every 502 took one or two milliseconds and every 504 took the configured timeout plus a few milliseconds. If your 5xx latency sits on a round number, it is a timeout. The reverse does not hold: a 502 is only fast when the upstream fails at once.
• uct separates the two 504s. A number means nginx connected and then waited for the header (slow upstream). A dash means it never connected (unreachable upstream).
• proxy_read_timeout is not a total. The drip request took 10 seconds under a 3 second read timeout and succeeded, because the nginx docs set the timeout "only between two successive read operations, not for the transmission of the whole response." It bounds silence, not duration.

The exact error text depends on the C library nginx was built against. The Alpine image (musl) prints (110: Operation timed out); the Debian nginx:1.27 image (glibc) prints the string most people search for, (110: Connection timed out), from the same nginx/1.27.5:

2026/10/29 15:57:37 [error] 22#22: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.156.1, server: , request: "GET /app/sleep/5 HTTP/1.1", upstream: "http://192.168.156.2:8080/sleep/5", host: "localhost:18505"

Search your logs for upstream timed out and the while ... clause, not the text in brackets.

How do you confirm the 504 comes from nginx in Kubernetes?

The lab scenario: a three-node kind cluster running the lab's shop app (a load generator calls frontend, nginx proxies /api/ to the api Service, the api pings redis). An api release adds a stock-level lookup to every /items request: urllib.request.urlopen("http://192.0.2.10:8080/stock") with no timeout. 192.0.2.10 is a TEST-NET-1 address that is never routed, standing in for a dependency behind a firewall that drops packets. /healthz does not make the call.

The first surprise: the load generator never sees a 504. It waits 3 seconds (wget -T 3), so it gives up first:

kubectl -n shop logs deploy/loadgen --tail=6

2026-10-29T16:05:12Z GET /api/items -> timeout
2026-10-29T16:05:17Z GET /api/items -> timeout
2026-10-29T16:05:22Z GET /api/items -> timeout
2026-10-29T16:05:27Z GET /api/items -> timeout
2026-10-29T16:05:32Z GET /api/items -> timeout
2026-10-29T16:05:37Z GET /api/items -> timeout

nginx records those as 499 (client closed the request). The one 504 in this output is the scenario's own check request, which waited long enough and got it 60 seconds after it arrived:

kubectl -n shop logs -l app=frontend --tail=200 --prefix | grep -v healthz | tail -12

[pod/frontend-76d98597b9-fxhdj/nginx] 10.244.1.15 - - [29/Oct/2026:16:04:27 +0000] "GET /api/items HTTP/1.1" 499 0 "-" "Wget" "-"
[pod/frontend-76d98597b9-fxhdj/nginx] 2026/10/29 16:04:28 [error] 28#28: *5 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.244.1.15, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.165.139:8080/items", host: "frontend"
[pod/frontend-76d98597b9-fxhdj/nginx] 10.244.1.15 - - [29/Oct/2026:16:04:28 +0000] "GET /api/items HTTP/1.1" 504 167 "-" "Wget" "-"
[pod/frontend-76d98597b9-fxhdj/nginx] 10.244.1.15 - - [29/Oct/2026:16:04:37 +0000] "GET /api/items HTTP/1.1" 499 0 "-" "Wget" "-"
[pod/frontend-76d98597b9-fxhdj/nginx] 10.244.1.15 - - [29/Oct/2026:16:04:47 +0000] "GET /api/items HTTP/1.1" 499 0 "-" "Wget" "-"

(trimmed: seven more 499 lines through 16:05:32. kubectl logs -l prints one pod after the other, so tail shows only the second pod.) A client with a longer patience, like a browser, gets the 504:

kubectl -n shop exec deploy/loadgen -- sh -c "time wget -q -S -O /dev/null -T 90 http://frontend/api/items"

  HTTP/1.1 504 Gateway Time-out
wget: server returned error: HTTP/1.1 504 Gateway Time-out
Command exited with non-zero status 1
real	1m 0.02s

One minute is nginx's default proxy_read_timeout; the frontend config sets only proxy_connect_timeout 2s. If you count only 504s you undercount this incident badly: the 499s are the same failure seen by impatient clients.

Look-alikes: a 504 whose body footer is not nginx came from a hop in front of it (check that hop's timeout and logs). A 502 with Connection refused is a Service with no endpoints or a wrong port (Service has no endpoints, targetPort mismatch). A 503 from the api itself (backend unavailable in the lab app) is the application answering, quickly, with an error.

How much is affected, and since when?

Everything on the routing layer is healthy, which is what makes 504s confusing:

kubectl -n shop get pods -o wide
kubectl -n shop get endpointslices -l kubernetes.io/service-name=api

NAME                        READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
api-78449dbcf8-6r4qj        1/1     Running   0          2m18s   10.244.2.10   r504-worker2   <none>           <none>
api-78449dbcf8-nj9kr        1/1     Running   0          2m19s   10.244.1.16   r504-worker    <none>           <none>
frontend-76d98597b9-fxhdj   1/1     Running   0          2m26s   10.244.2.9    r504-worker2   <none>           <none>
frontend-76d98597b9-q6rx2   1/1     Running   0          2m26s   10.244.1.14   r504-worker    <none>           <none>
loadgen-7c6676864-bd64s     1/1     Running   0          2m26s   10.244.1.15   r504-worker    <none>           <none>
redis-59458f55d8-r7x6w      1/1     Running   0          2m26s   10.244.1.11   r504-worker    <none>           <none>
worker-6cf88d4b58-8bjvm     1/1     Running   0          2m26s   10.244.1.13   r504-worker    <none>           <none>

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                 AGE
api-jpfcs   IPv4          8080    10.244.1.16,10.244.2.10   2m26s

Both api pods are Ready, the Service has both endpoints, nothing restarted. The api pods' AGE (2m18s, against 2m26s for everything else) is the scope clue: the api was rolled out after the rest, and the failures start with it. On a real cluster, put the first upstream timed out line next to kubectl -n <ns> rollout history deploy/<upstream> and the ConfigMap or image change that went with it. Scope by path too: in the lab only /items failed; /healthz on the same pods stayed fast, which already says the problem is inside one code path, not the pod or the network.

Is the upstream slow, unreachable, or is the timeout too low?

Cheapest check first. Stop at the step that answers it.

1. Read the while ... clause. while reading response header from upstream means connected and waiting (go to step 2). while connecting to upstream means the upstream address is not answering at TCP level: from the proxy's pod, connect to the address in the upstream: field; a NetworkPolicy, a dead node or a wrong IP drops the SYN (see NetworkPolicy blocking traffic). Through a ClusterIP there are two common versions. A port the Service does not publish times out rather than being refused: in our ConfigMap drift scenario one frontend pod proxied to api:8000, logged upstream timed out (110: Operation timed out) while connecting to upstream after its 2 second proxy_connect_timeout, and returned 504. Otherwise a pod endpoint is not answering at TCP level. A Service with no endpoints at all refuses instead, which is a fast 502.

2. Time the upstream directly, bypassing nginx. From any pod, call the Service or pod on its own port, once on the health path and once on the failing path:

kubectl -n shop exec deploy/loadgen -- sh -c "time wget -q -O - -T 5 http://api:8080/healthz"
kubectl -n shop exec deploy/loadgen -- sh -c "time wget -q -O - -T 5 http://api:8080/items"

{"status": "ok"}real	0m 0.00s

wget: download timed out
Command exited with non-zero status 1
real	0m 5.00s

If the direct call is also slow, nginx is reporting the truth: the upstream is slow. If the direct call is fast and only the proxied one times out, look at the proxy (wrong upstream, a buffered request body, a hop in between).

3. Find what the upstream is waiting on. A slow handler is usually blocked on something outside the process. Socket states inside the pod show it without any tracing tool (Alpine and busybox images ship netstat):

kubectl -n shop exec api-78449dbcf8-6r4qj -- sh -c "netstat -tn | awk 'NR>2{print \$6, \$5}' | sed -E 's/:[0-9]+\$//' | sort | uniq -c"

      8 CLOSE_WAIT 10.244.1.14
      1 CLOSE_WAIT 10.244.1.15
      5 CLOSE_WAIT 10.244.2.9
     14 SYN_SENT 192.0.2.10
     16 TIME_WAIT 10.244.2.1
      8 TIME_WAIT 10.96.245.99

Fourteen connections stuck in SYN_SENT to one address: the api is trying to reach 192.0.2.10 and getting nothing back. The CLOSE_WAIT rows from 10.244.1.14 and 10.244.2.9 are the two frontend pods: nginx already gave up and closed its side, and the api has not noticed. SYN_SENT means an unreachable dependency; many ESTABLISHED connections to a database with no SYN_SENT means a slow one. Either way, the next place to look is the code that makes that call (here, the new urlopen line in the api-code ConfigMap).

4. Read the upstream's own log, and distrust it. About two minutes after each request, the api finally wrote (kubectl -n shop logs api-78449dbcf8-6r4qj --tail=40, trimmed to these four lines; the first header line of a traceback was interleaved just above them):

stock lookup failed: <urlopen error [Errno 110] Operation timed out>
10.244.1.14 "GET /items HTTP/1.0" 200 -
request failed: redis error: [Errno 32] Broken pipe
10.244.1.14 "GET /items HTTP/1.0" 503 -

followed by a BrokenPipeError traceback. Three things in four lines are misleading. The api logged 200 for a request nginx had already given up on, as a 504 or a 499 (Python's http.server logs when it starts the response, not when it is delivered). It then blamed redis, because the lab app's catch-all handler labels every OSError a redis error, and the broken pipe to nginx is an OSError. And none of this appears until the dependency call gives up: with no timeout set, Python's urlopen falls back to the global default, which is no timeout, so the kernel decides. Linux retries a SYN tcp_syn_retries times (6 by default), and the kernel docs put the final timeout at 131 seconds; with the proxy timeout raised to 150 seconds (see the fix section), one request took 2 minutes 13 seconds end to end. During those two minutes the upstream's log is silent about the failing requests.

5. If the upstream is not broken, compare its normal latency with the timeout. This is the "timeout too low" case, and it looks different in the access log: some requests to one path succeed just under the timeout and the rest fail exactly at it. Same Docker nginx, proxy_read_timeout 3s, test requests that take 2.0 to 3.5 seconds:

"GET /app/sleep/2.5 HTTP/1.1" 200 rt=2.501 urt=2.501 uct=0.000 uht=2.501 us=200 ua=192.168.156.2:8080
"GET /app/sleep/2.9 HTTP/1.1" 200 rt=2.900 urt=2.901 uct=0.001 uht=2.901 us=200 ua=192.168.156.2:8080
"GET /app/sleep/3.1 HTTP/1.1" 504 rt=3.005 urt=3.006 uct=0.000 uht=- us=504 ua=192.168.156.2:8080
"GET /app/sleep/3.5 HTTP/1.1" 504 rt=3.002 urt=3.003 uct=0.000 uht=- us=504 ua=192.168.156.2:8080

(trimmed: lines cut to the request onward, and the 2.0 second 200 and the two error log lines left out.) The tell is a continuous latency distribution that runs into the timeout and is cut off there, with no change on the upstream side. A slow upstream instead shows a jump: latency was low until a deploy or a dependency event, then everything sits at the timeout. In the lab scenario, every /items request failed and none succeeded slowly; that is not a tight timeout.

6. If the 504 took longer than the timeout, look for retries. With more than one server in an upstream block, nginx's default proxy_next_upstream error timeout sends a timed-out request to the next server. Two slow servers, a 3 second read timeout (Debian nginx this time):

192.168.156.1 [29/Oct/2026:15:57:45 +0000] "GET /pair/sleep/5 HTTP/1.1" 504 rt=6.007 urt=3.004, 3.004 uct=0.000, 0.001 uht=-, - us=504, 504 ua=192.168.156.2:8080, 192.168.156.4:8080
2026/10/29 15:57:45 [warn] 25#25: *5 upstream server temporarily disabled while reading response header from upstream, client: 192.168.156.1, server: , request: "GET /pair/sleep/5 HTTP/1.1", upstream: "http://192.168.156.4:8080/sleep/5", host: "localhost:18505"

(trimmed to the access log line and the second server's warning; the first server's warning at 15:57:42 and both upstream timed out lines are left out.) The client waited 6 seconds, both upstreams received the request, and each was marked down for fail_timeout (10 seconds by default, upstream module). A plain nginx in front of a ClusterIP sees one address, so it does not retry; ingress-nginx sends to pod IPs directly, and it does (next section).

What changes with the Kubernetes ingress-nginx controller?

We installed ingress-nginx controller v1.15.1 in the same cluster (the project's kind manifest), put the broken api back after the fix above (that step is not in the transcript; the two api pods behind the Ingress, 10.244.1.21 and 10.244.2.16, are new ones), and pointed an Ingress at it, with nginx.ingress.kubernetes.io/proxy-read-timeout: "10" (the annotation takes plain seconds, annotations docs). One request:

  HTTP/1.1 504 Gateway Time-out
wget: server returned error: HTTP/1.1 504 Gateway Time-out
Command exited with non-zero status 1
real	0m 30.00s

Thirty seconds for a 10 second timeout. The controller's log shows why:

2026/10/29 16:11:34 [error] 383#383: *1125 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.244.1.15, server: _, request: "GET /items HTTP/1.1", upstream: "http://10.244.1.21:8080/items", host: "ingress-nginx-controller.ingress-nginx.svc"
2026/10/29 16:11:44 [error] 383#383: *1125 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.244.1.15, server: _, request: "GET /items HTTP/1.1", upstream: "http://10.244.2.16:8080/items", host: "ingress-nginx-controller.ingress-nginx.svc"
2026/10/29 16:11:54 [error] 383#383: *1125 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.244.1.15, server: _, request: "GET /items HTTP/1.1", upstream: "http://10.244.1.21:8080/items", host: "ingress-nginx-controller.ingress-nginx.svc"
10.244.1.15 - - [29/Oct/2026:16:11:54 +0000] "GET /items HTTP/1.1" 504 160 "-" "Wget" 110 30.002 [shop-api-8080] [] 10.244.1.21:8080, 10.244.2.16:8080, 10.244.1.21:8080 0, 0, 0 10.001, 10.001, 10.000 504, 504, 504 fe8729b1278e55a4dd47db6ce3f5437e

Three tries, two pods, the first pod twice. The controller's defaults are proxy-next-upstream: "error timeout" and proxy-next-upstream-tries: 3 (ConfigMap docs); the rendered nginx.conf, read after the change below, still shows proxy_next_upstream error timeout and proxy_next_upstream_tries 3 on the location we did not annotate. With the default 60 second read timeout, a GET to a hung backend holds the client for up to three minutes and runs the slow work three times. Removing timeout from the retry list for this Ingress brought it back to one try:

kubectl -n shop annotate ingress api nginx.ingress.kubernetes.io/proxy-next-upstream="error" --overwrite

real	0m 10.00s
10.244.1.15 - - [29/Oct/2026:16:12:23 +0000] "GET /items HTTP/1.1" 504 160 "-" "Wget" 110 10.002 [shop-api-8080] [] 10.244.1.21:8080 0 10.002 504 13c423d79a9bfc63ad9ec2d819fcd7cb

The ingress-nginx access log format (upstreaminfo) already carries $request_time, $upstream_addr, $upstream_response_time and $upstream_status, so the reading rules above apply without changing anything. Note that the Kubernetes project retired ingress-nginx: best-effort maintenance ended in March 2026, with no further releases or security fixes, though existing installs keep working. v1.15.1, the version tested here, was released on 2026-03-19 and is the last controller release; the GitHub repository is now archived.

What if the readiness probe says context deadline exceeded?

That is the same slowness seen by the kubelet. A Go HTTP client, which the kubelet's probe is, reports its own timeout as context deadline exceeded, and a probe's timeoutSeconds defaults to 1 (probe docs). We ran a copy of the broken api whose readiness probe called /items instead of /healthz:

Readiness probe failed: Get "http://10.244.1.22:8080/items": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The pod was still 0/1 after 40 seconds. We ran this on one extra Deployment only. Had every api pod used that probe, the Service would have lost its endpoints and the 504s would have turned into fast 502s with Connection refused (what the frontend logs in this lab while the api has no ready pods, for example during the selftest's startup), pointing the investigation at the Service layer instead of the dependency. A probe that checks a dependency moves the symptom; keep readiness on a path the pod itself controls.

How do you fix a 504 Gateway Timeout?

Fix the cause the steps above found. Mitigation and the real fix are separate here, because the quickest mitigation is the wrong one.

Slow upstream: fix or time-box the call. In the lab the change was the new dependency call, so the real fix is to revert the api-code ConfigMap change, or to give the call a timeout shorter than the proxy's and a fallback. We applied the second:

31c31
<                 with urllib.request.urlopen("http://192.0.2.10:8080/stock") as r:
---
>                 with urllib.request.urlopen("http://192.0.2.10:8080/stock", timeout=1) as r:

kubectl -n shop create cm api-code --from-file=api.py=api-fixed.py --from-file=worker.py=worker.py --dry-run=client -o yaml | kubectl apply -f -
# api-fixed.py is the live api.py with the one-line change; worker.py is unchanged
kubectl -n shop rollout restart deploy/api

Risk: a code rollout; a mistake in the patch takes the api down for every path, so watch the rollout. Rollback: apply the previous ConfigMap and restart again, or kubectl rollout undo deploy/api if the code lives in the image. After it, the same request returned 200 in one second:

{"items": ["kettle", "mug", "teapot"], "stock": {}}real	0m 1.00s

Be honest about what that bought: every request still spends one second failing to reach the stock service and returns no stock data. It turns an outage into a degraded feature. The dependency itself (the address, the firewall rule, the service) still needs fixing, and a circuit breaker that stops calling it after repeated failures would remove the one second too.

Expect the old pods to linger during the rollout (1 old replicas are pending termination in ours). The lab api runs python as PID 1 with no SIGTERM handler, so it ignores SIGTERM and stops only at SIGKILL when the 30 second grace period ends; requests still blocked inside it are cut off then.

Raising proxy_read_timeout: what it actually does. We set proxy_read_timeout 150s on the frontend and sent a patient request:

real	2m 13.35s
{"items": ["kettle", "mug", "teapot"], "stock": {}}

The 504 is gone and the user now waits 2 minutes 13 seconds for a page with no stock data. Every waiting request holds an nginx connection and an api thread, and clients with shorter timeouts (the load generator, mobile apps, the load balancer in front) still fail, now as 499s and their own timeouts that no 504 alert counts. Raise the timeout only when the endpoint is legitimately slow and you have measured it: set it in the one location (or, on ingress-nginx, the one Ingress) that needs it, a little above that endpoint's measured p99, never server-wide. Risk: longer-held connections and workers under load. Rollback: remove the line and reload nginx (nginx -s reload, or restart the pods when the config is a mounted ConfigMap, as here).

Unreachable upstream (while connecting). Fix the address or the network path: the Service's endpoints, the NetworkPolicy, the node. Lowering proxy_connect_timeout (ingress-nginx defaults to 5 seconds; stock nginx to 60) only makes the failure faster, which is worth doing so a dead peer costs seconds, not a minute, but it is not a fix.

Retries multiplying the wait. On ingress-nginx, nginx.ingress.kubernetes.io/proxy-next-upstream: "error" stops retrying timeouts for one Ingress, as shown above; proxy-next-upstream-timeout caps the total time spent retrying instead. Keep timeout retries only if the backend's work is safe to run twice. nginx does not retry non-idempotent methods such as POST once the request has been sent (proxy_next_upstream), but GETs that write are common. Risk of the annotation: a timeout on one pod is no longer hidden by a retry on a healthy one, so a single slow pod now shows as 504s. Rollback: kubectl -n shop annotate ingress api nginx.ingress.kubernetes.io/proxy-next-upstream- removes it.

How do you verify the fix?

Pass condition: on the failing path, a request through the proxy returns 200 well under the timeout, the client's own log shows 200s, and no new upstream timed out line appears for at least ten minutes of normal traffic (longer than one deploy cycle and the 131 second connect timeout). In the lab, the load generator's last five lines, starting 10 seconds after we began applying the fix at 16:09:38:

kubectl -n shop logs deploy/loadgen --tail=5

2026-10-29T16:09:48Z GET /api/items -> 200
2026-10-29T16:09:51Z GET /api/items -> 200
2026-10-29T16:09:54Z GET /api/items -> 200
2026-10-29T16:09:57Z GET /api/items -> 200
2026-10-29T16:10:00Z GET /api/items -> 200

Then confirm there is nothing left hanging inside the upstream: netstat -tn in a new api pod should show no pile of SYN_SENT or CLOSE_WAIT. The fixed pods still logged stock lookup failed: <urlopen error timed out> on every request, which is the expected degraded state, not a pass for the dependency.

How do you prevent it?

Alert on the timeout before it fires, and count 499s with 504s. With ingress-nginx's Prometheus metrics (nginx_ingress_controller_requests carries status, namespace, ingress and service labels in v1.15.1, source):

sum by (namespace, ingress) (rate(nginx_ingress_controller_requests{status=~"504|499"}[5m]))
  /
sum by (namespace, ingress) (rate(nginx_ingress_controller_requests[5m]))
  > 0.01

We did not run this query against a live Prometheus in the lab; it is built from the metric and label names in the controller source. If the controller runs with --report-status-classes, the label holds 4xx and 5xx instead of codes and this regex matches nothing. For latency, the histograms (nginx_ingress_controller_header_duration_seconds and the others) use Prometheus' default buckets, whose largest finite bucket is 10 seconds, so a "p99 near the 60 second timeout" alert cannot be computed until you add buckets with the controller's --time-buckets flag. On plain nginx, add $request_time, $upstream_connect_time and $upstream_header_time to the log format, as in our table; without them you cannot tell the three causes apart after the fact.

Guardrail: every timeout shorter than the one above it. The client's timeout should be longer than the proxy's, the proxy's longer than the upstream's own request budget, and every outbound call inside the upstream should have a timeout shorter than that budget. The lab failure is the inversion: the only finite timeout on the path was nginx's 60 seconds, so nginx was the one to give up.

Test. In CI, run the service with each dependency pointed at a blackholed address (192.0.2.0/24 drops packets on our network; check that yours drops rather than rejects it) and assert that a request still returns within your budget. We have not built that test; the timed requests above are its manual version, and they separate the lab's broken release (no answer in 60 seconds) from the fixed one (200 in one second).

Common mistakes

• Raising proxy_read_timeout as the fix. Shown above: the 504 became a 2 minute 13 second 200 with missing data. It is also listed as a wrong answer in the scenario's grading.
• Looking for the 504 in the application's logs. The application never sends it. Ours logged 200 and then 503 for requests nginx had already answered with 504 or logged as 499.
• Believing the upstream's error message. The lab api blamed redis for the broken pipe to nginx. redis was fine the whole time.
• Counting only 504s. Most failed requests in the lab were 499s, because the load generator's 3 second timeout is shorter than nginx's 60.
• Restarting pods. New pods run the same code and hang on the same call.
• Reading every 504 as a slow backend. while connecting to upstream against a ClusterIP can be a port the Service does not publish, with a perfectly healthy backend behind it. Read the while ... clause first.
• Ignoring retries. On ingress-nginx, a 504 at three times the timeout means each slow request ran three times on the backend, which can be what is making it slow.

What we saw in the lab

Scenario 21 is new in the Aurora lab as of 2026-10-29. ./lab.sh selftest 21-api-slow-upstream-timeout breaks the healthy shop app with break.sh and passes only when a request with a 90 second client timeout gets HTTP/1.1 504, a frontend pod has logged upstream timed out (110: Operation timed out) while reading response header from upstream, and the api Service still has endpoints. On our cluster the first check failed and the second passed (broken 21-api-slow-upstream-timeout after 5s, selftest: 1 passed, 0 failed, 3 minutes 34 seconds in all; the "5s" counts only the pause between checks, each check takes about a minute). The first check's request got a 200 with the old response body: an api pod from before the change was still answering just after the rollout. The output and the captured cluster state are in the lab folder.

What surprised us building it:

• The load generator never saw a 504. The earlier traffic scenarios (19 and 20, for example) show up as -> 504 or -> 502 in its log. Here it printed timeout and nginx logged 499, so the scenario's check sends its own patient request.
• The upstream's logs lie in three ways at once (a 200, then a 503, then a redis error, all for one request nginx had already given up on), and only after more than two minutes.
• ingress-nginx tripled the wait by default on a two-pod backend, and sent the same request to one pod twice.
• The error string differs by image. The glibc wording, (110: Connection timed out), is the one usually quoted; the Alpine nginx image our lab uses prints (110: Operation timed out).

The Aurora investigation for this scenario has not been run yet. When it is, the trace and the grade will go into the lab folder and onto this page, including any wrong turns. The obvious traps are the redis error in the api log and raising the proxy timeout.

How Aurora handles this

Aurora receives the alert and investigates with read-only kubectl under its default observability_only command policy, choosing the commands itself. For a 504 it would need the same evidence as this page: the proxy's error and access logs, pod and endpoint state, the rollout history and the code or config change, and what the upstream is waiting on. One limit we already know: that policy denies the whole rollout verb, read-only subcommands included, so on the ConfigMap drift scenario both of its runs got a tool error on rollout and compared the live pod template with a running pod of the old revision instead. It records every tool call and its output, cites them in the report, and lists suggested fixes separately; mutating commands are blocked by default, so it proposes the change rather than applying it. Its results on the other lab scenarios are on the benchmark.

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Book a demo: cal.com/arvo-ai/demo

Related: 502 Bad Gateway in nginx and Kubernetes, Service has no endpoints, Service targetPort mismatch, ConfigMap change applied to only some pods, NetworkPolicy blocking traffic, liveness probe failed, paste your kubectl output into the diagnoser, and the root cause analysis guide for SREs.

Sources and verification

• Lab scenario: scenarios/21-api-slow-upstream-timeout (break.sh, check.sh, scenario.yaml). Every Kubernetes output block is from runs/search-2026-10-29/21-api-slow-upstream-timeout/transcript.txt (scenario, fix, timeout raise), transcript-ingress-nginx.txt and transcript-probe.txt, recorded 2026-10-29 between 16:05 and 16:14 UTC on kind cluster r504, plus the selftest output and cluster state in selftest.txt and selftest-evidence/ (16:13 to 16:17 UTC). Versions in versions.txt.
• Output blocks are copied from those files; where a block was cut, it says trimmed. time output is shown without its user and sys lines, and kubectl exec output without the trailing command terminated with exit code 1.
• Docker reproductions: docker-nginx-transcript.txt with the test upstream and nginx configs in docker-repro/, recorded 2026-10-29 between 15:56 and 16:10 UTC, nginx:1.27-alpine and nginx:1.27, both nginx/1.27.5.
• RFC 9110, 15.6.5 504 Gateway Timeout; RFC 5737 for 192.0.2.0/24.
• nginx: ngx_http_proxy_module (proxy_read_timeout, proxy_connect_timeout, proxy_next_upstream), ngx_http_upstream_module (max_fails, fail_timeout), and NGX_HTTP_CLIENT_CLOSED_REQUEST 499 in ngx_http_request.h.
• ingress-nginx: annotations, ConfigMap defaults, metrics source at v1.15.1, retirement announcement.
• Linux tcp_syn_retries: ip-sysctl documentation. Python urlopen timeout: urllib.request. Probe timeoutSeconds: Kubernetes probes. All read 2026-10-29.