What it means
110 is the Linux error ETIMEDOUT. nginx writes this line when the backend accepted the connection but did not send a response header within proxy_read_timeout (fastcgi_read_timeout or uwsgi_read_timeout for those protocols). The client receives 504 Gateway Time-out.
[error] 1234#1234: *91 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.0.113.7, server: example.com, request: "POST /exports HTTP/1.1", upstream: "http://10.0.1.12:8080/exports"The same error ending in while connecting to upstream means the TCP connection never opened (see the FAQ).
Common causes
1. One slow route. A report, an export or an unindexed query takes more than 60 seconds. Time it straight against the backend, bypassing nginx:
curl -s -o /dev/null -w "%{http_code} %{time_starttransfer}s\n" http://10.0.1.12:8080/exports2. A saturated backend. Every worker is busy, so new requests wait in a queue. PHP-FPM logs it explicitly:
sudo grep "max_children" /var/log/php*-fpm.log3. A stuck dependency. A database lock or an outbound API call with no timeout. On PostgreSQL:
SELECT pid, now() - query_start AS runtime, state, query
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY runtime DESC;4. Network drops (connect variant). Test the port from the nginx host: nc -vz -w 5 10.0.1.12 8080.
How to fix
-
Read the
requestfield to find the route, then time it directly against the backend. -
Fix the cause: add the index, set timeouts on outbound calls, add workers, or move long work to a background job that the client polls.
-
If the route is legitimately long, raise the timeout for that location only:
location /exports/ { proxy_pass http://app; proxy_read_timeout 300s; } -
Raise the layers around it too: the application server's timeout and any load balancer in front.
-
Test and reload:
sudo nginx -t && sudo systemctl reload nginx.
Is it just you?
A 504 for you can be a 200 elsewhere if only one backend or region is slow. Check the URL from several regions with Is my site down?.
Get alerted next time
- An HTTP monitor fails when the route answers 504 instead of the status code you expect, and a check that gets no answer at all fails too.
- Each failure is rechecked from your other regions before an outage opens, so a single slow request does not page anyone.
- A TCP port monitor on a dependency's port, such as the database, tells you whether it still accepts connections.
FAQ
What is the default proxy_read_timeout in nginx? ▼
60 seconds. It is measured between two successive reads, not over the whole response: a backend that sends something at least every 60 seconds never triggers it. proxy_connect_timeout and proxy_send_timeout also default to 60 seconds.
What does while connecting to upstream mean instead? ▼
The TCP connection itself never completed within proxy_connect_timeout. That points at the network rather than the application: a firewall or security group silently dropping packets, a wrong IP, or a host that is down. A refused connection gives error 111 instead of 110.
Why do I still get a 504 after raising proxy_read_timeout? ▼
Another layer gave up first. A load balancer in front of nginx (an AWS ALB has a 60 second idle timeout by default), a CDN, or the application server's own timeout can each end the request before nginx's new limit is reached.