Server responds with HTTP 5xx

Problem

The last crawl received an HTTP 5xx response (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout...). The request reached the server, but the server-side failed.

Why it matters

• Users see an error page; repeated 5xx responses cause search engines to slow the crawl and eventually drop affected URLs.
• 5xx usually signals a real defect: crashing app code, an exhausted database pool, a dead upstream, or a misconfigured proxy.

How to fix

1. Read the application/proxy error logs for the exact timestamp.
2. Map the code: 500 app exception - 502/504 upstream dead or timing out - 503 overload or maintenance mode left on.
3. Fix the defect and add monitoring/alerting on the 5xx rate.
4. If maintenance is intentional, serve 503 with a Retry-After header — crawlers back off politely.