HTTP 410 Gone: what it means and how to fix it
Meaning
A 410 Gone means the server removed the URL on purpose and expects it to stay gone. What that means for a scraper or proxy, and what to change.
HTTP 410 means the server removed the URL on purpose and expects it to stay removed. The reason phrase is “Gone”, and the code is defined in RFC 9110, section 15.5.11. A 404 only says nothing is at that address. A 410 adds that something used to be there, that taking it away was deliberate, and that the owner would like links to it removed. When the server cannot tell whether a removal is permanent, the RFC says it ought to send 404 instead.
That makes 410 one of the few 4xx answers that tells a client exactly what to do: stop asking for that URL.
Reproduce a 410 locally
Python’s standard library is enough. This server answers 410 for one product page until a file named restored appears, and it marks the 410 as cacheable for five minutes with max-age=300.
# gone.py: answers 410 for a removed URL until a file named "restored" exists
import os
from http.server import BaseHTTPRequestHandler, HTTPServer
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/products/blue-kettle" and not os.path.exists("restored"):
body = b"This product was discontinued.\n"
self.send_response(410)
self.send_header("Content-Type", "text/plain")
self.send_header("Cache-Control", "max-age=300")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
return
body = b"Blue kettle, in stock.\n"
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
HTTPServer(("127.0.0.1", 8410), Handler).serve_forever()
Run it with python3 gone.py and request the page. This is the real output from Python 3.15.0 and curl 8.7.1 on macOS, with the Date header filtered out:
$ curl -si http://127.0.0.1:8410/products/blue-kettle | grep -v "^Date:"
HTTP/1.0 410 Gone
Server: BaseHTTP Python/3.15.0
Content-Type: text/plain
Cache-Control: max-age=300
Content-Length: 31
This product was discontinued.
Over HTTP/2 you will not see the word “Gone” at all. HTTP/2 carries only a numeric :status pseudo-header (RFC 9113, section 8.3.2), and RFC 9110 calls reason phrases recommendations that can be left out. This site’s own comments feed is a live example. Comments are closed, so the feed answers 410, and it forbids caching:
$ curl -s -o /dev/null -D - https://proxy.wiki/comments/feed/ | grep -iE "^(HTTP|cache-control)"
HTTP/2 410
cache-control: no-cache, must-revalidate, max-age=0, no-store, private
When a proxy keeps answering 410 after the page is back
RFC 9110 section 15.1 lists 410 among the heuristically cacheable codes, next to 200, 301 and 404. A cache may store one and reuse it without being told to, and an explicit max-age removes any doubt. The result is a 410 that outlives the removal. Here is that sequence through a local Squid 6.13 with memory caching on, sitting in front of the server above:
$ curl -s -o /dev/null -x 127.0.0.1:3128 -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
410
$ touch restored # the page comes back at the origin
$ curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
200
$ curl -s -o /dev/null -x 127.0.0.1:3128 -w "%{http_code} age=%header{age}\n" http://127.0.0.1:8410/products/blue-kettle
410 age=2
$ curl -s -o /dev/null -x 127.0.0.1:3128 -H "Cache-Control: no-cache" -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
200
Squid’s access log, with the timestamp column cut, records each step:
0 127.0.0.1 TCP_MISS/410 293 GET http://127.0.0.1:8410/products/blue-kettle - HIER_DIRECT/127.0.0.1 text/plain
0 127.0.0.1 TCP_MEM_HIT/410 302 GET http://127.0.0.1:8410/products/blue-kettle - HIER_NONE/- text/plain
0 127.0.0.1 TCP_MISS/200 255 GET http://127.0.0.1:8410/products/blue-kettle - HIER_DIRECT/127.0.0.1 text/plain
The origin said 200 while the proxy still said 410. Corporate forward proxies, CDNs and scraping APIs with a response cache can all do this. If a 410 appears through one path only, look for an Age header and ask for revalidation before you believe it.
What a scraper should do with it
Treat a confirmed 410 as final. Take the URL out of the crawl queue, mark the stored record as deleted, and do not retry it. Scrapy’s defaults already agree: its RETRY_HTTP_CODES is [500, 502, 503, 504, 522, 524, 408, 429], so a 410 is never retried. Search engines read it the same way. Google’s crawler documentation says every 4xx except 429 is handled alike, and an indexed URL that returns one is dropped from the index. Repeat fetches of dead URLs also eat into crawl budget, on your side and the site’s.
Confirm it once before acting. Fetch the URL with no proxy, or through a second exit in another country, as shown in how to use a proxy with curl. A real removal returns 410 everywhere. Then check the Server header and the body to see who answered: the origin, a CDN in front of it, or a proxy in between.
| What you see | Likely cause | What to change |
|---|---|---|
| 410 from every exit and with no proxy | The resource was removed on purpose | Drop the URL from the queue, stop retrying, and fix whatever keeps feeding it to you (sitemap, listing page, old export) |
410 through one proxy or cache only, often with an Age header |
A cached 410 that outlived the removal | Send Cache-Control: no-cache, purge the entry, or bypass the cache for that host |
| 410 on one exit, 200 on another | The site uses 410 as a refusal for some clients | Handle it as a block: compare the exits and headers, slow down, and do not mark the URL deleted |
| 410 on a whole API path or version | The endpoint was retired | Read the provider’s migration notes and move to the current version |
| 410 on your own site after a cleanup | Pages marked gone while links still point at them | Remove internal links and sitemap entries, and use a 301 instead where the content moved |
For the codes that look similar but mean a block or a rate limit, see 403 vs 407 vs 429. For code that handles these responses in Python, see using proxies with requests, httpx and aiohttp.
Frequently asked questions
What is the difference between 404 and 410?
Is 410 better than 404 for SEO?
Should a scraper retry a 410?
Can a proxy cause a 410?
Sources
- RFC 9110: HTTP Semantics, section 15.5.11, 410 Gone
- RFC 9110, section 15.1: Overview of Status Codes (heuristically cacheable codes, reason phrases)
- RFC 9113: HTTP/2, section 8.3.2, response pseudo-header fields
- Google Search Central: How HTTP status codes, and network and DNS errors affect Google Search
- Scrapy documentation: RetryMiddleware and RETRY_HTTP_CODES
- IANA: HTTP Status Code Registry