Skip to content
4104xx · Client errorThe client seems to have erred. RFC 9110 §15.5

HTTP 410 Gone: what it means and how to fix it

Meaning

A 410 Gone means the server removed the URL on purpose and expects it to stay gone. What that means for a scraper or proxy, and what to change.

HTTP 410 means the server removed the URL on purpose and expects it to stay removed. The reason phrase is “Gone”, and the code is defined in RFC 9110, section 15.5.11. A 404 only says nothing is at that address. A 410 adds that something used to be there, that taking it away was deliberate, and that the owner would like links to it removed. When the server cannot tell whether a removal is permanent, the RFC says it ought to send 404 instead.

That makes 410 one of the few 4xx answers that tells a client exactly what to do: stop asking for that URL.

Reproduce a 410 locally

Python’s standard library is enough. This server answers 410 for one product page until a file named restored appears, and it marks the 410 as cacheable for five minutes with max-age=300.

# gone.py: answers 410 for a removed URL until a file named "restored" exists
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/products/blue-kettle" and not os.path.exists("restored"):
            body = b"This product was discontinued.\n"
            self.send_response(410)
            self.send_header("Content-Type", "text/plain")
            self.send_header("Cache-Control", "max-age=300")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        body = b"Blue kettle, in stock.\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("127.0.0.1", 8410), Handler).serve_forever()

Run it with python3 gone.py and request the page. This is the real output from Python 3.15.0 and curl 8.7.1 on macOS, with the Date header filtered out:

$ curl -si http://127.0.0.1:8410/products/blue-kettle | grep -v "^Date:"
HTTP/1.0 410 Gone
Server: BaseHTTP Python/3.15.0
Content-Type: text/plain
Cache-Control: max-age=300
Content-Length: 31

This product was discontinued.

Over HTTP/2 you will not see the word “Gone” at all. HTTP/2 carries only a numeric :status pseudo-header (RFC 9113, section 8.3.2), and RFC 9110 calls reason phrases recommendations that can be left out. This site’s own comments feed is a live example. Comments are closed, so the feed answers 410, and it forbids caching:

$ curl -s -o /dev/null -D - https://proxy.wiki/comments/feed/ | grep -iE "^(HTTP|cache-control)"
HTTP/2 410
cache-control: no-cache, must-revalidate, max-age=0, no-store, private

When a proxy keeps answering 410 after the page is back

RFC 9110 section 15.1 lists 410 among the heuristically cacheable codes, next to 200, 301 and 404. A cache may store one and reuse it without being told to, and an explicit max-age removes any doubt. The result is a 410 that outlives the removal. Here is that sequence through a local Squid 6.13 with memory caching on, sitting in front of the server above:

$ curl -s -o /dev/null -x 127.0.0.1:3128 -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
410
$ touch restored    # the page comes back at the origin
$ curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
200
$ curl -s -o /dev/null -x 127.0.0.1:3128 -w "%{http_code} age=%header{age}\n" http://127.0.0.1:8410/products/blue-kettle
410 age=2
$ curl -s -o /dev/null -x 127.0.0.1:3128 -H "Cache-Control: no-cache" -w "%{http_code}\n" http://127.0.0.1:8410/products/blue-kettle
200

Squid’s access log, with the timestamp column cut, records each step:

     0 127.0.0.1 TCP_MISS/410 293 GET http://127.0.0.1:8410/products/blue-kettle - HIER_DIRECT/127.0.0.1 text/plain
     0 127.0.0.1 TCP_MEM_HIT/410 302 GET http://127.0.0.1:8410/products/blue-kettle - HIER_NONE/- text/plain
     0 127.0.0.1 TCP_MISS/200 255 GET http://127.0.0.1:8410/products/blue-kettle - HIER_DIRECT/127.0.0.1 text/plain

The origin said 200 while the proxy still said 410. Corporate forward proxies, CDNs and scraping APIs with a response cache can all do this. If a 410 appears through one path only, look for an Age header and ask for revalidation before you believe it.

A 410 is about the URL, not about you. Rotating to a new exit will not change a genuine one. If a different proxy exit gets a 200 for the same URL, the 410 is a decision about that client or region, not the end of the page.

What a scraper should do with it

Treat a confirmed 410 as final. Take the URL out of the crawl queue, mark the stored record as deleted, and do not retry it. Scrapy’s defaults already agree: its RETRY_HTTP_CODES is [500, 502, 503, 504, 522, 524, 408, 429], so a 410 is never retried. Search engines read it the same way. Google’s crawler documentation says every 4xx except 429 is handled alike, and an indexed URL that returns one is dropped from the index. Repeat fetches of dead URLs also eat into crawl budget, on your side and the site’s.

Confirm it once before acting. Fetch the URL with no proxy, or through a second exit in another country, as shown in how to use a proxy with curl. A real removal returns 410 everywhere. Then check the Server header and the body to see who answered: the origin, a CDN in front of it, or a proxy in between.

What you see Likely cause What to change
410 from every exit and with no proxy The resource was removed on purpose Drop the URL from the queue, stop retrying, and fix whatever keeps feeding it to you (sitemap, listing page, old export)
410 through one proxy or cache only, often with an Age header A cached 410 that outlived the removal Send Cache-Control: no-cache, purge the entry, or bypass the cache for that host
410 on one exit, 200 on another The site uses 410 as a refusal for some clients Handle it as a block: compare the exits and headers, slow down, and do not mark the URL deleted
410 on a whole API path or version The endpoint was retired Read the provider’s migration notes and move to the current version
410 on your own site after a cleanup Pages marked gone while links still point at them Remove internal links and sitemap entries, and use a 301 instead where the content moved

For the codes that look similar but mean a block or a rate limit, see 403 vs 407 vs 429. For code that handles these responses in Python, see using proxies with requests, httpx and aiohttp.

Frequently asked questions

What is the difference between 404 and 410?
A 404 says nothing is at the URL and makes no promise about the future. A 410 says the resource existed, was removed deliberately, and is likely to stay removed. RFC 9110 tells servers to send 404 when they cannot tell whether a removal is permanent.
Is 410 better than 404 for SEO?
Google's crawler documentation says it treats every 4xx status except 429 the same way, so an indexed URL that returns either one is dropped from the index. Send 410 when you know the removal is permanent, and a 301 when the content moved.
Should a scraper retry a 410?
No. Confirm it once from a second exit or without a proxy, then remove the URL from the queue. Scrapy's default retry list does not include 410, and retrying a removed page only spends requests.
Can a proxy cause a 410?
A proxy reports its own failures with codes such as 407, 502 and 504. A caching proxy can, however, keep serving a stored 410 after the origin has restored the page, because 410 is heuristically cacheable. Send Cache-Control: no-cache to force a fresh answer.

Sources

  1. RFC 9110: HTTP Semantics, section 15.5.11, 410 Gone
  2. RFC 9110, section 15.1: Overview of Status Codes (heuristically cacheable codes, reason phrases)
  3. RFC 9113: HTTP/2, section 8.3.2, response pseudo-header fields
  4. Google Search Central: How HTTP status codes, and network and DNS errors affect Google Search
  5. Scrapy documentation: RetryMiddleware and RETRY_HTTP_CODES
  6. IANA: HTTP Status Code Registry

All terms, A to Z