DNS audit: resolver consensus

2026-06-21 · updated 2026-09-02

The crawler visits millions of hosts; many disappear — domains expire, delegations vanish, hosts die. A naive rule ("first DNS error = dead domain") is dangerous: a transient failure from one resolver would mark thousands of live domains dead and drop them from the queue for good.

The consensus mechanism

The audit (cli audit, infra/audit.py) queries each host against three resolvers picked by pick_resolvers (infra/dns.py) from a pool of eight independent operators (Cloudflare, Google, Quad9, OpenDNS) — with a country-local resolver prepended for known ccTLDs.

A domain is declared dead only when a majority return NXDOMAIN and none answer positively. A single resolver that still sees the domain is enough to keep it. That separates real expiry from infrastructure flakes.

Four statuses

status	meaning	crawler action
`DNS_ALIVE`	has an A or AAAA record	index / re-crawl
`DNS_NXDOMAIN`	authoritatively absent (consensus)	store `dns.status`, do not set `blocked`
`DNS_PARKED`	zone exists (has NS) but no A/AAAA	observe, do not crawl
`DNS_TRANSIENT`	timeout / SERVFAIL / disagreement	retry later, never drop

What we store

{
  "dns": {
    "status": "DNS_ALIVE",
    "a": ["104.21.79.48", "172.67.142.4"],
    "ns": ["archer.ns.cloudflare.com", "pearl.ns.cloudflare.com"],
    "provider": "cloudflare",
    "checked_at": "2026-06-21T13:08:23Z"
  }
}

The provider field (derived from NS records) lets us measure infrastructure centralization — how much of the web sits on Cloudflare / AWS / Google versus self-hosted nameservers.

Circuit breaker

If too many recent checks come back DNS_TRANSIENT, a network/resolver outage is in progress — the audit stops before mislabelling live hosts as dead. Protecting data beats finishing the run.

Related: TLS audit & ranking.