Why a cloud range list is refused at load, not at verify
An operator's published IP range file is evidence because it lists the addresses that operator's crawlers send from, and nothing else. A general cloud provider list (AWS's ip-ranges.json, Google's cloud.json, Azure's service tags) lists every address a customer can rent, so a match there tells you only that the request came from a datacenter. Let both kinds of file become a range snapshot and every instance on a public cloud verifies as whatever crawler it claims to be, with the verdict carrying the same word, range, that a real verification does.
That is the problem the identity package in crawlwarrant has to solve before it does anything else. A vendor's robots.txt group governs a request only when evidence ties the request to that vendor; a request that merely names a product token is a claimant, and the * group governs it. The whole design rests on evidence actually being evidence. One misconfigured range file quietly converts the strict path into the permissive one.
Two files with identical shape and opposite meaning#
Both are JSON documents full of CIDR prefixes. Fetch them, parse them, and you hold the same Go value either way:
type Snapshot struct {
Operator string
Tokens []string
URL string // URL the file was requested from
FinalURL string // URL that answered, set when it differs
CreationTime time.Time // creationTime the operator put in the file
FetchedAt time.Time // when we read it
MaxAge time.Duration // how long it may be used, DefaultMaxAge when zero
Prefixes []netip.Prefix // ranges the file listed
}Nothing in Prefixes tells you which claim the publisher was making. A crawler operator publishes a range file to say these are the addresses our crawler sends from. A cloud provider publishes one to say these are the addresses we rent out. The first narrows the world to one sender. The second narrows it to roughly everyone who can run a container.
The only place that distinction survives is the URL the file came from. So the URL is the thing you have to check, and the moment to check it is when the snapshot is built.
Refusing at the boundary#
Load is the one constructor that turns a fetched file into a usable snapshot. It is also where the refusal lives:
// Load records a fetched range file as a snapshot. A general cloud provider
// list is refused here, at load, rather than left to verify requests it cannot
// speak for, and so is one a redirect led to.
func Load(s Snapshot) (Snapshot, error) {
if s.URL == "" {
return Snapshot{}, errors.New("identity: a range snapshot needs the URL it was fetched from")
}
for _, u := range []string{s.URL, s.FinalURL} {
if u == "" {
continue
}
if c, ok := cloudRangeList(u); ok {
return Snapshot{}, fmt.Errorf("identity: %s: %s: %w", u, c.note, ErrCloudRangeList)
}
}
if s.MaxAge == 0 {
s.MaxAge = DefaultMaxAge
}
return s, nil
}The alternative, letting the snapshot exist and adding a reason at verify time, looks more informative and is worse in practice. A bad range file is a configuration mistake, not a property of any request. Catch it at load and it surfaces once, at startup or at refresh, as an error with the offending URL in it, and somebody fixes the config. Catch it at verify and it surfaces as a line in an evidence chain, once per request, forever, and the chain now has to carry a kind of doubt that has nothing to do with the client being judged.
There is a second reason, which I learned from years of payment reconciliation rather than from crawler work: anything that can produce a false positive in a decision path will eventually be read by code that only checks the boolean. EvidenceKind.Verifies() returns true for EvidenceSigned, EvidenceRange and EvidenceFCrDNS. Some caller, somewhere, will branch on that and never look at the reasons. The reasons are for auditing. The refusal has to be structural.
Note also the missing-URL case, the same principle in a smaller form: a snapshot nobody can re-fetch cites nothing, so it is not a snapshot.
A redirect is part of the fetch#
FinalURL is set when a different URL answered for the published one. That field is not cosmetic. If a vendor's range URL 301s to a cloud provider's list, the bytes you ended up with are the cloud provider's bytes, and having started somewhere respectable changes nothing about what they prove. So both URLs go through the same check, and the test says why:
func TestLoadRefusesACloudListARedirectLedTo(t *testing.T) {
_, err := Load(Snapshot{
Operator: "Example",
URL: "https://crawler.example.com/ranges.json",
FinalURL: "https://ip-ranges.amazonaws.com/ip-ranges.json",
FetchedAt: fetched,
})
if !errors.Is(err, ErrCloudRangeList) {
t.Fatalf("error = %v, want %v: where the fetch ended decides what the file proves", err, ErrCloudRangeList)
}
}Where the fetch ended decides what the file proves. That sentence is the entire redirect policy.
Matching the list, and the limits of doing it this way#
The match itself is deliberately dull: a small table, compared by lowercased host with either an exact path or a path fragment:
var cloudLists = []cloudList{
{host: "ip-ranges.amazonaws.com", path: "/ip-ranges.json", note: "AWS lists every address its customers can rent"},
{host: "www.gstatic.com", path: "/ipranges/cloud.json", note: "Google Cloud lists the addresses of customer workloads"},
{host: "www.gstatic.com", path: "/ipranges/goog.json", note: "this file lists Google's whole address space, not its crawlers"},
{host: "download.microsoft.com", fragment: "servicetags", note: "Azure service tags cover every tenant in a region"},
}
func cloudRangeList(rawURL string) (cloudList, bool) {
u, err := url.Parse(rawURL)
if err != nil {
return cloudList{}, false
}
host, path := strings.ToLower(u.Hostname()), strings.ToLower(u.Path)
for _, c := range cloudLists {
if host != c.host {
continue
}
if c.path != "" && path == c.path {
return c, true
}
if c.fragment != "" && strings.Contains(path, c.fragment) {
return c, true
}
}
return cloudList{}, false
}Two things are worth saying plainly about this. The fragment case exists because Azure's download path carries a dated filename, so an exact path would stop matching on the next publication; the test fixture is a path containing ServiceTags_Public_20261019.json. And the goog.json entry is there because it is the trap most likely to be set by accident: Google publishes it, it sits next to a file that is genuinely about Google Cloud, and it lists Google's whole address space rather than its crawlers.
The honest limitation: this is a denylist. It names four files. It will not catch a regional mirror, a smaller provider, or a vendor file that has quietly become a reverse proxy for one. A denylist of known-bad sources cannot be complete, and the note on each row is as much documentation of why this class is wrong as it is an error message.
What the snapshot still refuses after it loads#
Passing Load is not the end of the scepticism. A zero MaxAge becomes DefaultMaxAge, 24 hours, and Verify builds its evidence in a fixed order, with the staleness check placed before any prefix is read:
if age := now.Sub(s.FetchedAt); age > s.maxAge() {
e.Reasons = append(e.Reasons, Reason{ReasonSnapshotStale, ...})
return e
}
if !s.contains(ip) {
e.Reasons = append(e.Reasons, Reason{ReasonOutOfRange, ...})
return e
}
e.Kind = EvidenceRangeThe early return is the point. Past its age the snapshot verifies nothing at all: Kind stays EvidenceAnonymous, Verifies() is false, and the message names the file, when it was fetched, how many hours ago that was, and how many hours it may be used. An expired file is not weaker evidence to be weighed. It is not evidence.
Two more reasons ride along without blocking anything. A Redirected() snapshot adds ReasonSnapshotRedirected even when the final URL was fine, because the file is re-verified at its published URL before it is trusted again. A zero CreationTime adds ReasonNoCreationTime, since without it you know when you read the file but not how old the operator's own data was when they published it.
And when len(s.Tokens) > 1, a successful match still adds ReasonOperatorOnly. One file shared across several products proves the operator sent the request, not which of its products did. That matters directly, because the next step applies a robots.txt group chosen by product token, and the address did not establish the token.
What I would do differently#
I would make the allowlist the primary mechanism and keep the denylist as the second line. The purpose package already stores, per product token, the RangeFileURL its operator publishes, read from a cited SourceURL on a recorded VerifiedOn date. A Load that took the token and required the snapshot's URL to be the published one would refuse every wrong file, not just the four I thought of, including the regional mirror and the proxied vendor path that the current table misses.
The denylist would still earn its place, because an allowlist fails closed in a way that is annoying to operate: a vendor changes a URL, the row goes stale, and verification stops. When someone reaches for a manual override in that moment, the four note strings are probably what stop them from pasting in ip-ranges.json because it was the only range file they could find.
Close#
A verification is only as good as the claim the source file was making. The cheapest way to keep that straight is to decide what a file can speak for once, where it enters the system, and to let everything downstream assume the question was already settled.