One path, many spellings: normalizing a robots.txt match
A URL path has more than one legal spelling, and a robots.txt matcher that compares a rule to the path as written gets some of them wrong in both directions. /%7Euser/notes and /~user/notes name the same resource, so a rule written Disallow: /~user/ has to catch both of them. /a%2Fb and /a/b do not name the same resource, so nothing in the matcher may quietly read %2F as a separator. A table of special cases per escape is not the way out. You rewrite both sides into one form before any comparison happens, and then you live with the fact that this moves the number which decides precedence.
Two failure modes, opposite directions#
The first is a rule that slips. A site owner writes Disallow: /~user/, because that is what the URLs look like in a browser. A crawler sends GET /%7euser/notes: lowercase hex, encoded tilde, entirely within its rights, since a tilde is unreserved and a client may encode it anyway. Byte comparison says no rule applies. The path comes back allowed, and if you are auditing access logs to decide whether a verified crawler respected the policy, you have just recorded a compliant fetch of something the site refused.
The second is a rule that over-reaches. Decode everything before comparing and /a%2Fb becomes /a/b, which Disallow: /a/ now matches. But an encoded slash is one octet inside a single path segment, not a segment boundary. The rule was about a directory; the path was never in it.
Both failures come from one place: the two inputs arrive at the comparison in different alphabets. In the robots package of crawlwarrant (https://github.com/polycratia/crawlwarrant) the only comparison that happens is between two strings that have already been through the same function.
One form, applied to both sides#
RFC 9309, Section 2.2.2, asks for exactly one transformation before matching: escapes of unreserved characters are decoded, every other escape is uppercased. Reserved octets stay escaped, and that is what keeps %2F from turning into a slash. The matcher does this to the path once, and to each pattern as it considers it:
func (g Group) Match(path string) (rule Rule, ok bool) {
p := normalize(path)
if p == "/robots.txt" {
return Rule{}, false
}
best := -1
for _, r := range g.Rules {
pattern := normalize(r.Pattern)
if pattern == "" || !matches(pattern, p) {
continue
}
if n := len(pattern); n > best || (n == best && r.Kind == Allow && rule.Kind == Disallow) {
rule, best = r, n
}
}
return rule, best >= 0
}Three things in those lines are worth naming.
The path is normalized before the /robots.txt exemption is checked, so /%72obots.txt gets the exemption too. The exemption belongs to the resource, not to one spelling of it.
A pattern that normalizes to the empty string is skipped. That is how an empty Disallow: value matches nothing: without the skip it would reach the prefix test as an empty prefix and match every path on the site, which is the opposite of what the line means.
And best holds the length of the normalized pattern, not of the pattern in the file. That is the whole cost of this design, and I come back to it below.
normalize#
const hexDigits = "0123456789ABCDEF"
func normalize(s string) string {
var b strings.Builder
b.Grow(len(s))
for i := 0; i < len(s); {
c := s[i]
switch {
case c == '%':
if v, ok := unescape(s[i:]); ok {
if unreserved(v) {
b.WriteByte(v)
} else {
writeEscape(&b, v)
}
i += 3
continue
}
writeEscape(&b, '%')
i++
case c <= ' ' || c >= 0x7f:
writeEscape(&b, c)
i++
default:
b.WriteByte(c)
i++
}
}
return b.String()
}
func unreserved(c byte) bool {
switch {
case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9':
return true
}
return c == '-' || c == '.' || c == '_' || c == '~'
}
func writeEscape(b *strings.Builder, c byte) {
b.WriteByte('%')
b.WriteByte(hexDigits[c>>4])
b.WriteByte(hexDigits[c&0xf])
}unescape reads the two hex digits in either case, so %7e and %7E arrive at the same byte. From there the rule is short: if the byte is a letter, a digit, a hyphen, a period, an underscore or a tilde, it is written plainly, and anything else is re-emitted as a percent sign plus two uppercase digits. A percent sign that does not begin a valid escape is itself escaped, to %25. Any byte at or below the space character, or at or above 0x7f, gets the same treatment.
Two properties matter more than the details here.
It is idempotent. Run it twice and nothing moves: %7E becomes a tilde and stays a tilde, a stray percent sign becomes %25, and %25 decodes to a percent sign, which is not unreserved, so it is written straight back as %25. That matters because patterns and paths go through the same function, and because a caller may hand the matcher a path that some earlier layer has already canonicalized.
It leaves * and $ untouched. They are printable ASCII and not percent signs, so they fall through to the default branch unchanged. %2A does not: an asterisk is a reserved character, unreserved rejects it, and writeEscape puts the escape back. So an encoded asterisk stays a literal octet instead of becoming a wildcard, and an encoded dollar sign does not become an anchor.
Wildcards are read on the normalized text#
func matches(pattern, path string) bool {
anchored := strings.HasSuffix(pattern, "$")
if anchored {
pattern = pattern[:len(pattern)-1]
}
parts := strings.Split(pattern, "*")
if len(parts) == 1 {
if anchored {
return path == pattern
}
return strings.HasPrefix(path, pattern)
}
if !strings.HasPrefix(path, parts[0]) {
return false
}
pos := len(parts[0])
for _, part := range parts[1 : len(parts)-1] {
i := strings.Index(path[pos:], part)
if i < 0 {
return false
}
pos += i + len(part)
}
last := parts[len(parts)-1]
if anchored {
return len(path)-pos >= len(last) && strings.HasSuffix(path, last)
}
return strings.Contains(path[pos:], last)
}The pattern is split on the asterisk. The first part must be a prefix of the path, each middle part is found leftmost after the previous one, and the last part must occur in what remains, or, when the pattern ended in $, be a suffix of the path beginning past the bytes already consumed. That last condition is what the length check enforces: without it an anchored suffix could overlap bytes a middle part already matched, and a pattern would match a path shorter than itself.
Because the split happens after normalization, the alphabet the splitter sees is fixed. No spelling of a path introduces a wildcard, and there is no spelling of a pattern where a literal asterisk hides inside an escape and then springs open during matching.
The cost: precedence is measured on the normalized form#
RFC 9309 breaks ties by octet count: the longest matching rule wins, and an allow beats a disallow of equal length. The count is taken after normalization, which means it is not the count in the file. Consider:
User-agent: ExampleBot
Disallow: /%7Euser/
Allow: /~user/As written, the disallow is eight bytes and the allow is seven, so the disallow is longer and would win. Normalized, both patterns are the same seven bytes, and the tie goes to the allow. /~user/notes is allowed, and the rule that decided it sits on line 3.
Normalization moves lengths the other way too. /a b/ is five bytes in the file and seven once the space is escaped, so it can take precedence over a competing rule it would have lost to on raw bytes. A rule can gain or lose precedence purely because of how its author spelled it.
I do not think a better answer is available. Ranking by the bytes in the file makes the outcome depend on exactly the authoring choice that normalization exists to erase, and it puts the first failure mode back at the tie-break instead of at the match. The specification already made this trade when it defined the comparison on normalized text, and measuring length anywhere else would be inconsistent with the comparison that produced the candidates.
What it does cost is explainability. The verdict chain in the identity package quotes the rule as the file wrote it (the kind, the pattern and the line), which is the right thing to show a site owner reading an audit. But when a rule loses on normalized length, a line naming /%7Euser/ and line 2 does not carry the reason it lost...
What I would change#
Normalize patterns at parse time rather than inside Match. Today every call re-normalizes every rule of the group, and an audit replaying an access log calls it once per request. The reason it is not done yet: Rule.Pattern is public and reports what the file said, so the normalized form has to arrive as a second field rather than a rewrite of the first.
And once that field exists, put it in the explanation: report the pattern as written, then add the normalized form when it differs. A site owner who wrote two spellings of the same prefix should be able to read that off the output instead of reconstructing it.
One thing I would not change is the handling of query strings. The path handed to Match includes the query as the request carried it, and normalize treats those bytes like any other, so ? and & pass through as printable ASCII. Query strings have their own encoding conventions, and a matcher that started applying them would be deciding things RFC 9309 never asked it to decide.
Close#
The discipline here is narrow and worth keeping narrow: a matcher should have exactly one place where its two inputs become comparable, and nothing downstream of that place should know anything about spellings. Once normalize is the only door, %2F cannot become a slash by accident, because nothing past the door is looking at escapes at all.