Why a Markdown scanner should blank code spans, not delete them
A link into a repository is worth reading only if it names a commit that cannot move underneath it. Checking that is the easy half: read the file, find the links, print a file:line for every ref that is not forty lowercase hex characters. The hard half starts when the same tool has to turn a branch link into a pinned one. Now it needs to write forty characters into a byte range of the file it just read, and every shortcut the reader took with positions is a bug waiting in the writer.
That pressure is what shaped srcproof, a small Go tool I wrote to keep repository links in prose honest. Two subcommands sit on one extractor. check reports links whose ref is not a full commit. pin resolves the branch and tag links of an article against a local clone and, with --write, rewrites them in place and records what it resolved in srcproof.lock. Both commands read the same Markdown through the same code. Only one of them edits, so the shared extractor has to serve the stricter consumer.
Masking preserves length; deleting does not#
A link inside a code span is not a link. `https://github.com/owner/name/blob/main/go.mod` is text you are being shown, not a reference you are expected to follow, and a tool that refuses branch links has to keep the right to print the form it refuses. So code spans must not be scanned.
The obvious way to not scan them is to cut them out of the line before looking for URLs. For a tool that only reports, that works perfectly. It also quietly destroys every column after the span: remove thirty bytes from the middle of a line and everything to the right of it now sits thirty bytes earlier than it does in the file. A rewriter that trusts those offsets writes into the wrong place.
So the span is overwritten instead of removed, with exactly as many spaces as it occupied:
// stripCodeSpans blanks out code spans without moving anything after them, so
// the column of a link is the column it has in the line as written.
func stripCodeSpans(line string) string {
var out strings.Builder
i := 0
for i < len(line) {
if line[i] != '`' {
out.WriteByte(line[i])
i++
continue
}
open := run(line, i)
close := -1
for j := i + open; j < len(line); {
if line[j] != '`' {
j++
continue
}
if length := run(line, j); length == open {
close = j
break
} else {
j += length
}
}
if close < 0 {
out.WriteString(line[i : i+open])
i += open
continue
}
out.WriteString(strings.Repeat(" ", close+open-i))
i = close + open
}
return out.String()
}
func run(line string, start int) int {
length := 0
for start+length < len(line) && line[start+length] == line[start] {
length++
}
return length
}The opening run is measured, and the span closes only on a later run of exactly the same length. Three backticks do not close a single-backtick span, which is why the inner loop advances by length rather than by one when it finds a run of the wrong size. The replacement is close+open-i spaces, which is precisely the width of the span from its first backtick to its last. The output string has the same length as the input, byte for byte, and every offset to the right of the span is the offset it has in the file.
The unclosed case is the one I like most. A run of backticks with no matching partner is written back unchanged, because that is both what CommonMark says it is (literal backticks, not a span) and the only thing that keeps the length intact. The two requirements happen to agree, and when correctness and layout agree, you take it.
Fenced blocks are skipped whole, and that costs nothing#
Fences are a different problem with an easier answer. A fenced block is a run of entire lines, so the extractor does not need to preserve anything inside it: it produces no links from those lines at all, and a line that yields no link has no column to protect.
func fenceMarker(line string) (byte, int, bool) {
i := 0
for i < 3 && i < len(line) && line[i] == ' ' {
i++
}
if i >= len(line) {
return 0, 0, false
}
char := line[i]
if char != '`' && char != '~' {
return 0, 0, false
}
length := run(line, i)
if length < 3 {
return 0, 0, false
}
return char, length, true
}
func closesFence(line string, char byte, length int) bool {
start := strings.IndexByte(line, char)
return strings.TrimSpace(line[start+length:]) == ""
}A fence opens on a line that starts, after at most three spaces, with three or more backticks or tildes, and closes only on a run of the same character at least as long with nothing but whitespace after it. Everything in between is skipped. Notice that this fenceMarker returns the character and the run length and throws the indent away. It does not need the indent, because it is discarding the lines. The quoting side of the tool has its own fenceMarker that also returns the indent, because it keeps the body lines and has to strip that indentation off them. Same syntax, two readers, two signatures. Trying to share one was worse than writing both.
The practical payoff is that srcproof's own README passes srcproof's own check while containing a table of the refs it refuses (main, v1.4.0, refs/heads/main) and a worked example of a link it would reject. Documentation that cannot show the bad form is documentation that cannot teach.
From a column to a splice point#
With the masked line in hand, finding URLs is unremarkable except for one detail:
func findURLs(line string) []candidate {
locations := urlPattern.FindAllStringIndex(line, -1)
candidates := make([]candidate, 0, len(locations))
for _, at := range locations {
url := strings.TrimRight(line[at[0]:at[1]], ".,;:!?")
if url == "" {
continue
}
candidates = append(candidates, candidate{url: url, col: at[0]})
}
return candidates
}A URL at the end of a sentence swallows the full stop, so the match is trimmed, but trimmed from the right only, and the recorded column is at[0], the start of the match. Sentence punctuation can shorten a URL. It cannot move one. That is the kind of invariant worth being deliberate about, because the alternative spelling (trim both ends, record the trimmed start) looks identical in every test that does not involve a leading character.
That column lands in the Col field of the link. A second offset lands next to it: when the URL is parsed, the position of the ref within the URL is computed by walking the segments that precede it.
func segmentOffset(start int, segments []string, index int) int {
for _, segment := range segments[:index] {
start += len(segment) + 1
}
return start
}For a github.com link the ref is the fourth segment, after owner, name and the blob, tree or commit keyword. For raw.githubusercontent.com it is the third segment, or three segments joined when that segment is refs, so that refs/heads/main is read as one ref rather than as refs. The parse drops the fragment and the query to decide these things, but the URL is kept exactly as it was written, and the offset is into that original string. Column plus ref offset is the byte range of the ref in the file.
The rewrite distrusts the position it was given#
An offset computed earlier is a claim about a string that may since have changed, so both ends check it before they use it.
Pinning a link refuses outright unless the ref offset is positive and the URL really does hold the ref at that offset. Only then does it splice the commit in and leave the rest of the URL, fragment included, as written. And when the rewrites are applied to the file, the same suspicion shows up again:
// apply rewrites a line from its right edge inwards, so the column of a link
// is still the column it was read at after an earlier link on the same line
// has grown into a 40-character commit.
func apply(source string, rewrites []Rewrite) string {
if len(rewrites) == 0 {
return source
}
grouped := make(map[int][]Rewrite)
for _, rewrite := range rewrites {
grouped[rewrite.Line] = append(grouped[rewrite.Line], rewrite)
}
lines := strings.Split(source, "\n")
for number, group := range grouped {
line := lines[number-1]
for i := len(group) - 1; i >= 0; i-- {
rewrite := group[i]
end := rewrite.Col + len(rewrite.From)
if end > len(line) || line[rewrite.Col:end] != rewrite.From {
continue
}
line = line[:rewrite.Col] + rewrite.To + line[end:]
}
lines[number-1] = line
}
return strings.Join(lines, "\n")
}Two things are happening here. The loop runs backwards through the rewrites of a line, so replacing a nine-character branch name with a forty-character commit cannot invalidate the column of a link further right on the same line: the right edge is edited first, and the left-hand columns are still the columns they were read at. The bytes at the recorded column are also compared against what the rewrite expected to find there. If they do not match, or the range runs past the end of the line, the rewrite is skipped and the line stays as it is. A stale position produces a link that was not pinned, reported as unresolved. It does not produce forty hex characters spliced into the middle of a sentence.
It is also why no column ever reaches srcproof.lock. A link directive records the article, the repository, the commit and the ref it was resolved from, and nothing about where in the file that happened. A byte offset is valid against one exact revision of one file, so persisting it would mean persisting something that the next edit silently falsifies.
What I would do differently#
The length-preservation contract lives in a comment above stripCodeSpans, which is probably the weakest place to put an invariant that three other packages depend on. It belongs in a test: for every line, the masked line and the original have the same length. That property is cheap to check, it fails loudly the first time someone replaces a span with the empty string, and it says the thing the comment only hints at.
The scanner is also line-by-line, and CommonMark lets a code span cross a newline. A URL inside such a span is read as a link today, which is wrong. The fix is not a different strategy but the same one at document scope: mask the span across the lines it covers, keeping the newlines, and offsets still hold. Indented code blocks have the same shape of gap and the same shape of answer.
The general lesson I keep relearning is that a position is not a diagnostic. The moment a tool grows from reporting to editing, every offset it computed becomes an address somebody will write to, and the transformations it performed on the way to finding that offset become part of its correctness. Masking rather than deleting is a one-line decision that keeps the two halves of the tool talking about the same file...