Skip to content

Handle very long lines in git rev-list --objects output#158

Open
simonkundrik wants to merge 1 commit into
github:masterfrom
simonkundrik:fix/long-rev-list-lines
Open

Handle very long lines in git rev-list --objects output#158
simonkundrik wants to merge 1 commit into
github:masterfrom
simonkundrik:fix/long-rev-list-lines

Conversation

@simonkundrik

Copy link
Copy Markdown

Problem

git-sizer aborts with:

error: error scanning repository: copy-oids: bufio.Scanner: token too long

on repositories that contain an object with a very long path name (the reporter of #157 saw a git rev-list --objects line of ~12 MiB). The copy-oids stage in git/obj_iter.go reads git rev-list --objects output via pipe.LinewiseFunction, whose underlying bufio.Scanner rejects any line longer than the default 64 KiB (bufio.MaxScanTokenSize).

Fix

Replace pipe.LinewiseFunction with pipe.ScannerFunction, supplying a bufio.Scanner that uses the same LF splitting (pipe.ScanLFTerminatedLines) and the same per-line logic, but whose buffer starts at the usual 64 KiB and is allowed to grow up to a new maxRevListLineLength (64 MiB). Because bufio.Scanner only grows its buffer as needed, normal usage still uses minimal memory, while pathologically long lines can now be processed — matching the "normally use fewer resources but deal with larger lines if need be" behaviour suggested in the issue.

The copy-oids per-line logic itself is unchanged.

Tests

Added TestObjectIterLongPath (git/obj_iter_test.go), which builds a tree whose single entry has a 100 KiB file name (created via git hash-object --literally to bypass git's path-length fsck check), then walks it with ObjectIter.

  • Before the fix: the test fails with copy-oids: bufio.Scanner: token too long.
  • After the fix: the test passes and the walk yields the commit, tree, and blob.

go test ./..., go vet ./..., and gofmt are all clean.

Note on the 64 MiB cap

bufio.Scanner requires an upper bound on token size, so this raises the limit rather than removing it entirely. 64 MiB is far beyond any realistic path name (and ~5× the largest line reported in #157) while still bounding per-line memory use. If you'd prefer a fully streaming approach instead — reading just the leading OID and discarding the rest of each line, so memory stays O(hash length) regardless of path length — I'm happy to take the PR in that direction.

Closes #157

The `copy-oids` stage used `pipe.LinewiseFunction`, whose scanner
rejects any line longer than 64 KiB with "bufio.Scanner: token too
long". A single line of `git rev-list --objects` output can exceed
that limit when an object has a very long path name, which aborts the
whole object walk.

Use `pipe.ScannerFunction` with a scanner that uses the same LF
line-splitting but whose buffer starts at the usual 64 KiB and may
grow up to 64 MiB, so normal usage stays cheap while pathologically
long lines can still be processed.

Closes github#157
@simonkundrik simonkundrik requested a review from a team as a code owner July 6, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

git-sizer can fail on some very long lines

1 participant