Handle very long lines in git rev-list --objects output#158
Open
simonkundrik wants to merge 1 commit into
Open
Handle very long lines in git rev-list --objects output#158simonkundrik wants to merge 1 commit into
git rev-list --objects output#158simonkundrik wants to merge 1 commit into
Conversation
The `copy-oids` stage used `pipe.LinewiseFunction`, whose scanner rejects any line longer than 64 KiB with "bufio.Scanner: token too long". A single line of `git rev-list --objects` output can exceed that limit when an object has a very long path name, which aborts the whole object walk. Use `pipe.ScannerFunction` with a scanner that uses the same LF line-splitting but whose buffer starts at the usual 64 KiB and may grow up to 64 MiB, so normal usage stays cheap while pathologically long lines can still be processed. Closes github#157
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
git-sizeraborts with:on repositories that contain an object with a very long path name (the reporter of #157 saw a
git rev-list --objectsline of ~12 MiB). Thecopy-oidsstage ingit/obj_iter.goreadsgit rev-list --objectsoutput viapipe.LinewiseFunction, whose underlyingbufio.Scannerrejects any line longer than the default 64 KiB (bufio.MaxScanTokenSize).Fix
Replace
pipe.LinewiseFunctionwithpipe.ScannerFunction, supplying abufio.Scannerthat uses the same LF splitting (pipe.ScanLFTerminatedLines) and the same per-line logic, but whose buffer starts at the usual 64 KiB and is allowed to grow up to a newmaxRevListLineLength(64 MiB). Becausebufio.Scanneronly grows its buffer as needed, normal usage still uses minimal memory, while pathologically long lines can now be processed — matching the "normally use fewer resources but deal with larger lines if need be" behaviour suggested in the issue.The
copy-oidsper-line logic itself is unchanged.Tests
Added
TestObjectIterLongPath(git/obj_iter_test.go), which builds a tree whose single entry has a 100 KiB file name (created viagit hash-object --literallyto bypass git's path-length fsck check), then walks it withObjectIter.copy-oids: bufio.Scanner: token too long.go test ./...,go vet ./..., andgofmtare all clean.Note on the 64 MiB cap
bufio.Scannerrequires an upper bound on token size, so this raises the limit rather than removing it entirely. 64 MiB is far beyond any realistic path name (and ~5× the largest line reported in #157) while still bounding per-line memory use. If you'd prefer a fully streaming approach instead — reading just the leading OID and discarding the rest of each line, so memory stays O(hash length) regardless of path length — I'm happy to take the PR in that direction.Closes #157