Bug: bufio: buffer full

[icambridge]

I'm currently using master from last night.

I get the following error message. I assume it'll be one of the three repositories that had just started processing. (I clearly need to improve the error logging to include the repository).

INFO[0362] Started processing repository https://github.com/zgpmax/Email-Sender 
INFO[0362] Started processing repository https://github.com/mingzong/connman 
INFO[0363] Started processing repository https://github.com/ricardocarneiro/jquery-1 
INFO[0363] Started processing repository https://github.com/flytzen/Autofac 
ERRO[0363] bufio: buffer full                           

With the error coming from the following code:

func (gc *GitRepository) Commits() (*[]Commit, error) {

	ref, err := gc.repository.Head()
	if err != nil {
		return nil, err
	}

	// ... retrieves the commit object
	commit, err := gc.repository.Commit(ref.Hash())
	if err != nil {
		return nil, err
	}
	commits, err := commit.History()

	if err != nil {
		return nil, err
	}
	output := []Commit{}
	for _, orgCommit := range commits {

		if err != nil {
			break
		}
		newCommit := convertCommit(*orgCommit)
		output = append(output, newCommit)
	}

	return &output, err
}

Which i think comes from

go-git/plumbing/object/tag.go

Line 85 in 965b8e4

line, err := r.ReadSlice('\n')

. Since it's the only method that you guys use that returns it that I can see in https://golang.org/pkg/bufio/.

[smola]

@icambridge Could you try reproducing it with a single repository or a use case as minimal as possible? Thank you!

[mcuadros]

Please let you know if still is happening

[hansrodtang]

This happens to me on this Chromium repository, using object.WalkCommitHistory on master, the point of the error for me was r.ReadSlice in commit.Decode.

Starting commit hash: b6b96c3de7c6db78762ce660b274514bd18a1c0b.

In the walker I just placed a simple counter, the last one counted was 451 (hash: 6d3b0113cb987a958bc03faf494a37a3750aff5f).

git rev-list --count master goes to 578829 and if I replace ReadSlice with ReadBytes as a quick workaround, my WalkCommitHistory test reaches the same number.

[lupine]

I'm also seeing this on the GitLab EE repository, walking backwards from 84ea4fa34fbcda996cc7c3fd970c47e90ae47aaf

The ReadSlice docs say:

ReadSlice fails with error ErrBufferFull if the buffer fills without a delim. Because the data returned from ReadSlice will be overwritten by the next I/O operation, most clients should use ReadBytes or ReadString instead.

Seems the default buffer size is 64KiB, if one isn't specified?

[lupine]

Here's the failing commit: https://gitlab.com/gitlab-org/gitlab-ee/commit/1bf2fe276ff084d3b2e0860710ec115a317dd9fc

That extremely long line is 5517 bytes...

[lupine]

i've added #314 - I'd really like to see this fixed ^^