1m commits

Author: cirosantilli

generate-commits-fast.md

@@ -2,6 +2,8 @@
 
 It is sometimes interesting to generate a ton of commits to test some edge case, but it is not trivial to go way above 1000 commits in a reasonable amount of time.
 
+Bottom line: don't use `git`. The manual Python code under [other-test-repos](other-test-repos/) presents a huge speedup. TODO: try gitlib2.
+
 1000 operations take on a my computer:
 
 - echo to file, add and commit: 43s

other-test-repos/.gitignore

@@ -1,3 +1,4 @@
 *.pyc
+/tmp/
 /*.tmp/
 __pycache__/

other-test-repos/README.md

@@ -7,6 +7,7 @@ Tests that are very large will not be included here to keep this repository smal
 - <https://github.com/cirosantilli/test-deep>
 - <https://github.com/cirosantilli/test-diff-many-files>
 - <https://github.com/cirosantilli/test-pr-many-commits>
+- <https://github.com/cirosantilli/test-many-commits-1m> [many-commits.py](many-commits.py)
 
 There are also some tests that could not be included here conveniently:
 
@@ -27,6 +28,7 @@ There are also some tests that could not be included here conveniently:
 - <https://github.com/cirosantilli/test-symlink-self>
 - <https://github.com/cirosantilli/test-symlink-start-null>
 - <https://gitlab.com/cirosantilli/test-GIT/tree/master> (fails on GitHub)
+- <https://gitlab.com/cirosantilli/test-commit-many-parents-1m>
 
 Other similar repos from other people:
 

other-test-repos/duplicate-parent-lowlevel.py

@@ -3,6 +3,12 @@
 """
 Git does not let a commit have twice the same parent, but GitHub does, and normally shows it.
 But as of 2016-05-17 they didn't page this edge case, and it 502's the commit for large numbers of links.
+
+If you increase the value a lot, when you clone and cd into the repo. your computer may bog down
+if you show a git status on the bash, because Git memory explodes trying to parse that.
+Actual compressed size is very small though, since gzip compresses all that repeated data very efficiently.
+
+push would require an obscene ammount of memory (malloc fails on `ulimit -Sv`), so I couldn't test it.
 """
 
 import itertools
@@ -13,7 +19,7 @@
 
 tree = util.create_tree_with_one_file()
 commit, _, _ = util.save_commit_object(tree, author_name=b'a')
-commit, _, _ = util.save_commit_object(tree, itertools.repeat(commit, 1000000), author_name=b'b')
+commit, _, _ = util.save_commit_object(tree, itertools.repeat(commit, 10000000), author_name=b'b')
 
 # Finish.
 util.create_master(commit)

other-test-repos/many-commits.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+
+"""
+You may want to create this in a tmpfs or ramfs, since deleting the generate repository can take a **huge** ammount of time.
+
+    ulimit -Sv 500000
+    sudo umount tmp && \
+        sudo mount -t tmpfs -o size=1g tmpfs tmp && \
+        sudo chown $USER:$USER tmp &&
+        ./imagine-all-the-people.py
+
+The tags can be used to push by parts to GitHub, which does not accept 1M at once:
+
+    remote='[email protected]:cirosantilli/test-many-commits-1m.git'
+    for i in `seq 10 10 100`; do
+        git --git-dir=tmp/repo.tmp/.git push -f "$remote" "$i:master"
+    done
+    # TODO for some reason I needed this afterwards.
+    git --git-dir=tmp/repo.tmp/.git push "$remote" 'master'
+"""
+
+import datetime
+import subprocess
+
+import util
+
+util.init()
+
+tree = util.create_tree_with_one_file()
+commit = None
+n = 1000000
+percent = (n / 100)
+p = 0
+for i in range(n):
+    commit, _, _ = util.save_commit_object(tree, (commit,),
+            message=(str(i).encode('ascii')))
+    if i % percent == 0:
+        print(p)
+        print(datetime.datetime.now())
+        p += 1
+
+        # Lose objects are too large and blow up the tmpfs.
+
+        # Does clean packets, but the calculation takes more and more memory,
+        # and slows down and blows up at the end. TODO which subcommand blows up eactly?.
+        #subprocess.check_output(['git', 'gc'])
+
+        subprocess.check_output(['git', 'repack'])
+        subprocess.check_output(['git', 'prune-packed'])
+
+        subprocess.check_output(['git', 'tag', str(p), commit])
+
+# Finish.
+util.create_master(commit)
+util.clone()

other-test-repos/util.py

@@ -1,3 +1,8 @@
+"""
+TODO packfile operations instead of just object. Could be more efficient.
+But also harder to implement that format.
+"""
+
 import hashlib
 import os
 import shutil
@@ -30,7 +35,7 @@
 default_parents = ()
 
 def init():
-    repo = 'repo.tmp'
+    repo = 'tmp/repo.tmp'
     for d in (repo, 'clone.tmp'):
         shutil.rmtree(d, ignore_errors=True)
     os.mkdir(repo)
@@ -66,7 +71,7 @@ def save_commit_object(
         committer_email=default_committer_email,
         committer_date=default_committer_date,
         message=default_message):
-    if parents:
+    if parents and parents[0]:
         parents_bytes = b''
         sep = b'\nparent '
         parents_bytes = sep + sep.join(parents) + b'\n'