maintenance: auto-size incremental-repack batch

When repacking during the 'incremental-repack' task, we use the
--batch-size option in 'git multi-pack-index repack'. The initial setting
used --batch-size=0 to repack everything into a single pack-file. This is
not sustainable for a large repository. The amount of work required is
also likely to use too many system resources for a background job.

Update the 'incremental-repack' task by dynamically computing a
--batch-size option based on the current pack-file structure.

The dynamic default size is computed with this idea in mind for a client
repository that was cloned from a very large remote: there is likely one
"big" pack-file that was created at clone time. Thus, do not try
repacking it as it is likely packed efficiently by the server.

Instead, we select the second-largest pack-file, and create a batch size
that is one larger than that pack-file. If there are three or more
pack-files, then this guarantees that at least two will be combined into
a new pack-file.

Of course, this means that the second-largest pack-file size is likely
to grow over time and may eventually surpass the initially-cloned
pack-file. Recall that the pack-file batch is selected in a greedy
manner: the packs are considered from oldest to newest and are selected
if they have size smaller than the batch size until the total selected
size is larger than the batch size. Thus, that oldest "clone" pack will
be first to repack after the new data creates a pack larger than that.

We also want to place some limits on how large these pack-files become,
in order to bound the amount of time spent repacking. A maximum
batch-size of two gigabytes means that large repositories will never be
packed into a single pack-file using this job, but also that repack is
rather expensive. This is a trade-off that is valuable to have if the
maintenance is being run automatically or in the background. Users who
truly want to optimize for space and performance (and are willing to pay
the upfront cost of a full repack) can use the 'gc' task to do so.

Create a test for this two gigabyte limit by creating an EXPENSIVE test
that generates two pack-files of roughly 2.5 gigabytes in size, then
performs an incremental repack. Check that the --batch-size argument in
the subcommand uses the hard-coded maximum.

Helped-by: Chris Torek <chris.torek@gmail.com>
Reported-by: Son Luong Ngoc <sluongng@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2 files changed
tree: 3ba0da89780a3e44fedb043cc4e4e8d8558e13f4
  1. .cirrus.yml
  2. .clang-format
  3. .editorconfig
  4. .gitattributes
  5. .github/
  6. .gitignore
  7. .gitmodules
  8. .mailmap
  9. .travis.yml
  10. .tsan-suppressions
  11. CODE_OF_CONDUCT.md
  12. COPYING
  13. Documentation/
  14. GIT-VERSION-GEN
  15. INSTALL
  16. LGPL-2.1
  17. Makefile
  18. README.md
  19. abspath.c
  20. aclocal.m4
  21. add-interactive.c
  22. add-interactive.h
  23. add-patch.c
  24. advice.c
  25. advice.h
  26. alias.c
  27. alias.h
  28. alloc.c
  29. alloc.h
  30. apply.c
  31. apply.h
  32. archive-tar.c
  33. archive-zip.c
  34. archive.c
  35. archive.h
  36. attr.c
  37. attr.h
  38. banned.h
  39. base85.c
  40. bisect.c
  41. bisect.h
  42. blame.c
  43. blame.h
  44. blob.c
  45. blob.h
  46. block-sha1/
  47. bloom.c
  48. bloom.h
  49. branch.c
  50. branch.h
  51. bugreport.c
  52. builtin.h
  53. builtin/
  54. bulk-checkin.c
  55. bulk-checkin.h
  56. bundle.c
  57. bundle.h
  58. cache-tree.c
  59. cache-tree.h
  60. cache.h
  61. chdir-notify.c
  62. chdir-notify.h
  63. check-builtins.sh
  64. check_bindir
  65. checkout.c
  66. checkout.h
  67. ci/
  68. color.c
  69. color.h
  70. column.c
  71. column.h
  72. combine-diff.c
  73. command-list.txt
  74. commit-graph.c
  75. commit-graph.h
  76. commit-reach.c
  77. commit-reach.h
  78. commit-slab-decl.h
  79. commit-slab-impl.h
  80. commit-slab.h
  81. commit.c
  82. commit.h
  83. common-main.c
  84. compat/
  85. config.c
  86. config.h
  87. config.mak.dev
  88. config.mak.in
  89. config.mak.uname
  90. configure.ac
  91. connect.c
  92. connect.h
  93. connected.c
  94. connected.h
  95. contrib/
  96. convert.c
  97. convert.h
  98. copy.c
  99. credential-cache--daemon.c
  100. credential-cache.c
  101. credential-store.c
  102. credential.c
  103. credential.h
  104. csum-file.c
  105. csum-file.h
  106. ctype.c
  107. daemon.c
  108. date.c
  109. decorate.c
  110. decorate.h
  111. delta-islands.c
  112. delta-islands.h
  113. delta.h
  114. detect-compiler
  115. diff-delta.c
  116. diff-lib.c
  117. diff-no-index.c
  118. diff.c
  119. diff.h
  120. diffcore-break.c
  121. diffcore-delta.c
  122. diffcore-order.c
  123. diffcore-pickaxe.c
  124. diffcore-rename.c
  125. diffcore.h
  126. dir-iterator.c
  127. dir-iterator.h
  128. dir.c
  129. dir.h
  130. editor.c
  131. entry.c
  132. environment.c
  133. ewah/
  134. exec-cmd.c
  135. exec-cmd.h
  136. fast-import.c
  137. fetch-negotiator.c
  138. fetch-negotiator.h
  139. fetch-pack.c
  140. fetch-pack.h
  141. fmt-merge-msg.c
  142. fmt-merge-msg.h
  143. fsck.c
  144. fsck.h
  145. fsmonitor.c
  146. fsmonitor.h
  147. fuzz-commit-graph.c
  148. fuzz-pack-headers.c
  149. fuzz-pack-idx.c
  150. generate-cmdlist.sh
  151. generate-configlist.sh
  152. gettext.c
  153. gettext.h
  154. git-add--interactive.perl
  155. git-archimport.perl
  156. git-bisect.sh
  157. git-compat-util.h
  158. git-cvsexportcommit.perl
  159. git-cvsimport.perl
  160. git-cvsserver.perl
  161. git-difftool--helper.sh
  162. git-filter-branch.sh
  163. git-gui/
  164. git-instaweb.sh
  165. git-merge-octopus.sh
  166. git-merge-one-file.sh
  167. git-merge-resolve.sh
  168. git-mergetool--lib.sh
  169. git-mergetool.sh
  170. git-p4.py
  171. git-parse-remote.sh
  172. git-quiltimport.sh
  173. git-rebase--preserve-merges.sh
  174. git-request-pull.sh
  175. git-send-email.perl
  176. git-sh-i18n.sh
  177. git-sh-setup.sh
  178. git-submodule.sh
  179. git-svn.perl
  180. git-web--browse.sh
  181. git.c
  182. git.rc
  183. gitk-git/
  184. gitweb/
  185. gpg-interface.c
  186. gpg-interface.h
  187. graph.c
  188. graph.h
  189. grep.c
  190. grep.h
  191. hash.h
  192. hashmap.c
  193. hashmap.h
  194. help.c
  195. help.h
  196. hex.c
  197. http-backend.c
  198. http-fetch.c
  199. http-push.c
  200. http-walker.c
  201. http.c
  202. http.h
  203. ident.c
  204. imap-send.c
  205. interdiff.c
  206. interdiff.h
  207. iterator.h
  208. json-writer.c
  209. json-writer.h
  210. khash.h
  211. kwset.c
  212. kwset.h
  213. levenshtein.c
  214. levenshtein.h
  215. line-log.c
  216. line-log.h
  217. line-range.c
  218. line-range.h
  219. linear-assignment.c
  220. linear-assignment.h
  221. list-objects-filter-options.c
  222. list-objects-filter-options.h
  223. list-objects-filter.c
  224. list-objects-filter.h
  225. list-objects.c
  226. list-objects.h
  227. list.h
  228. ll-merge.c
  229. ll-merge.h
  230. lockfile.c
  231. lockfile.h
  232. log-tree.c
  233. log-tree.h
  234. ls-refs.c
  235. ls-refs.h
  236. mailinfo.c
  237. mailinfo.h
  238. mailmap.c
  239. mailmap.h
  240. match-trees.c
  241. mem-pool.c
  242. mem-pool.h
  243. merge-blobs.c
  244. merge-blobs.h
  245. merge-recursive.c
  246. merge-recursive.h
  247. merge.c
  248. mergesort.c
  249. mergesort.h
  250. mergetools/
  251. midx.c
  252. midx.h
  253. name-hash.c
  254. negotiator/
  255. notes-cache.c
  256. notes-cache.h
  257. notes-merge.c
  258. notes-merge.h
  259. notes-utils.c
  260. notes-utils.h
  261. notes.c
  262. notes.h
  263. object-store.h
  264. object.c
  265. object.h
  266. oid-array.c
  267. oid-array.h
  268. oidmap.c
  269. oidmap.h
  270. oidset.c
  271. oidset.h
  272. pack-bitmap-write.c
  273. pack-bitmap.c
  274. pack-bitmap.h
  275. pack-check.c
  276. pack-objects.c
  277. pack-objects.h
  278. pack-revindex.c
  279. pack-revindex.h
  280. pack-write.c
  281. pack.h
  282. packfile.c
  283. packfile.h
  284. pager.c
  285. parse-options-cb.c
  286. parse-options.c
  287. parse-options.h
  288. patch-delta.c
  289. patch-ids.c
  290. patch-ids.h
  291. path.c
  292. path.h
  293. pathspec.c
  294. pathspec.h
  295. perl/
  296. pkt-line.c
  297. pkt-line.h
  298. po/
  299. ppc/
  300. preload-index.c
  301. pretty.c
  302. pretty.h
  303. prio-queue.c
  304. prio-queue.h
  305. progress.c
  306. progress.h
  307. promisor-remote.c
  308. promisor-remote.h
  309. prompt.c
  310. prompt.h
  311. protocol.c
  312. protocol.h
  313. prune-packed.c
  314. prune-packed.h
  315. quote.c
  316. quote.h
  317. range-diff.c
  318. range-diff.h
  319. reachable.c
  320. reachable.h
  321. read-cache.c
  322. rebase-interactive.c
  323. rebase-interactive.h
  324. rebase.c
  325. rebase.h
  326. ref-filter.c
  327. ref-filter.h
  328. reflog-walk.c
  329. reflog-walk.h
  330. refs.c
  331. refs.h
  332. refs/
  333. refspec.c
  334. refspec.h
  335. remote-curl.c
  336. remote-testsvn.c
  337. remote.c
  338. remote.h
  339. replace-object.c
  340. replace-object.h
  341. repo-settings.c
  342. repository.c
  343. repository.h
  344. rerere.c
  345. rerere.h
  346. reset.c
  347. reset.h
  348. resolve-undo.c
  349. resolve-undo.h
  350. revision.c
  351. revision.h
  352. run-command.c
  353. run-command.h
  354. send-pack.c
  355. send-pack.h
  356. sequencer.c
  357. sequencer.h
  358. serve.c
  359. serve.h
  360. server-info.c
  361. setup.c
  362. sh-i18n--envsubst.c
  363. sha1-file.c
  364. sha1-lookup.c
  365. sha1-lookup.h
  366. sha1-name.c
  367. sha1dc/
  368. sha1dc_git.c
  369. sha1dc_git.h
  370. sha256/
  371. shallow.c
  372. shallow.h
  373. shell.c
  374. shortlog.h
  375. sideband.c
  376. sideband.h
  377. sigchain.c
  378. sigchain.h
  379. split-index.c
  380. split-index.h
  381. stable-qsort.c
  382. strbuf.c
  383. strbuf.h
  384. streaming.c
  385. streaming.h
  386. string-list.c
  387. string-list.h
  388. strvec.c
  389. strvec.h
  390. sub-process.c
  391. sub-process.h
  392. submodule-config.c
  393. submodule-config.h
  394. submodule.c
  395. submodule.h
  396. symlinks.c
  397. t/
  398. tag.c
  399. tag.h
  400. tar.h
  401. tempfile.c
  402. tempfile.h
  403. templates/
  404. thread-utils.c
  405. thread-utils.h
  406. tmp-objdir.c
  407. tmp-objdir.h
  408. trace.c
  409. trace.h
  410. trace2.c
  411. trace2.h
  412. trace2/
  413. trailer.c
  414. trailer.h
  415. transport-helper.c
  416. transport-internal.h
  417. transport.c
  418. transport.h
  419. tree-diff.c
  420. tree-walk.c
  421. tree-walk.h
  422. tree.c
  423. tree.h
  424. unicode-width.h
  425. unimplemented.sh
  426. unix-socket.c
  427. unix-socket.h
  428. unpack-trees.c
  429. unpack-trees.h
  430. upload-pack.c
  431. upload-pack.h
  432. url.c
  433. url.h
  434. urlmatch.c
  435. urlmatch.h
  436. usage.c
  437. userdiff.c
  438. userdiff.h
  439. utf8.c
  440. utf8.h
  441. varint.c
  442. varint.h
  443. vcs-svn/
  444. version.c
  445. version.h
  446. versioncmp.c
  447. walker.c
  448. walker.h
  449. wildmatch.c
  450. wildmatch.h
  451. worktree.c
  452. worktree.h
  453. wrap-for-bin.sh
  454. wrapper.c
  455. write-or-die.c
  456. ws.c
  457. wt-status.c
  458. wt-status.h
  459. xdiff-interface.c
  460. xdiff-interface.h
  461. xdiff/
  462. zlib.c
README.md

Build status

Git - fast, scalable, distributed revision control system

Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.

Git is an Open Source project covered by the GNU General Public License version 2 (some parts of it are under different licenses, compatible with the GPLv2). It was originally written by Linus Torvalds with help of a group of hackers around the net.

Please read the file INSTALL for installation instructions.

Many Git online resources are accessible from https://git-scm.com/ including full documentation and Git related tools.

See Documentation/gittutorial.txt to get started, then see Documentation/giteveryday.txt for a useful minimum set of commands, and Documentation/git-<commandname>.txt for documentation of each command. If git has been correctly installed, then the tutorial can also be read with man gittutorial or git help tutorial, and the documentation of each command with man git-<commandname> or git help <commandname>.

CVS users may also want to read Documentation/gitcvs-migration.txt (man gitcvs-migration or git help cvs-migration if git is installed).

The user discussion and development of Git take place on the Git mailing list -- everyone is welcome to post bug reports, feature requests, comments and patches to git@vger.kernel.org (read Documentation/SubmittingPatches for instructions on patch submission). To subscribe to the list, send an email with just “subscribe git” in the body to majordomo@vger.kernel.org. The mailing list archives are available at https://lore.kernel.org/git/, http://marc.info/?l=git and other archival sites.

Issues which are security relevant should be disclosed privately to the Git Security mailing list git-security@googlegroups.com.

The maintainer frequently sends the “What's cooking” reports that list the current status of various development topics to the mailing list. The discussion following them give a good reference for project status, development direction and remaining tasks.

The name “git” was given by Linus Torvalds when he wrote the very first version. He described the tool as “the stupid content tracker” and the name as (depending on your mood):

  • random three-letter combination that is pronounceable, and not actually used by any common UNIX command. The fact that it is a mispronunciation of “get” may or may not be relevant.
  • stupid. contemptible and despicable. simple. Take your pick from the dictionary of slang.
  • “global information tracker”: you're in a good mood, and it actually works for you. Angels sing, and a light suddenly fills the room.
  • “goddamn idiotic truckload of sh*t”: when it breaks