maintenance: add incremental-repack task

The previous change cleaned up loose objects using the
'loose-objects' that can be run safely in the background. Add a
similar job that performs similar cleanups for pack-files.

One issue with running 'git repack' is that it is designed to
repack all pack-files into a single pack-file. While this is the
most space-efficient way to store object data, it is not time or
memory efficient. This becomes extremely important if the repo is
so large that a user struggles to store two copies of the pack on
their disk.

Instead, perform an "incremental" repack by collecting a few small
pack-files into a new pack-file. The multi-pack-index facilitates
this process ever since 'git multi-pack-index expire' was added in
19575c7 (multi-pack-index: implement 'expire' subcommand,
2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1
(midx: implement midx_repack(), 2019-06-10).

The 'incremental-repack' task runs the following steps:

1. 'git multi-pack-index write' creates a multi-pack-index file if
   one did not exist, and otherwise will update the multi-pack-index
   with any new pack-files that appeared since the last write. This
   is particularly relevant with the background fetch job.

   When the multi-pack-index sees two copies of the same object, it
   stores the offset data into the newer pack-file. This means that
   some old pack-files could become "unreferenced" which I will use
   to mean "a pack-file that is in the pack-file list of the
   multi-pack-index but none of the objects in the multi-pack-index
   reference a location inside that pack-file."

2. 'git multi-pack-index expire' deletes any unreferenced pack-files
   and updaes the multi-pack-index to drop those pack-files from the
   list. This is safe to do as concurrent Git processes will see the
   multi-pack-index and not open those packs when looking for object
   contents. (Similar to the 'loose-objects' job, there are some Git
   commands that open pack-files regardless of the multi-pack-index,
   but they are rarely used. Further, a user that self-selects to
   use background operations would likely refrain from using those
   commands.)

3. 'git multi-pack-index repack --bacth-size=<size>' collects a set
   of pack-files that are listed in the multi-pack-index and creates
   a new pack-file containing the objects whose offsets are listed
   by the multi-pack-index to be in those objects. The set of pack-
   files is selected greedily by sorting the pack-files by modified
   time and adding a pack-file to the set if its "expected size" is
   smaller than the batch size until the total expected size of the
   selected pack-files is at least the batch size. The "expected
   size" is calculated by taking the size of the pack-file divided
   by the number of objects in the pack-file and multiplied by the
   number of objects from the multi-pack-index with offset in that
   pack-file. The expected size approximates how much data from that
   pack-file will contribute to the resulting pack-file size. The
   intention is that the resulting pack-file will be close in size
   to the provided batch size.

   The next run of the incremental-repack task will delete these
   repacked pack-files during the 'expire' step.

   In this version, the batch size is set to "0" which ignores the
   size restrictions when selecting the pack-files. It instead
   selects all pack-files and repacks all packed objects into a
   single pack-file. This will be updated in the next change, but
   it requires doing some calculations that are better isolated to
   a separate change.

These steps are based on a similar background maintenance step in
Scalar (and VFS for Git) [1]. This was incredibly effective for
users of the Windows OS repository. After using the same VFS for Git
repository for over a year, some users had _thousands_ of pack-files
that combined to up to 250 GB of data. We noticed a few users were
running into the open file descriptor limits (due in part to a bug
in the multi-pack-index fixed by af96fe3 (midx: add packs to
packed_git linked list, 2019-04-29).

These pack-files were mostly small since they contained the commits
and trees that were pushed to the origin in a given hour. The GVFS
protocol includes a "prefetch" step that asks for pre-computed pack-
files containing commits and trees by timestamp. These pack-files
were grouped into "daily" pack-files once a day for up to 30 days.
If a user did not request prefetch packs for over 30 days, then they
would get the entire history of commits and trees in a new, large
pack-file. This led to a large number of pack-files that had poor
delta compression.

By running this pack-file maintenance step once per day, these repos
with thousands of packs spanning 200+ GB dropped to dozens of pack-
files spanning 30-50 GB. This was done all without removing objects
from the system and using a constant batch size of two gigabytes.
Once the work was done to reduce the pack-files to small sizes, the
batch size of two gigabytes means that not every run triggers a
repack operation, so the following run will not expire a pack-file.
This has kept these repos in a "clean" state.

[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
4 files changed
tree: eec2fdbd162e5797eb509ce0d39cdfa21a5345a8
  1. .cirrus.yml
  2. .clang-format
  3. .editorconfig
  4. .gitattributes
  5. .github/
  6. .gitignore
  7. .gitmodules
  8. .mailmap
  9. .travis.yml
  10. .tsan-suppressions
  11. CODE_OF_CONDUCT.md
  12. COPYING
  13. Documentation/
  14. GIT-VERSION-GEN
  15. INSTALL
  16. LGPL-2.1
  17. Makefile
  18. README.md
  19. abspath.c
  20. aclocal.m4
  21. add-interactive.c
  22. add-interactive.h
  23. add-patch.c
  24. advice.c
  25. advice.h
  26. alias.c
  27. alias.h
  28. alloc.c
  29. alloc.h
  30. apply.c
  31. apply.h
  32. archive-tar.c
  33. archive-zip.c
  34. archive.c
  35. archive.h
  36. attr.c
  37. attr.h
  38. banned.h
  39. base85.c
  40. bisect.c
  41. bisect.h
  42. blame.c
  43. blame.h
  44. blob.c
  45. blob.h
  46. block-sha1/
  47. bloom.c
  48. bloom.h
  49. branch.c
  50. branch.h
  51. bugreport.c
  52. builtin.h
  53. builtin/
  54. bulk-checkin.c
  55. bulk-checkin.h
  56. bundle.c
  57. bundle.h
  58. cache-tree.c
  59. cache-tree.h
  60. cache.h
  61. chdir-notify.c
  62. chdir-notify.h
  63. check-builtins.sh
  64. check_bindir
  65. checkout.c
  66. checkout.h
  67. ci/
  68. color.c
  69. color.h
  70. column.c
  71. column.h
  72. combine-diff.c
  73. command-list.txt
  74. commit-graph.c
  75. commit-graph.h
  76. commit-reach.c
  77. commit-reach.h
  78. commit-slab-decl.h
  79. commit-slab-impl.h
  80. commit-slab.h
  81. commit.c
  82. commit.h
  83. common-main.c
  84. compat/
  85. config.c
  86. config.h
  87. config.mak.dev
  88. config.mak.in
  89. config.mak.uname
  90. configure.ac
  91. connect.c
  92. connect.h
  93. connected.c
  94. connected.h
  95. contrib/
  96. convert.c
  97. convert.h
  98. copy.c
  99. credential-cache--daemon.c
  100. credential-cache.c
  101. credential-store.c
  102. credential.c
  103. credential.h
  104. csum-file.c
  105. csum-file.h
  106. ctype.c
  107. daemon.c
  108. date.c
  109. decorate.c
  110. decorate.h
  111. delta-islands.c
  112. delta-islands.h
  113. delta.h
  114. detect-compiler
  115. diff-delta.c
  116. diff-lib.c
  117. diff-no-index.c
  118. diff.c
  119. diff.h
  120. diffcore-break.c
  121. diffcore-delta.c
  122. diffcore-order.c
  123. diffcore-pickaxe.c
  124. diffcore-rename.c
  125. diffcore.h
  126. dir-iterator.c
  127. dir-iterator.h
  128. dir.c
  129. dir.h
  130. editor.c
  131. entry.c
  132. environment.c
  133. ewah/
  134. exec-cmd.c
  135. exec-cmd.h
  136. fast-import.c
  137. fetch-negotiator.c
  138. fetch-negotiator.h
  139. fetch-pack.c
  140. fetch-pack.h
  141. fmt-merge-msg.c
  142. fmt-merge-msg.h
  143. fsck.c
  144. fsck.h
  145. fsmonitor.c
  146. fsmonitor.h
  147. fuzz-commit-graph.c
  148. fuzz-pack-headers.c
  149. fuzz-pack-idx.c
  150. generate-cmdlist.sh
  151. generate-configlist.sh
  152. gettext.c
  153. gettext.h
  154. git-add--interactive.perl
  155. git-archimport.perl
  156. git-bisect.sh
  157. git-compat-util.h
  158. git-cvsexportcommit.perl
  159. git-cvsimport.perl
  160. git-cvsserver.perl
  161. git-difftool--helper.sh
  162. git-filter-branch.sh
  163. git-gui/
  164. git-instaweb.sh
  165. git-merge-octopus.sh
  166. git-merge-one-file.sh
  167. git-merge-resolve.sh
  168. git-mergetool--lib.sh
  169. git-mergetool.sh
  170. git-p4.py
  171. git-parse-remote.sh
  172. git-quiltimport.sh
  173. git-rebase--preserve-merges.sh
  174. git-request-pull.sh
  175. git-send-email.perl
  176. git-sh-i18n.sh
  177. git-sh-setup.sh
  178. git-submodule.sh
  179. git-svn.perl
  180. git-web--browse.sh
  181. git.c
  182. git.rc
  183. gitk-git/
  184. gitweb/
  185. gpg-interface.c
  186. gpg-interface.h
  187. graph.c
  188. graph.h
  189. grep.c
  190. grep.h
  191. hash.h
  192. hashmap.c
  193. hashmap.h
  194. help.c
  195. help.h
  196. hex.c
  197. http-backend.c
  198. http-fetch.c
  199. http-push.c
  200. http-walker.c
  201. http.c
  202. http.h
  203. ident.c
  204. imap-send.c
  205. interdiff.c
  206. interdiff.h
  207. iterator.h
  208. json-writer.c
  209. json-writer.h
  210. khash.h
  211. kwset.c
  212. kwset.h
  213. levenshtein.c
  214. levenshtein.h
  215. line-log.c
  216. line-log.h
  217. line-range.c
  218. line-range.h
  219. linear-assignment.c
  220. linear-assignment.h
  221. list-objects-filter-options.c
  222. list-objects-filter-options.h
  223. list-objects-filter.c
  224. list-objects-filter.h
  225. list-objects.c
  226. list-objects.h
  227. list.h
  228. ll-merge.c
  229. ll-merge.h
  230. lockfile.c
  231. lockfile.h
  232. log-tree.c
  233. log-tree.h
  234. ls-refs.c
  235. ls-refs.h
  236. mailinfo.c
  237. mailinfo.h
  238. mailmap.c
  239. mailmap.h
  240. match-trees.c
  241. mem-pool.c
  242. mem-pool.h
  243. merge-blobs.c
  244. merge-blobs.h
  245. merge-recursive.c
  246. merge-recursive.h
  247. merge.c
  248. mergesort.c
  249. mergesort.h
  250. mergetools/
  251. midx.c
  252. midx.h
  253. name-hash.c
  254. negotiator/
  255. notes-cache.c
  256. notes-cache.h
  257. notes-merge.c
  258. notes-merge.h
  259. notes-utils.c
  260. notes-utils.h
  261. notes.c
  262. notes.h
  263. object-store.h
  264. object.c
  265. object.h
  266. oid-array.c
  267. oid-array.h
  268. oidmap.c
  269. oidmap.h
  270. oidset.c
  271. oidset.h
  272. pack-bitmap-write.c
  273. pack-bitmap.c
  274. pack-bitmap.h
  275. pack-check.c
  276. pack-objects.c
  277. pack-objects.h
  278. pack-revindex.c
  279. pack-revindex.h
  280. pack-write.c
  281. pack.h
  282. packfile.c
  283. packfile.h
  284. pager.c
  285. parse-options-cb.c
  286. parse-options.c
  287. parse-options.h
  288. patch-delta.c
  289. patch-ids.c
  290. patch-ids.h
  291. path.c
  292. path.h
  293. pathspec.c
  294. pathspec.h
  295. perl/
  296. pkt-line.c
  297. pkt-line.h
  298. po/
  299. ppc/
  300. preload-index.c
  301. pretty.c
  302. pretty.h
  303. prio-queue.c
  304. prio-queue.h
  305. progress.c
  306. progress.h
  307. promisor-remote.c
  308. promisor-remote.h
  309. prompt.c
  310. prompt.h
  311. protocol.c
  312. protocol.h
  313. prune-packed.c
  314. prune-packed.h
  315. quote.c
  316. quote.h
  317. range-diff.c
  318. range-diff.h
  319. reachable.c
  320. reachable.h
  321. read-cache.c
  322. rebase-interactive.c
  323. rebase-interactive.h
  324. rebase.c
  325. rebase.h
  326. ref-filter.c
  327. ref-filter.h
  328. reflog-walk.c
  329. reflog-walk.h
  330. refs.c
  331. refs.h
  332. refs/
  333. refspec.c
  334. refspec.h
  335. remote-curl.c
  336. remote-testsvn.c
  337. remote.c
  338. remote.h
  339. replace-object.c
  340. replace-object.h
  341. repo-settings.c
  342. repository.c
  343. repository.h
  344. rerere.c
  345. rerere.h
  346. reset.c
  347. reset.h
  348. resolve-undo.c
  349. resolve-undo.h
  350. revision.c
  351. revision.h
  352. run-command.c
  353. run-command.h
  354. send-pack.c
  355. send-pack.h
  356. sequencer.c
  357. sequencer.h
  358. serve.c
  359. serve.h
  360. server-info.c
  361. setup.c
  362. sh-i18n--envsubst.c
  363. sha1-file.c
  364. sha1-lookup.c
  365. sha1-lookup.h
  366. sha1-name.c
  367. sha1dc/
  368. sha1dc_git.c
  369. sha1dc_git.h
  370. sha256/
  371. shallow.c
  372. shallow.h
  373. shell.c
  374. shortlog.h
  375. sideband.c
  376. sideband.h
  377. sigchain.c
  378. sigchain.h
  379. split-index.c
  380. split-index.h
  381. stable-qsort.c
  382. strbuf.c
  383. strbuf.h
  384. streaming.c
  385. streaming.h
  386. string-list.c
  387. string-list.h
  388. strvec.c
  389. strvec.h
  390. sub-process.c
  391. sub-process.h
  392. submodule-config.c
  393. submodule-config.h
  394. submodule.c
  395. submodule.h
  396. symlinks.c
  397. t/
  398. tag.c
  399. tag.h
  400. tar.h
  401. tempfile.c
  402. tempfile.h
  403. templates/
  404. thread-utils.c
  405. thread-utils.h
  406. tmp-objdir.c
  407. tmp-objdir.h
  408. trace.c
  409. trace.h
  410. trace2.c
  411. trace2.h
  412. trace2/
  413. trailer.c
  414. trailer.h
  415. transport-helper.c
  416. transport-internal.h
  417. transport.c
  418. transport.h
  419. tree-diff.c
  420. tree-walk.c
  421. tree-walk.h
  422. tree.c
  423. tree.h
  424. unicode-width.h
  425. unimplemented.sh
  426. unix-socket.c
  427. unix-socket.h
  428. unpack-trees.c
  429. unpack-trees.h
  430. upload-pack.c
  431. upload-pack.h
  432. url.c
  433. url.h
  434. urlmatch.c
  435. urlmatch.h
  436. usage.c
  437. userdiff.c
  438. userdiff.h
  439. utf8.c
  440. utf8.h
  441. varint.c
  442. varint.h
  443. vcs-svn/
  444. version.c
  445. version.h
  446. versioncmp.c
  447. walker.c
  448. walker.h
  449. wildmatch.c
  450. wildmatch.h
  451. worktree.c
  452. worktree.h
  453. wrap-for-bin.sh
  454. wrapper.c
  455. write-or-die.c
  456. ws.c
  457. wt-status.c
  458. wt-status.h
  459. xdiff-interface.c
  460. xdiff-interface.h
  461. xdiff/
  462. zlib.c
README.md

Build status

Git - fast, scalable, distributed revision control system

Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.

Git is an Open Source project covered by the GNU General Public License version 2 (some parts of it are under different licenses, compatible with the GPLv2). It was originally written by Linus Torvalds with help of a group of hackers around the net.

Please read the file INSTALL for installation instructions.

Many Git online resources are accessible from https://git-scm.com/ including full documentation and Git related tools.

See Documentation/gittutorial.txt to get started, then see Documentation/giteveryday.txt for a useful minimum set of commands, and Documentation/git-<commandname>.txt for documentation of each command. If git has been correctly installed, then the tutorial can also be read with man gittutorial or git help tutorial, and the documentation of each command with man git-<commandname> or git help <commandname>.

CVS users may also want to read Documentation/gitcvs-migration.txt (man gitcvs-migration or git help cvs-migration if git is installed).

The user discussion and development of Git take place on the Git mailing list -- everyone is welcome to post bug reports, feature requests, comments and patches to git@vger.kernel.org (read Documentation/SubmittingPatches for instructions on patch submission). To subscribe to the list, send an email with just “subscribe git” in the body to majordomo@vger.kernel.org. The mailing list archives are available at https://lore.kernel.org/git/, http://marc.info/?l=git and other archival sites.

Issues which are security relevant should be disclosed privately to the Git Security mailing list git-security@googlegroups.com.

The maintainer frequently sends the “What's cooking” reports that list the current status of various development topics to the mailing list. The discussion following them give a good reference for project status, development direction and remaining tasks.

The name “git” was given by Linus Torvalds when he wrote the very first version. He described the tool as “the stupid content tracker” and the name as (depending on your mood):

  • random three-letter combination that is pronounceable, and not actually used by any common UNIX command. The fact that it is a mispronunciation of “get” may or may not be relevant.
  • stupid. contemptible and despicable. simple. Take your pick from the dictionary of slang.
  • “global information tracker”: you're in a good mood, and it actually works for you. Angels sing, and a light suddenly fills the room.
  • “goddamn idiotic truckload of sh*t”: when it breaks