odb/source-files: move alternates into the backend

Originally, when designing pluggable object databases the goal was that
the object database can have multiple sources, and every source attached
to it could use a different backend. This would have allowed for quite a
lot of flexibility, as you could trivially mix and match different kinds
of object storages in whatever way you like.

But while well-intentioned, this design led to a bunch of conceptual
problems:

  - We're now trying to read objects in source order, whereas we
    previously tried to read objects via packfiles before trying to read
    them via loose objects. This led to a performance regression when
    using alternates or when using a quarantine directory.

  - Some data structures are supposed to only ever exist once, like for
    example bitmaps and commit graphs. At the same time, those data
    structures also span across the union of all objects, so they may
    cross sources.

  - It is unclear how we can extend GIT_OBJECT_DIRECTORY or
    GIT_ALTERNATE_OBJECT_DIRECTORIES to become backend-agnostic in a
    backwards-compatible way. In general, introducing an object storage
    extension into the current status quo where alternates may have to
    be extended to become generic was proving to be painful.

  - Some mechanisms of alternates assume way too much about how exactly
    their backends work. Alternate refs for example assume that the
    alternate is backed by a filesystem path, and that this filesystem
    path may also allow us to read references. This is not a given
    though, as backends may not even have local data at all.

In short, there are a bunch of conceptual mismatches when we have
alternates and pluggable object databases coexist. So while the original
idea was nice, it does not result in a system that is easy to reason
about.

Correct course by moving alternates into the "files" source itself so
that it becomes an implementation detail thereof so that we can avoid
all of these shortcomings. While it's unfortunate that we cannot easily
mix and match sources now, that ability doesn't go away. It's still very
much feasible to introduce a new backend that allows for exactly that
use case, and such a backend may also be a lot more flexible as we can
now add new logic to determine which objects should be stored where. So
the original motivation for having per-source backends can still be
realized with the new architecture.

Note that as part of this move, we also handle the GIT_OBJECT_DIRECTORY
and GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables in the
"files" backend. This may be surprising at first, but object directories
are very much a concept of that backend, too. So these variables would
have bad interactions with other backends, and they create a bit of a
mismatch with the eventual object storage extension that we plan to
introduce.

Note that this commit is way larger than I'd like it to be. I'm sorry, I
couldn't find a way to split it up further. That being said, most of the
changes are straight-forward conversions that go from iterating over
sources to iterating over object directories. The more involved changes
are moving the infrastructure to track individual sources from "odb.c"
and moving them into "odb/source-files.c".

This also serves to show a bit of a who's-who of commands that don't
work properly with pluggable object databases. Almost all of these
commands are related to housekeeping though -- some of them will be
converted eventually, like for example commit graphs or MIDXs. But many
of them will stay incompatible going forward as they are simply too
specific to the "files" backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
35 files changed
tree: 90f5775b5fd250f7afc90fbdeb0a02d8bbed74f3
  1. .github/
  2. bin-wrappers/
  3. block-sha1/
  4. builtin/
  5. ci/
  6. compat/
  7. compiler-tricks/
  8. contrib/
  9. Documentation/
  10. ewah/
  11. git-gui/
  12. gitk-git/
  13. gitweb/
  14. mergetools/
  15. negotiator/
  16. odb/
  17. oss-fuzz/
  18. perl/
  19. po/
  20. refs/
  21. reftable/
  22. sha1/
  23. sha1dc/
  24. sha256/
  25. src/
  26. subprojects/
  27. t/
  28. templates/
  29. tools/
  30. trace2/
  31. xdiff/
  32. .b4-config
  33. .b4-cover-template
  34. .cirrus.yml
  35. .clang-format
  36. .editorconfig
  37. .gitattributes
  38. .gitignore
  39. .gitlab-ci.yml
  40. .gitmodules
  41. .mailmap
  42. .tsan-suppressions
  43. abspath.c
  44. abspath.h
  45. aclocal.m4
  46. add-interactive.c
  47. add-interactive.h
  48. add-patch.c
  49. add-patch.h
  50. advice.c
  51. advice.h
  52. alias.c
  53. alias.h
  54. alloc.c
  55. alloc.h
  56. apply.c
  57. apply.h
  58. archive-tar.c
  59. archive-zip.c
  60. archive.c
  61. archive.h
  62. attr.c
  63. attr.h
  64. banned.h
  65. base85.c
  66. base85.h
  67. bisect.c
  68. bisect.h
  69. blame.c
  70. blame.h
  71. blob.c
  72. blob.h
  73. bloom.c
  74. bloom.h
  75. branch.c
  76. branch.h
  77. build.rs
  78. builtin.h
  79. bundle-uri.c
  80. bundle-uri.h
  81. bundle.c
  82. bundle.h
  83. cache-tree.c
  84. cache-tree.h
  85. Cargo.toml
  86. cbtree.c
  87. cbtree.h
  88. chdir-notify.c
  89. chdir-notify.h
  90. checkout.c
  91. checkout.h
  92. chunk-format.c
  93. chunk-format.h
  94. CODE_OF_CONDUCT.md
  95. color.c
  96. color.h
  97. column.c
  98. column.h
  99. combine-diff.c
  100. command-list.txt
  101. commit-graph.c
  102. commit-graph.h
  103. commit-reach.c
  104. commit-reach.h
  105. commit-slab-decl.h
  106. commit-slab-impl.h
  107. commit-slab.h
  108. commit.c
  109. commit.h
  110. common-exit.c
  111. common-init.c
  112. common-init.h
  113. common-main.c
  114. config.c
  115. config.h
  116. config.mak.dev
  117. config.mak.in
  118. config.mak.uname
  119. configure.ac
  120. connect.c
  121. connect.h
  122. connected.c
  123. connected.h
  124. convert.c
  125. convert.h
  126. copy.c
  127. copy.h
  128. COPYING
  129. credential.c
  130. credential.h
  131. csum-file.c
  132. csum-file.h
  133. ctype.c
  134. daemon.c
  135. date.c
  136. date.h
  137. decorate.c
  138. decorate.h
  139. delta-islands.c
  140. delta-islands.h
  141. delta.h
  142. diagnose.c
  143. diagnose.h
  144. diff-delta.c
  145. diff-lib.c
  146. diff-merges.c
  147. diff-merges.h
  148. diff-no-index.c
  149. diff.c
  150. diff.h
  151. diffcore-break.c
  152. diffcore-delta.c
  153. diffcore-order.c
  154. diffcore-pickaxe.c
  155. diffcore-rename.c
  156. diffcore-rotate.c
  157. diffcore.h
  158. dir-iterator.c
  159. dir-iterator.h
  160. dir.c
  161. dir.h
  162. editor.c
  163. editor.h
  164. entry.c
  165. entry.h
  166. environment.c
  167. environment.h
  168. exec-cmd.c
  169. exec-cmd.h
  170. fetch-negotiator.c
  171. fetch-negotiator.h
  172. fetch-object-info.c
  173. fetch-object-info.h
  174. fetch-pack.c
  175. fetch-pack.h
  176. fmt-merge-msg.c
  177. fmt-merge-msg.h
  178. for-each-ref.h
  179. fsck.c
  180. fsck.h
  181. fsmonitor--daemon.h
  182. fsmonitor-ipc.c
  183. fsmonitor-ipc.h
  184. fsmonitor-ll.h
  185. fsmonitor-path-utils.h
  186. fsmonitor-settings.c
  187. fsmonitor-settings.h
  188. fsmonitor.c
  189. fsmonitor.h
  190. gettext.c
  191. gettext.h
  192. git-archimport.perl
  193. GIT-BUILD-OPTIONS.in
  194. git-compat-util.h
  195. git-curl-compat.h
  196. git-cvsexportcommit.perl
  197. git-cvsimport.perl
  198. git-cvsserver.perl
  199. git-difftool--helper.sh
  200. git-filter-branch.sh
  201. git-instaweb.sh
  202. git-merge-octopus.sh
  203. git-merge-one-file.sh
  204. git-merge-resolve.sh
  205. git-mergetool--lib.sh
  206. git-mergetool.sh
  207. git-p4.py
  208. git-quiltimport.sh
  209. git-request-pull.sh
  210. git-send-email.perl
  211. git-sh-i18n.sh
  212. git-sh-setup.sh
  213. git-submodule.sh
  214. git-svn.perl
  215. GIT-VERSION-FILE.in
  216. GIT-VERSION-GEN
  217. git-web--browse.sh
  218. git-zlib.c
  219. git-zlib.h
  220. git.c
  221. git.rc.in
  222. gpg-interface.c
  223. gpg-interface.h
  224. graph.c
  225. graph.h
  226. grep.c
  227. grep.h
  228. hash-lookup.c
  229. hash-lookup.h
  230. hash.c
  231. hash.h
  232. hashmap.c
  233. hashmap.h
  234. help.c
  235. help.h
  236. hex-ll.c
  237. hex-ll.h
  238. hex.c
  239. hex.h
  240. hook.c
  241. hook.h
  242. http-backend.c
  243. http-fetch.c
  244. http-push.c
  245. http-walker.c
  246. http.c
  247. http.h
  248. ident.c
  249. ident.h
  250. imap-send.c
  251. INSTALL
  252. iterator.h
  253. json-writer.c
  254. json-writer.h
  255. khash.h
  256. kwset.c
  257. kwset.h
  258. levenshtein.c
  259. levenshtein.h
  260. LGPL-2.1
  261. line-log.c
  262. line-log.h
  263. line-range.c
  264. line-range.h
  265. linear-assignment.c
  266. linear-assignment.h
  267. list-objects-filter-options.c
  268. list-objects-filter-options.h
  269. list-objects-filter.c
  270. list-objects-filter.h
  271. list-objects.c
  272. list-objects.h
  273. list.h
  274. lockfile.c
  275. lockfile.h
  276. log-tree.c
  277. log-tree.h
  278. loose.c
  279. loose.h
  280. ls-refs.c
  281. ls-refs.h
  282. mailinfo.c
  283. mailinfo.h
  284. mailmap.c
  285. mailmap.h
  286. Makefile
  287. match-trees.c
  288. match-trees.h
  289. mem-pool.c
  290. mem-pool.h
  291. merge-blobs.c
  292. merge-blobs.h
  293. merge-ll.c
  294. merge-ll.h
  295. merge-ort-wrappers.c
  296. merge-ort-wrappers.h
  297. merge-ort.c
  298. merge-ort.h
  299. merge.c
  300. merge.h
  301. mergesort.h
  302. meson.build
  303. meson_options.txt
  304. midx-write.c
  305. midx.c
  306. midx.h
  307. name-hash.c
  308. name-hash.h
  309. notes-cache.c
  310. notes-cache.h
  311. notes-merge.c
  312. notes-merge.h
  313. notes-utils.c
  314. notes-utils.h
  315. notes.c
  316. notes.h
  317. object-file-convert.c
  318. object-file-convert.h
  319. object-file.c
  320. object-file.h
  321. object-name.c
  322. object-name.h
  323. object.c
  324. object.h
  325. odb.c
  326. odb.h
  327. oid-array.c
  328. oid-array.h
  329. oidmap.c
  330. oidmap.h
  331. oidset.c
  332. oidset.h
  333. oidtree.c
  334. oidtree.h
  335. pack-bitmap-write.c
  336. pack-bitmap.c
  337. pack-bitmap.h
  338. pack-check.c
  339. pack-mtimes.c
  340. pack-mtimes.h
  341. pack-objects.c
  342. pack-objects.h
  343. pack-refs.c
  344. pack-refs.h
  345. pack-revindex.c
  346. pack-revindex.h
  347. pack-write.c
  348. pack.h
  349. packfile-list.c
  350. packfile-list.h
  351. packfile.c
  352. packfile.h
  353. pager.c
  354. pager.h
  355. parallel-checkout.c
  356. parallel-checkout.h
  357. parse-options-cb.c
  358. parse-options.c
  359. parse-options.h
  360. parse.c
  361. parse.h
  362. patch-delta.c
  363. patch-ids.c
  364. patch-ids.h
  365. path-walk.c
  366. path-walk.h
  367. path.c
  368. path.h
  369. pathspec.c
  370. pathspec.h
  371. pkt-line.c
  372. pkt-line.h
  373. preload-index.c
  374. preload-index.h
  375. pretty.c
  376. pretty.h
  377. prio-queue.c
  378. prio-queue.h
  379. progress.c
  380. progress.h
  381. promisor-remote.c
  382. promisor-remote.h
  383. prompt.c
  384. prompt.h
  385. protocol-caps.c
  386. protocol-caps.h
  387. protocol.c
  388. protocol.h
  389. prune-packed.c
  390. prune-packed.h
  391. pseudo-merge.c
  392. pseudo-merge.h
  393. quote.c
  394. quote.h
  395. range-diff.c
  396. range-diff.h
  397. reachable.c
  398. reachable.h
  399. read-cache-ll.h
  400. read-cache.c
  401. read-cache.h
  402. README.md
  403. rebase-interactive.c
  404. rebase-interactive.h
  405. rebase.c
  406. rebase.h
  407. ref-filter.c
  408. ref-filter.h
  409. reflog-walk.c
  410. reflog-walk.h
  411. reflog.c
  412. reflog.h
  413. refs.c
  414. refs.h
  415. refspec.c
  416. refspec.h
  417. remote-curl.c
  418. remote.c
  419. remote.h
  420. repack-cruft.c
  421. repack-filtered.c
  422. repack-geometry.c
  423. repack-midx.c
  424. repack-promisor.c
  425. repack.c
  426. repack.h
  427. replace-object.c
  428. replace-object.h
  429. replay.c
  430. replay.h
  431. repo-settings.c
  432. repo-settings.h
  433. repository.c
  434. repository.h
  435. rerere.c
  436. rerere.h
  437. reset.c
  438. reset.h
  439. resolve-undo.c
  440. resolve-undo.h
  441. revision.c
  442. revision.h
  443. run-command.c
  444. run-command.h
  445. sane-ctype.h
  446. scalar.c
  447. SECURITY.md
  448. send-pack.c
  449. send-pack.h
  450. sequencer.c
  451. sequencer.h
  452. serve.c
  453. serve.h
  454. server-info.c
  455. server-info.h
  456. setup.c
  457. setup.h
  458. sh-i18n--envsubst.c
  459. sha1dc_git.c
  460. sha1dc_git.h
  461. shallow.c
  462. shallow.h
  463. shared.mak
  464. shell.c
  465. shortlog.h
  466. sideband.c
  467. sideband.h
  468. sigchain.c
  469. sigchain.h
  470. simple-ipc.h
  471. sparse-index.c
  472. sparse-index.h
  473. split-index.c
  474. split-index.h
  475. stable-qsort.c
  476. stash.h
  477. statinfo.c
  478. statinfo.h
  479. strbuf.c
  480. strbuf.h
  481. string-list.c
  482. string-list.h
  483. strmap.c
  484. strmap.h
  485. strvec.c
  486. strvec.h
  487. sub-process.c
  488. sub-process.h
  489. submodule-config.c
  490. submodule-config.h
  491. submodule.c
  492. submodule.h
  493. symlinks.c
  494. symlinks.h
  495. tag.c
  496. tag.h
  497. tar.h
  498. tempfile.c
  499. tempfile.h
  500. thread-utils.c
  501. thread-utils.h
  502. tmp-objdir.c
  503. tmp-objdir.h
  504. trace.c
  505. trace.h
  506. trace2.c
  507. trace2.h
  508. trailer.c
  509. trailer.h
  510. transport-helper.c
  511. transport-internal.h
  512. transport.c
  513. transport.h
  514. tree-diff.c
  515. tree-walk.c
  516. tree-walk.h
  517. tree.c
  518. tree.h
  519. unicode-width.h
  520. unimplemented.sh
  521. unix-socket.c
  522. unix-socket.h
  523. unix-stream-server.c
  524. unix-stream-server.h
  525. unpack-trees.c
  526. unpack-trees.h
  527. upload-pack.c
  528. upload-pack.h
  529. url.c
  530. url.h
  531. urlmatch.c
  532. urlmatch.h
  533. usage.c
  534. userdiff.c
  535. userdiff.h
  536. utf8.c
  537. utf8.h
  538. varint.c
  539. varint.h
  540. version-def.h.in
  541. version.c
  542. version.h
  543. versioncmp.c
  544. versioncmp.h
  545. walker.c
  546. walker.h
  547. wildmatch.c
  548. wildmatch.h
  549. worktree.c
  550. worktree.h
  551. wrapper.c
  552. wrapper.h
  553. write-or-die.c
  554. write-or-die.h
  555. ws.c
  556. ws.h
  557. wt-status.c
  558. wt-status.h
  559. xdiff-interface.c
  560. xdiff-interface.h
README.md

GitHub build status GitLab build status

Git - fast, scalable, distributed revision control system

Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.

Git is an Open Source project covered by the GNU General Public License version 2 (some parts of it are under different licenses, compatible with the GPLv2). It was originally written by Linus Torvalds with help of a group of hackers around the net.

Please read the file INSTALL for installation instructions.

Many Git online resources are accessible from https://git-scm.com/ including full documentation and Git related tools.

See Documentation/gittutorial.adoc to get started, then see Documentation/giteveryday.adoc for a useful minimum set of commands, and Documentation/git-<commandname>.adoc for documentation of each command. If git has been correctly installed, then the tutorial can also be read with man gittutorial or git help tutorial, and the documentation of each command with man git-<commandname> or git help <commandname>.

CVS users may also want to read Documentation/gitcvs-migration.adoc (man gitcvs-migration or git help cvs-migration if git is installed).

The user discussion and development of Git take place on the Git mailing list -- everyone is welcome to post bug reports, feature requests, comments and patches to git@vger.kernel.org (read Documentation/SubmittingPatches for instructions on patch submission and Documentation/CodingGuidelines).

Those wishing to help with error message, usage and informational message string translations (localization l10) should see po/README.md (a po file is a Portable Object file that holds the translations).

To subscribe to the list, send an email to git+subscribe@vger.kernel.org (see https://subspace.kernel.org/subscribing.html for details). The mailing list archives are available at https://lore.kernel.org/git/, https://marc.info/?l=git and other archival sites.

Issues which are security relevant should be disclosed privately to the Git Security mailing list git-security@googlegroups.com.

The maintainer frequently sends the “What's cooking” reports that list the current status of various development topics to the mailing list. The discussion following them give a good reference for project status, development direction and remaining tasks.

The name “git” was given by Linus Torvalds when he wrote the very first version. He described the tool as “the stupid content tracker” and the name as (depending on your mood):

  • random three-letter combination that is pronounceable, and not actually used by any common UNIX command. The fact that it is a mispronunciation of “get” may or may not be relevant.
  • stupid. contemptible and despicable. simple. Take your pick from the dictionary of slang.
  • “global information tracker”: you're in a good mood, and it actually works for you. Angels sing, and a light suddenly fills the room.
  • “goddamn idiotic truckload of sh*t”: when it breaks