)]}'
{
  "commit": "2d86a962203fe0faf6ae441cbe21b92a4bb03c20",
  "tree": "e91079634e80deff67ac3c28bd5ca759558b149c",
  "parents": [
    "48bf2fa8bad054d66bd79c6ba903c89c704201f7"
  ],
  "author": {
    "name": "Jeff King",
    "email": "peff@peff.net",
    "time": "Thu May 13 02:25:53 2021 -0400"
  },
  "committer": {
    "name": "Junio C Hamano",
    "email": "gitster@pobox.com",
    "time": "Thu May 13 15:50:44 2021 +0900"
  },
  "message": "t: avoid sed-based chain-linting in some expensive cases\n\nCommit 878f988350 (t/test-lib: teach --chain-lint to detect broken\n\u0026\u0026-chains in subshells, 2018-07-11) introduced additional chain-lint\ntests which add an extra \"sed\" pipeline to each test we run. This has a\nmeasurable impact on runtime. Here are timings with and without a new\nenvironment variable (added by this patch) that lets you disable just\nthe additional sed-based chain-lint tests:\n\n  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER\u003d1 make test\n    Time (mean ± σ):     64.202 s ±  1.030 s    [User: 622.469 s, System: 301.402 s]\n    Range (min … max):   61.571 s … 65.662 s    10 runs\n\n  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER\u003d0 make test\n    Time (mean ± σ):     57.591 s ±  0.333 s    [User: 529.368 s, System: 270.618 s]\n    Range (min … max):   57.143 s … 58.309 s    10 runs\n\n  Summary\n    \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d0 make test\u0027 ran\n      1.11 ± 0.02 times faster than \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d1 make test\u0027\n\nOf course those extra lint checks are doing something useful, so paying\na few extra seconds (at least on Linux) isn\u0027t so bad (though note the\nCPU time; we\u0027re bounded in our parallel run here by the slowest test, so\nit really is ~120s of CPU improvement).\n\nBut we can observe that there are some test scripts where they produce a\nmuch stronger effect, and provide less value. In t0027 and t3070 we run\na very large number of small tests, all driven by a series of\nfunctions/loops which are filling in the test bodies. There we get much\nless bang for our buck in terms of bug-finding versus CPU cost.\n\nThis patch introduces a mechanism for controlling when those extra\nlint checks are run, at two levels:\n\n  - a user can ask to disable or to force-enable the checks by setting\n    GIT_TEST_CHAIN_LINT_HARDER\n\n  - if the user hasn\u0027t specified a preference, individual scripts can\n    disable the checks by setting GIT_TEST_CHAIN_LINT_HARDER_DEFAULT;\n    scripts which don\u0027t set that get the current behavior of enabling\n    them.\n\nIn addition, this patch flips the default for t0027 and t3070\u0027s\nmass-generated sections to disable the extra checks. Here are the timing\nresults for t0027:\n\n  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER\u003d1 ./t0027-auto-crlf.sh\n    Time (mean ± σ):     17.078 s ±  0.848 s    [User: 14.878 s, System: 7.075 s]\n    Range (min … max):   15.952 s … 18.421 s    10 runs\n\n  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER\u003d0 ./t0027-auto-crlf.sh\n    Time (mean ± σ):      9.063 s ±  0.759 s    [User: 7.890 s, System: 3.362 s]\n    Range (min … max):    7.747 s … 10.619 s    10 runs\n\n  Benchmark #3: ./t0027-auto-crlf.sh\n    Time (mean ± σ):      9.186 s ±  0.881 s    [User: 7.957 s, System: 3.427 s]\n    Range (min … max):    7.796 s … 10.498 s    10 runs\n\n  Summary\n    \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d0 ./t0027-auto-crlf.sh\u0027 ran\n      1.01 ± 0.13 times faster than \u0027./t0027-auto-crlf.sh\u0027\n      1.88 ± 0.18 times faster than \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d1 ./t0027-auto-crlf.sh\u0027\n\nWe can see that disabling the checks for the whole script buys us an\nalmost 2x speedup. But the new default behavior, disabling them only for\nthe mass-generated part, gets us most of that speedup (but still leaves\nthe checks on for further manual tests people might write).\n\n  As a side note, I\u0027d caution about comparing runtimes and CPU seconds\n  between this timing and the earlier \"make test\" one. In \"make test\",\n  we\u0027re running a lot of scripts in parallel, so the CPU is throttling\n  down (and thus a CPU second saved here would count for more during a\n  parallel run; the same work takes more CPU seconds there).\n\nWe get similar results for t3070:\n\n  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER\u003d1 ./t3070-wildmatch.sh\n    Time (mean ± σ):     20.054 s ±  3.967 s    [User: 16.003 s, System: 8.286 s]\n    Range (min … max):   11.891 s … 23.671 s    10 runs\n\n  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER\u003d0 ./t3070-wildmatch.sh\n    Time (mean ± σ):     12.399 s ±  2.256 s    [User: 7.542 s, System: 5.342 s]\n    Range (min … max):    9.606 s … 15.727 s    10 runs\n\n  Benchmark #3: ./t3070-wildmatch.sh\n    Time (mean ± σ):     10.726 s ±  3.476 s    [User: 6.790 s, System: 4.365 s]\n    Range (min … max):    5.444 s … 15.376 s    10 runs\n\n  Summary\n    \u0027./t3070-wildmatch.sh\u0027 ran\n      1.16 ± 0.43 times faster than \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d0 ./t3070-wildmatch.sh\u0027\n      1.87 ± 0.71 times faster than \u0027GIT_TEST_CHAIN_LINT_HARDER\u003d1 ./t3070-wildmatch.sh\u0027\n\nAgain, we get almost a 2x speedup disabling these. In this case, there\nare no tests not covered by the script\u0027s \"default to disable\" behavior,\nso the second two benchmarks should be the same (and while they do\ndiffer, you can see the variance is quite high but they\u0027re within one\nstandard deviation).\n\nSo it seems like for these two scripts, at least, disabling the extra\nchecks is a reasonable tradeoff. Sadly, the overall runtime of \"make\ntest\" on my system doesn\u0027t get much faster. But that\u0027s because we\u0027re\nmostly limited by the cost of the single biggest test. Here are the\ntop-5 tests by wall-clock time from a parallel run, before my patch:\n\n  57.9192368984222 t9001-send-email.sh\n  45.6329638957977 t0027-auto-crlf.sh\n  32.5278220176697 t3070-wildmatch.sh\n  22.2701289653778 t7610-mergetool.sh\n  20.8635759353638 t1701-racy-split-index.sh\n\nAnd after:\n\n  57.1476998329163 t9001-send-email.sh\n  33.776211977005 t0027-auto-crlf.sh\n  21.3116669654846 t7610-mergetool.sh\n  20.7748689651489 t1701-racy-split-index.sh\n  19.6957249641418 t7112-reset-submodule.sh\n\nWe dropped 12s from t0027, and t3070 dropped off our list entirely at\naround 16s. In both cases we\u0027re bound by t9001, but its slowness is\ndue to the actual tests, so we\u0027ll have to deal with it in a different\nway. But this reduces overall CPU, and means that dealing with t9001 (by\nimproving the speed of send-email or splitting it apart) will let us\nreduce our overall runtime even on multi-core machines.\n\nSigned-off-by: Jeff King \u003cpeff@peff.net\u003e\nSigned-off-by: Junio C Hamano \u003cgitster@pobox.com\u003e\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "593d4a4e270cc07b0f8e7d93e0da4d3c0b73eebd",
      "old_mode": 33188,
      "old_path": "t/README",
      "new_id": "63f9e07605d5c1a0935e21c91a08efaf3376da36",
      "new_mode": 33188,
      "new_path": "t/README"
    },
    {
      "type": "modify",
      "old_id": "d24d5acfbc7947c97250da98d50b8612ef0da459",
      "old_mode": 33261,
      "old_path": "t/t0027-auto-crlf.sh",
      "new_id": "4a5c5c602cfa8f53ecb9e329a20e67a008fb0e00",
      "new_mode": 33261,
      "new_path": "t/t0027-auto-crlf.sh"
    },
    {
      "type": "modify",
      "old_id": "891d4d7cb9f1a7d1a8caa75f30e6d4eb4ea0df14",
      "old_mode": 33261,
      "old_path": "t/t3070-wildmatch.sh",
      "new_id": "56ea4bda133d7443ba0d67c082d157cd491bd9cf",
      "new_mode": 33261,
      "new_path": "t/t3070-wildmatch.sh"
    },
    {
      "type": "modify",
      "old_id": "d3f6af6a65451cdd7868e5dc355b44fed49a0670",
      "old_mode": 33188,
      "old_path": "t/test-lib.sh",
      "new_id": "fac144adcec7a15da24f03014e31d4d3354473d2",
      "new_mode": 33188,
      "new_path": "t/test-lib.sh"
    }
  ]
}