| Git performance tests | |
| ===================== | |
| This directory holds performance testing scripts for git tools. The | |
| first part of this document describes the various ways in which you | |
| can run them. | |
| When fixing the tools or adding enhancements, you are strongly | |
| encouraged to add tests in this directory to cover what you are | |
| trying to fix or enhance. The later part of this short document | |
| describes how your test scripts should be organized. | |
| Running Tests | |
| ------------- | |
| The easiest way to run tests is to say "make". This runs all | |
| the tests on the current git repository. | |
| === Running 2 tests in this tree === | |
| [...] | |
| Test this tree | |
| --------------------------------------------------------- | |
| 0001.1: rev-list --all 0.54(0.51+0.02) | |
| 0001.2: rev-list --all --objects 6.14(5.99+0.11) | |
| 7810.1: grep worktree, cheap regex 0.16(0.16+0.35) | |
| 7810.2: grep worktree, expensive regex 7.90(29.75+0.37) | |
| 7810.3: grep --cached, cheap regex 3.07(3.02+0.25) | |
| 7810.4: grep --cached, expensive regex 9.39(30.57+0.24) | |
| Output format is in seconds "Elapsed(User + System)" | |
| You can compare multiple repositories and even git revisions with the | |
| 'run' script: | |
| $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh | |
| where . stands for the current git tree. The full invocation is | |
| ./run [<revision|directory>...] [--] [<test-script>...] | |
| A '.' argument is implied if you do not pass any other | |
| revisions/directories. | |
| You can also manually test this or another git build tree, and then | |
| call the aggregation script to summarize the results: | |
| $ ./p0001-rev-list.sh | |
| [...] | |
| $ ./run /path/to/other/git -- ./p0001-rev-list.sh | |
| [...] | |
| $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh | |
| aggregate.perl has the same invocation as 'run', it just does not run | |
| anything beforehand. | |
| You can set the following variables (also in your config.mak): | |
| GIT_PERF_REPEAT_COUNT | |
| Number of times a test should be repeated for best-of-N | |
| measurements. Defaults to 3. | |
| GIT_PERF_MAKE_OPTS | |
| Options to use when automatically building a git tree for | |
| performance testing. E.g., -j6 would be useful. Passed | |
| directly to make as "make $GIT_PERF_MAKE_OPTS". | |
| GIT_PERF_MAKE_COMMAND | |
| An arbitrary command that'll be run in place of the make | |
| command, if set the GIT_PERF_MAKE_OPTS variable is | |
| ignored. Useful in cases where source tree changes might | |
| require issuing a different make command to different | |
| revisions. | |
| This can be (ab)used to monkeypatch or otherwise change the | |
| tree about to be built. Note that the build directory can be | |
| re-used for subsequent runs so the make command might get | |
| executed multiple times on the same tree, but don't count on | |
| any of that, that's an implementation detail that might change | |
| in the future. | |
| GIT_PERF_REPO | |
| GIT_PERF_LARGE_REPO | |
| Repositories to copy for the performance tests. The normal | |
| repo should be at least git.git size. The large repo should | |
| probably be about linux.git size for optimal results. | |
| Both default to the git.git you are running from. | |
| GIT_PERF_EXTRA | |
| Boolean to enable additional tests. Most test scripts are | |
| written to detect regressions between two versions of Git, and | |
| the output will compare timings for individual tests between | |
| those versions. Some scripts have additional tests which are not | |
| run by default, that show patterns within a single version of | |
| Git (e.g., performance of index-pack as the number of threads | |
| changes). These can be enabled with GIT_PERF_EXTRA. | |
| GIT_PERF_USE_SCALAR | |
| Boolean indicating whether to register test repo(s) with Scalar | |
| before executing tests. | |
| You can also pass the options taken by ordinary git tests; the most | |
| useful one is: | |
| --root=<directory>:: | |
| Create "trash" directories used to store all temporary data during | |
| testing under <directory>, instead of the t/ directory. | |
| Using this option with a RAM-based filesystem (such as tmpfs) | |
| can massively speed up the test suite. | |
| Naming Tests | |
| ------------ | |
| The performance test files are named as: | |
| pNNNN-commandname-details.sh | |
| where N is a decimal digit. The same conventions for choosing NNNN as | |
| for normal tests apply. | |
| Writing Tests | |
| ------------- | |
| The perf script starts much like a normal test script, except it | |
| sources perf-lib.sh: | |
| #!/bin/sh | |
| # | |
| # Copyright (c) 2005 Junio C Hamano | |
| # | |
| test_description='xxx performance test' | |
| . ./perf-lib.sh | |
| After that you will want to use some of the following: | |
| test_perf_fresh_repo # sets up an empty repository | |
| test_perf_default_repo # sets up a "normal" repository | |
| test_perf_large_repo # sets up a "large" repository | |
| test_perf_default_repo sub # ditto, in a subdir "sub" | |
| test_checkout_worktree # if you need the worktree too | |
| At least one of the first two is required! | |
| You can use test_expect_success as usual. In both test_expect_success | |
| and in test_perf, running "git" points to the version that is being | |
| perf-tested. The $MODERN_GIT variable points to the git wrapper for the | |
| currently checked-out version (i.e., the one that matches the t/perf | |
| scripts you are running). This is useful if your setup uses commands | |
| that only work with newer versions of git than what you might want to | |
| test (but obviously your new commands must still create a state that can | |
| be used by the older version of git you are testing). | |
| For actual performance tests, use | |
| test_perf 'descriptive string' ' | |
| command1 && | |
| command2 | |
| ' | |
| test_perf spawns a subshell, for lack of better options. This means | |
| that | |
| * you _must_ export all variables that you need in the subshell | |
| * you _must_ flag all variables that you want to persist from the | |
| subshell with 'test_export': | |
| test_perf 'descriptive string' ' | |
| foo=$(git rev-parse HEAD) && | |
| test_export foo | |
| ' | |
| The so-exported variables are automatically marked for export in the | |
| shell executing the perf test. For your convenience, test_export is | |
| the same as export in the main shell. | |
| This feature relies on a bit of magic using 'set' and 'source'. | |
| While we have tried to make sure that it can cope with embedded | |
| whitespace and other special characters, it will not work with | |
| multi-line data. | |
| Rather than tracking the performance by run-time as `test_perf` does, you | |
| may also track output size by using `test_size`. The stdout of the | |
| function should be a single numeric value, which will be captured and | |
| shown in the aggregated output. For example: | |
| test_perf 'time foo' ' | |
| ./foo >foo.out | |
| ' | |
| test_size 'output size' | |
| wc -c <foo.out | |
| ' | |
| might produce output like: | |
| Test origin HEAD | |
| ------------------------------------------------------------- | |
| 1234.1 time foo 0.37(0.79+0.02) 0.26(0.51+0.02) -29.7% | |
| 1234.2 output size 4.3M 3.6M -14.7% | |
| The item being measured (and its units) is up to the test; the context | |
| and the test title should make it clear to the user whether bigger or | |
| smaller numbers are better. Unlike test_perf, the test code will only be | |
| run once, since output sizes tend to be more deterministic than timings. |