[sferik]

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against master and the branch side by side, in alternating worker processes, and fails the build if the branch is slower or allocates more. It judges CPU time and instruction counts with confidence intervals, and counts allocations exactly. When something regresses, it profiles both versions with StackProf and says where the extra time went.

There are 36 benchmarks, and I tried to follow what benchmark/parser.rb, benchmark/encoder.rb and the performance PRs of the last year or so already care about. Parsing covers twitter.json (plain, with symbolize_names and with freeze), citm_catalog.json, canada.json and activitypub.json from benchmark/data, plus small payloads where the per-call overhead dominates: the small hash and the nested document from oj's tests, and a 1 KB API response through JSON.parse, with symbolize_names, and through JSON::Coder. Then the paths on their own: 1,000 integers, 1,000 short decimals, strings with escapes, long ASCII strings for the SIMD scan, and a stream of documents fed to ResumableParser in 4 KB chunks. Generating covers the same documents, twitter.json pretty and not, 10,000 points from canada.json for float formatting, and the 1 KB response through JSON.generate, pretty_generate, Hash#to_json, JSON::Coder and JSON.dump into an IO. It also covers integers, decimals, strings that need escaping, long strings that are ASCII but for one character or entirely multibyte, script_safe, ascii_only, and objects that go through their own #to_json or #to_s, as in the encoder benchmark's "many #to_json calls" (with a fixed Time and Date instead of Time.now). The fixtures are read from benchmark/data, so nothing is duplicated.

The scripts in benchmark/ are good at comparing json with Oj and RapidJSON, but someone has to run them and read the numbers, and nothing runs them in CI. Duel compares each pull request with master instead, and fails it when json itself gets slower, even by a percent or two of instructions.

The tricky part was json being json. Duel's worker process requires json for its own protocol before it loads the version it measures, and under the bundle that require finds the candidate's lib/ through the gemspec. So the baseline's workers were loading the candidate's parser.bundle and generator.bundle first, and then the baseline's on top of them. I checked this with lsof in each worker, and both sets of extensions were mapped in every baseline worker. I fixed that in Duel 0.1.4, whose workers load nothing a version could be before the version itself, and which now fails a run where a worker loads an extension twice. So .duel.rb needs no workaround. It sets c.build to bundle exec rake compile, so each version is measured with its own build. The workflow also runs rake compile once before Duel starts, because the duel command loads json itself (to read the pull request's labels and write its report), and here the bundle finds this checkout's. That json never runs any of the benchmarks, but its extensions have to exist.

To check that it actually catches things, I planted three regressions, one at a time. An extra rb_str_dup in build_string in parser.c failed 12 of the 16 parsing benchmarks, on allocations every time and on instructions and CPU time for most (1.04x to 1.34x). The four that passed were the ones that never reach that line: freeze interns its strings, the integer and decimal arrays have no strings, and escaped strings take the unescape path. Switching the generator's escape search from the NEON scan back to the scalar loop failed 15 of the 20 generating benchmarks, all on instructions and all but one on CPU time, from 1.02x the instructions for the small array up to 5.3x for the 4 KB strings. Its profiles pointed at State.generate. A Ruby-level change, an extra options.transform_keys(&:to_sym) in JSON.parse, failed exactly the 14 benchmarks that go through JSON.parse (one more allocation each). The profile named Hash#transform_keys, and the JSON::Coder and ResumableParser benchmarks, which skip that method, passed. Comparing master with itself passed three times in a row.

A full run takes about 7 seconds of measuring on my laptop, plus compiling both versions, so I'd expect the job to spend most of its time on setup.