DirectFileStore export and writes are several times slower in 5.0.0 with long label values #341

Since 5.0.0 (#330), DirectFileStore encodes and decodes label sets with URI.encode_www_form / URI.decode_www_form. When label values are a few hundred characters long (SQL statements, for example), both are several times slower than the CGI based code they replaced. This shows up in two places.

Export. MetricStore#all_values decodes the key of every entry in every process's file. Each process's file usually holds the same keys, so the cost is multiplied by the number of processes. With the repro below (10 forked writers, 150 histogram label sets with values of 100–300 characters), a full Formats::Text.marshal on Ruby 3.3.13 takes 99 ms with 4.2.5 and 402 ms with 5.0.0. In a larger case (10 processes, ~5,200 samples, values up to ~3,900 characters), it takes 596 ms with 4.2.5 and 3,707 ms with 5.0.0. Upgrading to Ruby 4.0 recovers only a small part of this.

In our case, this made a Puma app's /metrics endpoint slow enough to set off our alerts on scrape duration after a few days of uptime.

Write path. store_key re-encodes the label set on every set / increment / get, and a histogram observe does this twice. One histogram observe plus one counter increment with a long label takes 21.5 µs with 4.2.5 and 167 µs with 5.0.0.

Repro

require 'prometheus/client'
require 'prometheus/client/data_stores/direct_file_store'
require 'prometheus/client/formats/text'
require 'prometheus/client/version'
require 'fileutils'

dir = '/tmp/dfs-repro'
FileUtils.rm_rf(dir)
Prometheus::Client.config.data_store = Prometheus::Client::DataStores::DirectFileStore.new(dir: dir)
registry = Prometheus::Client::Registry.new
hist = registry.histogram(:query_seconds, docstring: 'q', labels: %i[db statement])
statements = Array.new(150) do |i|
  "SELECT a, b, c FROM ks.t#{i} WHERE id = ? AND x IN (#{(['?'] * (10 + i % 40)).join(', ')})"
end

10.times do
  Process.wait(fork do
    statements.each { |s| [0.003, 0.03, 0.3, 3].each { |v| hist.observe(v, labels: { db: 'main', statement: s }) } }
    exit!(0)
  end)
end

registry.metrics.each(&:values) # warm up
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
5.times { Prometheus::Client::Formats::Text.marshal(registry) }
ms = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) * 1000 / 5
puts "ruby #{RUBY_VERSION}, prometheus-client #{Prometheus::Client::VERSION}: #{ms.round} ms per export"

Proposed fix

I'm opening a PR alongside this issue with two small changes. Neither changes the exposition output or adds a dependency:

1. Decode each distinct key once per all_values call instead of once per process file.
2. Cache encoded keys per label set on the write path.

Together, these bring export back to where 4.2.5 was (600 ms against 596 ms in the larger case) and make the write path faster than 4.2.5 (13.7 µs against 21.5 µs).

Optional follow-ups

Two more changes speed up export further. I've left them out of the PR so they can be discussed separately, but I have both ready and can open PRs if you're interested:

3. Decode keys with CGI.unescape from cgi/escape instead of URI.decode_www_form. I know Fix Ruby 4.0 compatibility: replace CGI with URI in DirectFileStore #330 moved away from CGI deliberately. However, cgi/escape is still part of Ruby 4.0 after the rest of CGI was removed from the default gems, and JRuby 9.4, 10.0 and 10.1 ship it too, so no dependency is needed.
   • To return the same result as URI.decode_www_form, the output must be forced to UTF-8 and scrubbed, because keys read back from the file are binary strings.
   • With that, it matches URI.decode_www_form over every Unicode codepoint, invalid UTF-8, and keys written by the pre-5.0.0 CGI::escape encoder.
4. Read each store file in one go. FileMappedDict#all_values and #populate_positions would read the used region of the file once and unpack entries at their offsets (unpack(offset:), Ruby 3.1+), instead of doing a seek and a read per entry.

On top of the PR, in the larger case:

The exposition output stays byte-identical with both.