[Dev-iL]

• closes: Move to stable ABI. #95

Every new CPython release currently needs a new set of version-specific wheels. This change builds regular CPython wheels once, as cp311-abi3 against the Python 3.11 stable ABI, as requested in #95 and prescribed in xopen#182 and python-isal#240. Free-threaded CPython and PyPy keep version-specific full-API wheels.

Result: compression, CRC32, bounded decompression, and gzip reads run at baseline speed. Unbounded decompression is slower, because the Limited API has no _PyBytes_Resize() and the result must be copied once from a native buffer into its bytes object. At 16 MiB + 137 bytes of output those calls take up to 19% longer, where 6 of the 12 cases are beyond the control's variation, and the worst case, raw deflate decompression of 1 MiB + 137 bytes of zeros, takes 2.744 times as long. Those calls also briefly hold one extra output-sized allocation.

Benchmark

The baseline is commit 39cec01, unchanged. The benchmark times installed wheels of the baseline and this candidate, both built with CPython 3.11.15, GCC 15.2.0, and the same static zlib-ng 2.3.3 archive. Each of nine randomized rounds times the baseline, a second baseline process as a control, and the candidate on identical inputs, measuring thread CPU time on CPU 1 of a Linux x86_64 Intel Core i7-3770 without AVX2. The 128 cases combine four inputs (FASTQ, standard-library Python source, seeded random bytes, and zeros), four output sizes (1 KiB, 64 KiB, 1 MiB + 137 bytes, and 16 MiB + 137 bytes), and eight operations. A case's ratio is the median over the rounds of candidate time divided by baseline time, and a ratio above 1 is slower. The control deviates from 1 by 3.830% at the 95th percentile (inclusive method) and by 9.626% at most, and a case counts as slower or faster only beyond that maximum. Desktop processes were running during timing; no build, test, or other benchmark was.

Operation	Call timed	Median ratio (16 cases)	Range of case ratios
Level-1 compression	compress(data, 1)	1.000×	0.948-1.050×
zlib decompression	decompress(data)	1.085×	0.915-2.270×
Raw deflate decompression	decompress(data, wbits=-15)	1.049×	0.936-2.744×
Streaming decompression	decompressobj().decompress(data) and flush()	1.071×	0.925-2.246×
Bounded decompression	_ZlibDecompressor(wbits=-15).decompress(data, 65536)	1.009×	0.991-1.094×
gzip read	gzip_ng.open(...).read(131072)	1.004×	0.979-1.039×
gzip readinto	readinto() into a 131072-byte bytearray	0.999×	0.966-1.053×
CRC32	crc32(data)	1.004×	0.983-1.047×

No case of compression, bounded decompression, the gzip reads, or CRC32 differs from the baseline by more than the control maximum. The benchmark report has the method, per-case results, raw samples, build hashes, and the reproduction command. Running python benchmark_scripts/benchmark_abi3.py --summarize benchmark_scripts/abi3_results/results.json --library zlib_ng regenerates the numbers above. Links to files in this branch resolve once it is pushed.

Worst-case regressions

Of the 48 unbounded decoder cases, 19 are slower beyond the control maximum:

Input and output size	zlib	Raw deflate	Streaming
Zeros, 1 MiB + 137 bytes	2.270×	2.744×	2.246×
Random, 1 MiB + 137 bytes	1.512×	1.787×	1.530×
Random, 64 KiB	1.214×	1.252×	1.172×
Zeros, 16 MiB + 137 bytes	1.152×	1.190×	1.162×
Random, 16 MiB + 137 bytes	1.151×	1.157×	1.144×
Python source, 1 MiB + 137 bytes	1.135×	1.133×	1.124×
Zeros, 1 KiB	1.135×	inside the control	inside the control

The zeros 1 KiB zlib case had a control ratio of its own of 0.943, so part of its distance may be noise. No case is faster beyond the control maximum; the nearest, FASTQ at 1 MiB + 137 bytes through the three unbounded decoders at 0.915-0.936 times the baseline time, were not profiled.

Memory and the copy

On stable ABI builds, compress(), decompress(), and compressobj() output, and decompressobj().decompress() and _ZlibDecompressor.decompress() without max_length, grow their output in a buffer managed with PyMem_Malloc() and PyMem_Realloc() and copy it into the result. Both exist while the copy is made, so peak memory rises by one output-sized allocation over the baseline's in-place resize. This follows from the code (finish_output()); it was not benchmarked.

Calls that allocate their whole limit at once return a filled bytes object directly: _ZlibDecompressor.decompress(data, max_length) with max_length up to 16 MiB, the gzip_ng file reader behind read(size) and readinto(), and decompressobj().decompress(data, max_length) with max_length up to 16 KiB. test_filled_bounded_decompressor_output_is_not_copied and test_filled_gzip_reader_chunk_is_not_copied check this with tracemalloc.

These bounded calls still copy in three situations:

• Output shorter than the allocation, such as a _ZlibDecompressor.decompress(data, max_length) call that returns less than max_length, is copied while the full allocation is still held. Peak memory is then the limit plus the output, where the original code shrank the allocation in place and needed only the limit.
• _ZlibDecompressor.decompress(data, max_length) with max_length above 16 MiB starts from a 16 MiB buffer, as the original code did, grows it natively, and copies the result.
• decompressobj().decompress(data, max_length) with max_length above 16 KiB starts from a 16 KiB buffer, grows it natively, and copies, so a call that fills its limit briefly holds about twice the limit.

The initial allocations are deliberately not raised to the limit, which would change memory use for callers that pass large limits and receive short results. The benchmark does not time these cases.

Repair check

The first version of this branch (6488941) was later repaired: the fixes under "Fixed on the way", plus 99b8c49 (review fix of the new preset-dictionary support), 9e2dc69 (codec allocator on full-API builds), d7b8e06 (_ParallelCompress docstring), cdf4f03 (restored test-static jobs), a557497 (Windows ARM64 wheel tests), and 7c5af43 (release-wheel audit). Timing the first candidate against the final one (936ab8f) with the same protocol gives operation medians of 0.993-1.005 and case ratios of 0.874-1.065, against a control 95th percentile of 4.262% and maximum of 8.183%. Only zeros 1 KiB raw deflate decompression, at 0.874 with a control ratio of 0.953, lies beyond the control maximum, so there is no measurable change. Both new runs are noisier than the first candidate's own comparison (2.884% and 5.314%); interpretation: machine state, not code.

Relationship to the maintainer's design

The design is to use the 3.11 Limited API, replace _PyBytes_Resize() with buffers managed by PyMem_Malloc() and PyMem_Resize(), create the result with PyBytes_FromStringAndSize() at the end, and accept the performance cost. This branch follows it, and the measured cost is above. Refinements and deviations:

• Filled bounded output is returned directly instead of copied, for the calls listed under "Memory and the copy".
• End-of-stream stop: output growth stops when zlib-ng reports the end of the stream, even if the buffer is exactly full. test_stream_end_at_full_output_does_not_grow shows the saved growth (about 16 KiB of traced peak for a 16384-byte stream). It applies to full-API builds too. In the prototype study below, it changes no timing measurably over the 24 large decoder cases (median 1.010 without it), except zeros 1 MiB + 137 bytes through decompressobj(), which is faster without the stop (0.886), unexplained. At 64 KiB, an exact buffer boundary, the original code without it takes 1.385-1.821 times as long for random input and stays inside the control for the other inputs.
• Codec state allocator: zlib-ng calls its allocation callbacks without an attached thread state, where PyMem_Malloc() is not allowed. PyMem_RawMalloc() is allowed but joins the stable ABI only in 3.13, so Limited-API builds use malloc() and free(), and tracemalloc does not see codec state there. Full-API builds keep PyMem_RawMalloc() and PyMem_RawFree().
• Free-threaded CPython keeps the full API, because a cp311 stable ABI wheel cannot load there; its locks, thread-state release, and no-GIL declarations are unchanged. The abi3t stable ABI of PEP 803 (Python 3.15 and later) could cover it later.
• PyPy keeps its existing static types to preserve existing behavior. Its 3.11.17 headers do define Py_TPFLAGS_IMMUTABLETYPE and Py_TPFLAGS_DISALLOW_INSTANTIATION; heap types are used on CPython only.
• CPython types are heap types from specifications; compressobj() and decompressobj() take their types from module state with traverse and clear hooks. PyObject_New() and FASTCALL are kept. The process-global zlib_ng.error is unchanged, so this adds no subinterpreter isolation.
• requires-python rises from >=3.9 to >=3.11, the support floor already stated in the 1.1.0 changelog.

Validation

• One cp311 stable ABI wheel, built with Py_LIMITED_API=0x030B0000, passes abi3audit --strict with no violations and a computed baseline of 3.11. The same wheel installs on CPython 3.11.15, 3.12.14, 3.13.12, 3.14.4, 3.15.0a8 (alpha), and 3.15.0; smoke checks against standard-library oracles pass on all six. On each, tests/test_stable_abi.py gives 52 passed and 10 skipped (no _testcapi), or 60 passed and 2 skipped on 3.14.4. The full suite was not run on the other five versions.
• Full test suite on the installed wheels, each including the standard-library compliance tests:
  • CPython 3.14.4 with PYTHONDEVMODE=1: 14,625 passed and 4 skipped (compliance alone: 138 passed and 2 skipped).
  • Free-threaded CPython 3.13.11 (uv build) with PYTHON_GIL unset, the GIL staying off after import: 14,619 passed and 10 skipped. Eight skips need _testcapi, which that build lacks, one is a 4 GiB bigmem test, and one needs sys.gettotalrefcount.
  • PyPy 3.11.17 (PyPy 8.0.1): 14,597 passed and 32 skipped.
• test_full_api_codec_state_uses_the_raw_python_allocator skips on the stable ABI wheel by design, because codec state uses malloc() there.
• flake8, mypy (17 files), actionlint, git diff --check, strict Sphinx, and twine check pass.
• New tests cover type lifetimes and ownership, immutable and factory-only types, output boundaries and direct return, allocation-failure cleanup and the release of output after failed or invalid decodes, module initialization failure, and that importing on a free-threaded build leaves the GIL disabled.
• CI is configured to build and audit one cp311-abi3 wheel on Linux, macOS Intel, macOS ARM, Windows x64, and Windows ARM64, and to test that same wheel on CPython 3.11-3.15; the release job audits the wheels it uploads. Windows 3.14t builds from source to exercise its free-threading macro.

Limits: timings and tests come from one Linux x86_64 machine. Not run:

• No CI has run for this branch.
• The macOS, Windows, and ARM64 CI jobs (including the new windows-11-arm jobs and Windows 3.14t), the release job's abi3audit step, and the AddressSanitizer and ThreadSanitizer jobs.

Fixed on the way

Each fix is in its own commit. Unless noted, the problem predates the migration (present in 39cec01):

• c820004: crc32_combine(), _ParallelCompress(), _GzipReader.seek(), and _GzipReader.read() raised SystemError instead of TypeError for bad arguments, because PyArg_Parse* results were compared with < 0.
• a7dca65, 0bb3996: _GzipReader and _ParallelCompress carried isal_zlib type names.
• d94cb5c: _ZlibDecompressor lost unconsumed input beyond 4 GiB when it hit its output limit.
• 8f6408f: decompressobj().decompress() lost unread input beyond the first 4 GiB of one call when the output limit or the end of the stream was reached.
• f603f0c: _ZlibDecompressor raised SystemError for a stream that needs a preset dictionary, whether or not zdict was given. 99b8c49 is a review fix of this new support: with max_length=0 the applied dictionary was reported as an error and the input discarded. That defect never existed outside this branch.
• 936ab8f: _GzipReader.readinto() wrote into read-only buffers and crashed on a read-only mmap; it now raises TypeError.
• 1954f7d: _GzipReader turned any error from looking up read into an unrelated TypeError.
• 7c540b9: module initialization ignored failures and leaked the module and the gzip module on error exits. One part came with the migration: a pending exception between the two type creations, introduced in 6488941.
• e692065: the PyPy Py_UNREACHABLE fallback redefined PyPy's own macro.
• e2ccb50, 758e880, 925524a: the sanitizer job and the gzip_ng.open docstring named python-isal and ISA-L, and the license package-data entry named a file that does not exist.
• cd6aedb, partly pre-existing: the cibuildwheel skips for cp38 on macOS ARM64 and Windows ARM64 were already stale under the baseline's >=3.9, while the cp39 and cp310 skips became stale with this branch's >=3.11. The same commit adds tests/test_stable_abi.py to the cibuildwheel tests.

Debt left

• CI paths-ignore skips the whole workflow, including the docs build, for pull requests that only touch docs/ or .rst files. That is the existing CI-cost policy, left as is.
• Importing zlib_ng in a subinterpreter replaces the process-wide zlib_ng.error class. That is existing module policy and out of scope here.
• Free-threaded stable ABI wheels (abi3t) are a follow-up.
• The bounded-call copies listed under "Memory and the copy" are by design.
• No CI has run for this branch: the remote jobs (macOS, Windows, and ARM64, including the new windows-11-arm jobs and the release abi3audit step) and the ASan and TSan jobs still need to run.

Earlier attempts and considered approaches

Alternative output strategies: prototype benchmark and reasons for rejecting them

A separate prototype study, run against the original code, compared four ways of growing unbounded output. It is not the final-wheel comparison above, and its ratios must not be compared with it: it used prototype binaries, a different reference, and a Python source input repeated to size with a 29,521-byte period, shorter than deflate's 32 KiB window. Every variant adds the end-of-stream stop to the original code and is timed over nine repeats against the original code with the stop alone (baseline-eof). The decoders are decompress(), _ZlibDecompressor(wbits=-15).decompress() without a limit, and decompressobj().decompress(). The control (eof-control) has an inclusive 95th percentile of 8.195% and a maximum of 8.739% over all 72 cases. Large means 1 MiB + 137 and 16 MiB + 137 bytes; small means 1 KiB and 64 KiB.

Strategy (variant ID)	Large decoder cases (24): median	Range	Small decoder cases (24): median	Range
Native buffer with a final copy, selected (buffers-eof)	1.181×	0.888-1.911×	1.016×	0.821-1.519×
Joined segments (segmented-eof)	0.998×	0.392-1.651×	1.107×	0.978-1.723×
Eager BytesIO (bytesio-eof)	1.373×	1.049-1.880×	1.703×	1.006-4.752×
Delayed BytesIO, created on first growth (bytesio-hybrid-eof)	1.368×	1.139-1.921×	1.057×	0.887-2.009×

The original code without the stop (baseline) has a median of 1.010 over the large cases, inside the control.

• Joined segments are faster than the native buffer on large outputs (a median of 0.998 against 1.181, a gap wider than the control, and faster in 16 of 24 cases) and slower on small ones (1.107 against 1.016). On large-output speed, the evidence does not favour the native buffer. The repeated Python input cuts both ways: it gives segments their slowest case (1.651) and some of their largest wins (0.878-0.980 against 1.506-1.623 at 1 MiB + 137 bytes). Without that input, segments lead by more, with a median of 0.964 against 1.155 and faster in 13 of 18 cases. The native buffer was kept because it is the maintainer's prescribed design and needs less machinery (no segment list and no join pass); that this is simpler to maintain is a judgment, not a measurement. Joined segments would be a defensible alternative for a later change.
• Delayed BytesIO beats the native buffer at 1 MiB + 137 bytes in 9 of 12 decoder cases but loses all 12 at 16 MiB + 137 bytes. Eager BytesIO reaches 4.752 times the reference on small outputs. Both add Python-level calls and writable buffer exports, so neither was adopted.

The per-case results, per-decoder medians, raw repeats, and study settings are committed; the prototype harness, binaries, and inputs are not.

Checklist

• Pull request details were added to CHANGELOG.rst
• Documentation was updated (if needed)

🤖 Generated with Claude Code