[Dev-iL]

• closes: Priority for next release: Move to stable ABI!!! #240

Regular CPython wheels currently have to be rebuilt and re-released for every new Python version. This PR builds them against the Python 3.11 stable ABI instead (cp311-abi3), so one wheel serves CPython 3.11 and every later release, as requested in python-isal#240. Free-threaded CPython and PyPy keep their full-API, version-specific wheels.

The cost is an extra copy when decompression returns its whole output at once: on the measured machine those calls take up to 23% longer at 16 MiB + 137 bytes of output, where 6 of the 12 cases are beyond the control's variation, and up to 2.160 times as long for 1 MiB + 137 bytes of incompressible raw deflate data. Compression, CRC-32, gzip file reads, and bounded IgzipDecompressor output show no measurable difference apart from three outliers listed below.

How output is returned

_PyBytes_Resize is not in the Limited API (the C API subset that builds against the stable ABI). Output that grows during a call is therefore kept in a PyMem buffer and copied into a bytes object with PyBytes_FromStringAndSize when the call finishes, following the design that maintainer rhpvorderman set out in a comment on xopen#182. The benchmark report calls this native output. CPython types become heap types created from specs, and private or layout-dependent APIs are replaced with public ones.

These calls avoid the copy, because their output goes straight into a Python-owned object; output shorter than max_length or the requested size is still copied:

• igzip_lib.IgzipDecompressor().decompress(data, max_length) when max_length is at most 16 MiB and the output fills it.
• isal_zlib.decompressobj().decompress(data, max_length) when max_length is at most 16 KiB and the output fills it.
• read() and readinto() on igzip.open() files, which read through _GzipReader.readinto().

isal_zlib.decompressobj().decompress(data, max_length) with max_length above 16 KiB still starts from a 16 KiB allocation, grows it as native output and copies it once, so it holds at least twice the output in memory during the copy, where the original wheel resized one bytes object. The first allocation is deliberately not raised to max_length, so that a large limit does not allocate memory that a shorter output never uses. The same applies to IgzipDecompressor with max_length above 16 MiB. The benchmark below does not time either case.

When IgzipDecompressor().decompress() returns less than a max_length of at most 16 MiB, or decompressobj().decompress() less than a max_length of at most 16 KiB, the copy is made while the max_length-sized allocation is still held. Peak memory is then the limit plus the output, where the original wheel shrank the allocation in place and needed only the limit: for 12 MiB of output under a 16 MiB limit, 28 MiB against 16 MiB. The random 1 KiB bounded case below shows the time cost of such a copy for a small output.

Final-wheel benchmark

The baseline is commit 2498167, unchanged. The benchmark times installed wheels over nine randomized paired rounds of 128 cases. Each round times the original baseline, the baseline again as a control, and the candidate on identical inputs, with iteration counts calibrated on the baseline and then frozen. Both wheels use CPython 3.11.15, GCC 15.2.0 with -O3 -DNDEBUG, and ISA-L 2.32.1. The worker runs pinned to CPU 1 of a Linux x86_64 Intel i7-3770 (no AVX2). The inputs are four corpora (the FASTQ fixture, stdlib Python source, seeded random bytes, and zeros) cut to 1 KiB, 64 KiB, 1 MiB + 137, and 16 MiB + 137 bytes. Each ratio is the median of nine paired ratios of candidate CPU time to baseline CPU time; above 1 means slower. The control deviates from 1 by 4.345% at p95 (inclusive percentile) and 9.116% at most, and a case counts as slower or faster only beyond that maximum. No build, test, or other benchmark ran during the timings; desktop processes did.

16 MiB + 137 bytes	FASTQ	Python source	Random	Zeros
isal_zlib.compress(data, 1)	1.021×	1.005×	0.943×	1.005×
isal_zlib.decompress()	1.060×	1.076×	1.193×	1.153×
igzip_lib.decompress(), raw deflate	1.070×	1.085×	1.231×	1.195×
decompressobj(), decompress and flush	1.052×	1.125×	1.208×	1.087×
IgzipDecompressor, 64 KiB max_length	1.047×	1.029×	1.036×	1.122×
igzip.open().read(131072)	1.007×	1.003×	0.973×	1.006×
igzip.open().readinto(), 128 KiB	0.996×	1.005×	1.039×	0.997×
isal_zlib.crc32()	0.998×	1.014×	1.017×	1.014×

Worst cases: random 1 MiB + 137 raw decoding at 2.160× (isal_zlib.decompress() 1.504×, decompressobj() 1.519× for the same input) and zeros 1 MiB + 137 raw decoding at 1.636×. Outside whole-output decoding, three cases exceed the control: random 1 KiB bounded decoding at 1.151× (plausibly the copy of output shorter than its 64 KiB limit, not profiled), zeros 16 MiB + 137 bounded decoding at 1.122× (not copied; the earlier candidate's run measured 0.995×, so run-to-run variation is the likely cause), and Python 1 KiB readinto() at 1.115× (unexplained). Three FASTQ cases are faster: 1 MiB + 137 streaming at 0.859× and decompress() at 0.896×, and 1 KiB decompress() at 0.890×, whose own control was 0.909. Their cause was not profiled.

A repair check timed the earlier candidate (65d3eb3) as baseline and control against this one with the same protocol. Per-operation medians range 0.992-1.006, so the repairs since 65d3eb3 show no measurable effect at the operation level. The control was 4.175% at p95 and 6.836% at most; five individual cases exceed that maximum in both directions, four of them at 16 MiB + 137 bytes.

The benchmark report has every summary, outlier, and limit. cpython311.json keeps every raw run with input hashes, iteration counts, and extension paths; cpython311.csv has every case; build-metadata.json has the wheel and source hashes and the build recipe; repair-check.json has the repair check. python benchmark_scripts/benchmark_abi3.py --library isal --summarize <file> prints the summaries. These links resolve once the branch is pushed.

Relationship to the maintainer's design

• Kept: the Python 3.11 ABI floor, PyMem growth for output with a final PyBytes_FromStringAndSize, and measurement of the cost.
• Refinement, bytes-owned bounded output: when the first allocation already equals a finite limit, a bytes object owns the output and a filled result is returned without a copy; output shorter than the limit is still copied, at the memory cost described above. The bounded study below measured the copy this avoids. The first allocation is never enlarged to reach this path.
• Refinement, end-of-stream stop: decompression stops growing its buffer once ISA-L reports the end of the stream, instead of growing it once more when the output exactly filled it. Full-API builds get the same change.
• Free-threaded CPython: the cp311-abi3 floor cannot cover free-threaded builds, so they keep the full API and bytes resizing. PEP 803 ("abi3t", Python 3.15 and later) is a possible follow-up.
• PyPy: keeps the full API and its existing static types, preserving existing behaviour; its wheels are version-specific, so heap types would bring no ABI benefit. PyPy 3.11.17's headers define Py_TPFLAGS_IMMUTABLETYPE and Py_TPFLAGS_DISALLOW_INSTANTIATION, and PyPy honours the latter on these static types.
• Instance allocation: PyObject_New, which the Limited API provides, paired with PyObject_Free. The design comment did not prescribe this detail.
• Scope: the modules keep single-phase initialization and shared exception objects, and the Compress and Decompress type references live for the process. No per-interpreter module isolation is added.

Validation

All runtime checks install wheels built from this branch outside the source tree, on Linux x86_64 only; validation.txt has the details.

• All three extensions build with -DPy_LIMITED_API=0x030B0000. abi3audit --strict on the cp311-abi3 wheel: 0 violations, computed baseline 3.11. Requires-Python is >=3.11 in pyproject.toml and the sdist and wheel metadata.
• Full suite with the cp311-abi3 wheel on system CPython 3.14.4 with PYTHONDEVMODE=1: 9,832 passed, 11 skipped (platform-, size-, and build-specific tests), including the stdlib compliance tests.
• The same wheel on CPython 3.11.15, 3.12.14, 3.13.12, 3.15.0a8 (alpha), and 3.15.0: stdlib-oracle smoke checks pass, and test_stable_abi.py gives 54 passed, 10 skipped on each (no _testcapi). The full suite was not run on these versions.
• Free-threaded CPython 3.13.11t, cp313-cp313t wheel, PYTHON_GIL unset: 9,813 passed, 30 skipped (14 need _testcapi, which this build lacks, including the 9 allocation-failure tests). The GIL stays disabled after importing all three C modules.
• PyPy 3.11.17 (PyPy 8.0.1), pp311-pypy311_pp80 wheel: 9,797 passed, 46 skipped (tests that need _testcapi, tracemalloc, or reference counts).
• The nine allocation-failure sweeps pass on CPython 3.14.4, within test_stable_abi.py (63 passed, 1 skipped).
• flake8, mypy (31 files), strict Sphinx build, twine check, and git diff --check pass. actionlint's only finding is the unchanged codecov/codecov-action@v1 line listed under Debt left.

Fixed on the way

These problems predate the port; each is fixed in its own commit:

• 56112c7: _GzipReader.readall() leaked its first chunk when PyList_New() failed.
• 9259f2f: IgzipDecompressor and isal_zlib.decompressobj() lost unread input beyond the first 4 GiB of one call when the output limit or the end of the stream was reached.
• 4b21b60: _ParallelCompress() raised MemoryError instead of ValueError for a negative buffer size.
• 3043f24: _GzipReader() turned any error from looking up read, such as MemoryError, into "a bytes-like object is required".
• 1cdbee1: when IGzipFile() rejected a compression level, gzip.GzipFile.__del__ reported an unraisable AttributeError.
• b618b4a: PyPy builds warned that Py_UNREACHABLE was redefined.
• 7c9606e: the CI step that installs Python 3.11 was named "Set up Python 3.10".

Debt left

• Free-threaded stable ABI wheels (abi3t, PEP 803) were not attempted; they would be a follow-up.
• decompressobj().decompress(data, max_length) with max_length above 16 KiB holds at least twice the output during the copy, by design (see above).
• Not run: the remote CI jobs (shared-wheel tests on Linux, macOS Intel, macOS ARM, and Windows; the free-threaded and PyPy jobs; Windows 3.14t; release cibuildwheel builds with their abi3audit step), any Windows, macOS, or ARM runtime check, AddressSanitizer, UndefinedBehaviorSanitizer, and ThreadSanitizer on this candidate, and shellcheck. No CI has run on this branch yet.
• .github/workflows/ci.yml:106 still uses codecov/codecov-action@v1, which actionlint reports as too old. The line predates this PR and is unchanged; python-zlib-ng uses @v7. Upgrading needs the Codecov token or tokenless setting checked in live CI, so it is left to the maintainers.

Earlier attempts and considered approaches

Output strategies that were measured, and why native output was kept

These are prototype studies with their own harness and a full-API reference that already includes the end-of-stream stop, so the stop cannot hide the copy cost. Their ratios are not comparable with the final-wheel table: their Python corpus came from repeated package sources, their files record neither the interpreter nor input hashes (the versions below come from the report committed at 65d3eb3), and the bounded study, the free-threaded probe, and the 64 MiB memory test use exact power-of-two sizes. Each ran nine randomized paired rounds. Two strategies differ measurably in a case only when their ratios differ by more than the study's control maximum. The table covers the 24 large decoding cases of the irregular-size study (four corpora; 1 MiB + 137 and 16 MiB + 137 bytes; three decoders); across all 52 cases its control was 5.32% at p95 and 6.32% at most. Ranges are across case medians.

Strategy (variant ID)	Median	Range
Native output, selected (abi3-eof)	1.183×	0.837-1.731×
Joined 1 MiB chunks (abi3-chunks)	1.207×	0.419-1.555×
Eager BytesIO (abi3-bytesio)	1.334×	1.146-1.858×
Delayed BytesIO, created on the first growth (abi3-hybrid)	1.316×	1.151-1.851×

• Joined chunks: the evidence does not favour native output over it on speed or memory. The medians differ by less than the control. Joined chunks has the lower median in 12 of the 24 cases (10 beyond the control for joined chunks, 11 for native output) and the lower worst case, and native output has the higher traced peak at 16 MiB + 137 (3.00 against 2.06 output sizes; RSS growth 2.02 against 2.01). Joined chunks also measured 0.419-0.514× for random 16 MiB + 137 output, faster than the reference, which is unexplained. Native output was kept because it is the maintainer's prescribed design and needs no chunk list or join step; that this is simpler to maintain is a judgment, not a measurement. Joined chunks would be a defensible alternative.
• Eager and delayed BytesIO: their medians are higher, but they are not worse everywhere. At 1 MiB + 137, delayed BytesIO has the lower median in 9 of the 12 decoding cases (3 beyond the control for each side); at 16 MiB + 137 native output is faster beyond the control in all 12. For the natural FASTQ file (3,578,369 bytes), eager BytesIO measured 1.003-1.045× against 1.049-1.097× for native output, within the control in every case. A CPython 3.15.0a8 (alpha) repeat of native output against delayed BytesIO, for decompress() and igzip_lib.decompress() only, gave 1.210× (0.846-2.002×) against 1.403× (1.177-1.868×): native output was faster beyond the control in all 8 cases at 16 MiB + 137, while at 1 MiB + 137 delayed BytesIO had the lower median in 6 of the 8 (1 beyond the control, against 2 for native output). At exactly 64 MiB, BytesIO grows RSS by about one output size against two for native output, but at 16 MiB + 137 it grows RSS as much as native output, likely because of zero-filled spare capacity (interpretation). BytesIO was rejected for its higher median and worse 16 MiB + 137 results, and because it calls into io.BytesIO from C.
• Copying bounded output from a native scratch buffer (abi3-fast, despite the ID): for one call that fills a 16 MiB limit it measured 1.517× (FASTQ), 8.162× (Python), 1.915× (random), and 5.134× (zeros), and 2.369× for random at 1 MiB. Returning the bytes object directly (abi3-bounded) measured 1.000×, 0.971×, 0.989×, 0.994×, and 0.973× for those cases and 0.948-1.052× across all 24. The traced peak for a 16 MiB output fell from 33,641,953 to 16,864,737 bytes, the study's full-API baseline figure. The direct return was adopted.
• Native output on free-threaded CPython 3.13.11t: against the shipped bytes resizing it was faster beyond the control in 11 of the 12 cases at 1 MiB but had outliers up to 2.217× at 16 MiB, where bytes resizing stayed within 0.972-1.052×. Free-threaded builds keep bytes resizing.
• PyType_GenericAlloc for every instance: an early prototype that used it, zeroing the whole instance including the ISA-L state, appeared slower on small calls. This was not measured in a committed study; PyObject_New was kept.

Data: mitigation-irregular.json, mitigation-irregular-summary.json, mitigation-315-summary.json, mitigation-memory.json, mitigation-bounded.json, mitigation-bounded-memory.json, and free-threaded-probe.json. The report's "Output strategy studies" section lists the controls of each study and every variant ID.

Checklist

• Pull request details were added to CHANGELOG.rst
• Documentation was updated (if needed)

🤖 Generated with Claude Code