Skip to content

CFFI backend: crash when ZstdDecompressor instances share a ZstdCompressionDict #361

Open

Description

@pylaterreur

With the CFFI backend, using one ZstdCompressionDict with more than one ZstdDecompressor can crash the interpreter, or fail with ZstdError: zstd decompress error: Dictionary mismatch. The C backend isn't affected.

This reproducer doesn't need threads:

import zstandard as zstd

samples = [b"sample %d " % i + b"data " * (i % 50) for i in range(1000)]
dict_data = zstd.train_dictionary(8192, samples)
data = b"".join(samples)
frame = zstd.ZstdCompressor(dict_data=dict_data).compress(data)

reader = zstd.ZstdDecompressor(dict_data=dict_data).stream_reader(frame)
zstd.ZstdDecompressor(dict_data=dict_data)  # another decompressor, same dictionary
assert reader.read() == data

With PYTHON_ZSTANDARD_IMPORT_POLICY=cffi, every run either segfaulted or raised ZstdError: zstd decompress error: Dictionary mismatch. I ran it with:

  • zstandard 0.25.0 (PyPI wheel) on CPython 3.14.8;
  • main (a2bb3c0) on CPython 3.15.0 and 3.15.0t.

With PYTHON_ZSTANDARD_IMPORT_POLICY=cext, it passes on all three. I also ran a stress test where 8 threads each create their own decompressors for one shared dictionary. With the CFFI backend it segfaults within a few hundred operations, both with and without the GIL.

Cause. ZstdCompressionDict._ddict in zstandard/backend_cffi.py (lines 2748–2766) is meant to create the DDict once and cache it in self.__dict__["_ddict"]. But property is a data descriptor, so it takes precedence over the instance __dict__ and the cached value is never read.

  • Every access creates a new DDict and stores it. That drops the previous DDict, and its ffi.gc destructor frees it.
  • ZstdDecompressor._ensure_dctx() runs when a decompressor is created and at the start of every operation. It passes self._dict_data._ddict to ZSTD_DCtx_refDDict() (line 4473), which only keeps a pointer.
  • So creating or using a second decompressor with the same dictionary frees the DDict that the first decompressor's context still points to.

The C backend creates the DDict once in ensure_ddict() (c-ext/compressiondict.c:148) and keeps it as long as the dictionary. The CFFI code has been like this since bef32a9 (2018), whose commit message says a DDict "can be used by multiple decompressors".

I have a fix that creates the DDict once, under a lock so that concurrent first uses don't create two, plus a regression test. I'll open a PR.

Prepared with the help of Claude Code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions