You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CFFI backend: crash when ZstdDecompressor instances share a ZstdCompressionDict #361
With the CFFI backend, using one ZstdCompressionDict with more than one ZstdDecompressor can crash the interpreter, or fail with ZstdError: zstd decompress error: Dictionary mismatch. The C backend isn't affected.
This reproducer doesn't need threads:
importzstandardaszstdsamples= [b"sample %d "%i+b"data "* (i%50) foriinrange(1000)]
dict_data=zstd.train_dictionary(8192, samples)
data=b"".join(samples)
frame=zstd.ZstdCompressor(dict_data=dict_data).compress(data)
reader=zstd.ZstdDecompressor(dict_data=dict_data).stream_reader(frame)
zstd.ZstdDecompressor(dict_data=dict_data) # another decompressor, same dictionaryassertreader.read() ==data
With PYTHON_ZSTANDARD_IMPORT_POLICY=cffi, every run either segfaulted or raised ZstdError: zstd decompress error: Dictionary mismatch. I ran it with:
With PYTHON_ZSTANDARD_IMPORT_POLICY=cext, it passes on all three. I also ran a stress test where 8 threads each create their own decompressors for one shared dictionary. With the CFFI backend it segfaults within a few hundred operations, both with and without the GIL.
Cause.ZstdCompressionDict._ddict in zstandard/backend_cffi.py (lines 2748–2766) is meant to create the DDict once and cache it in self.__dict__["_ddict"]. But property is a data descriptor, so it takes precedence over the instance __dict__ and the cached value is never read.
Every access creates a new DDict and stores it. That drops the previous DDict, and its ffi.gc destructor frees it.
ZstdDecompressor._ensure_dctx() runs when a decompressor is created and at the start of every operation. It passes self._dict_data._ddict to ZSTD_DCtx_refDDict() (line 4473), which only keeps a pointer.
So creating or using a second decompressor with the same dictionary frees the DDict that the first decompressor's context still points to.
The C backend creates the DDict once in ensure_ddict() (c-ext/compressiondict.c:148) and keeps it as long as the dictionary. The CFFI code has been like this since bef32a9 (2018), whose commit message says a DDict "can be used by multiple decompressors".
I have a fix that creates the DDict once, under a lock so that concurrent first uses don't create two, plus a regression test. I'll open a PR.
Activity