Implement PEP 784 - Adding Zstandard to the Python standard library #132983

emmatyping · 2025-04-26T00:55:59Z

Feature or enhancement

This is a tracking issue for implementing PEP 784. See the PEP text for more details.

Since the diff is significant (~10k lines) I wanted to split up the PRs a bit.

Implementation Plan:

Linked PRs

The text was updated successfully, but these errors were encountered:

picnixz · 2025-04-26T17:37:36Z

It's something I should have asked on discourse, but is compression the final name of the public module? I see in the PEP that we want to mimic hashlib.algorithms_guaranteed as well and one way to mitigate the name shadowing is to actually use compressionlib as the name (we have hashlib, annotationlib, and we could have compressionlib as well; I also think compressionlib is a less likely name to be chosen but it's not the best one as well). Feel free to ignore this question as the PEP has been accepted.

emmatyping · 2025-04-26T18:35:25Z

It's something I should have asked on discourse, but is compression the final name of the public module?

Yes, the decision was to use compression. A search of Github code showed that it was used very infrequently. Literally one in a million files https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/66

…dule (GH-133018) * Introduces `compression` package for https://peps.python.org/pep-0784/ This commit introduces the `compression` package, specified in PEP 784 to re-export the `lzma`, `bz2`, `gzip`, and `zlib` modules. Introduction of `compression.zstd` will be completed in a future commit once the `_zstd` module is merged. This commit also moves the `_compression` private module to `compression._common._streams`. * Re-exports existing module docstrings.

Include compression package contents as part of installs.

Add an __init__ file. Fix test_tools.test_freeze.

Add an "__init__.py" file. Fix test_tools.test_freeze.

* Add _zstd module for https://peps.python.org/pep-0784/ This commit introduces the `_zstd` module, with bindings to libzstd from the pyzstd project. It also includes the unix build system configuration. Windows build system support will be integrated independently as it depends on integration with cpython-source-deps. * Add _zstd to modules * Fix path for compression.zstd module * Ignore _zstd module like _io * Expand module state macros to improve code quality Also removes module state references from the classes in the _zstd module and instead uses PyType_GetModuleState() * Remove backticks suggested in review Co-authored-by: Stan Ulbrych <[email protected]> * Use critical sections to lock object state This should avoid races and deadlocks. * Remove compress/decompress and mark module as not reliant on the GIL The `compress`/`decompress` functions will be moved to Python code for simplicity. C implementations can always be re-added in the future. Also, mark _zstd as not requiring the GIL. * Lift critical section to avoid clang warning * Respond to comments by picnixz * Call out pyzstd explicitly in license description Co-authored-by: Adam Turner <[email protected]> * Use a much more robust implementation... ... for `get_zstd_state_from_type` Co-authored-by: Bénédikt Tran <[email protected]> * Use PyList_GetItemRef for thread safety purposes * Use a macro for the minimum supported version * remove const from primivite types * Use PyMem_New in another spot * Simplify error handling in _get_frame_size * Another simplification of error handling in get_frame_info * Rename _module_state to mod_state * Rewrite comment explaining the context of the code * Add link to pyzstd * Add TODO about refactoring dict training code * Use PyModule_AddObjectRef over PyModule_AddObject PyModule_AddObject is soft-deprecated, so we should use PyModule_AddObjectRef * Check result of OutputBufferGrow * Simplify return logic in `add_constant_to_type` Co-authored-by: Bénédikt Tran <[email protected]> * Ignore return value of _zstd_clear() Co-authored-by: Bénédikt Tran <[email protected]> * Remove redundant comments * Remove __reduce__ from ZstdDict We should instead document that to pickle a dictionary a user should use the `.dict_content` attribute. * Use PyUnicode_FromFormat instead of a buffer * Don't use C constants/types in error messages * Make error messages easier to understand for Python users * Lower minimum required version 1.4.0 * Use casts and make slot function signatures correct * Be consistent with CPython on const usage * Make else clauses in line with PEP 7 * Fix over-indented blocks in argument clinic * Add critical section around ZSTD_DCtx_setParameter * Add a TODO about refactoring critical sections * Use Py_UNREACHABLE * Move bytes operations out of Py_BEGIN_ALLOW_THREADS * Add TODO about ensuring a lock is held * Remove asserts that may not be correct * Add TODO to make ZstdDict and others GC objects * Make objects GC tracked * Remove unused include * Fix some memory issues * Fix refleaks on module and in ZstdDict * Update configure to check for ZDICT_finalizeDictionary * Properly check version in configure * exit(1) if check fails * Use AC_RUN_IFELSE * Use a define() to re-use version check * Actually properly set _zstd module status based on version --------- Co-authored-by: Stan Ulbrych <[email protected]> Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Bénédikt Tran <[email protected]>

AA-Turner · 2025-05-04T05:12:30Z

I've had a go at adding support for building on Windows in #133366, let me know thoughts.

A

* origin/main: (111 commits) pythongh-91048: Add filename and line number to external inspection routines (pythonGH-133385) pythongh-131178: Add tests for `ast` command-line interface (python#133329) Regenerate pcbuild.sln in Visual Studio 2022 (python#133394) pythongh-133042: disable HACL* HMAC on Emscripten (python#133064) pythongh-133351: Fix remote PDB's multi-line block tab completion (python#133387) pythongh-109700: Improve stress tests for interpreter creation (pythonGH-109946) pythongh-81793: Skip tests for os.link() to symlink on Android (pythonGH-133388) pythongh-126835: Rename `ast_opt.c` to `ast_preprocess.c` and related stuff after moving const folding to the peephole optimizier (python#131830) pythongh-91048: Relax test_async_global_awaited_by to fix flakyness (python#133368) pythongh-132457: make staticmethod and classmethod generic (python#132460) pythongh-132805: annotationlib: Fix handling of non-constant values in FORWARDREF (python#132812) pythongh-132426: Add get_annotate_from_class_namespace replacing get_annotate_function (python#132490) pythongh-81793: Always call linkat() from os.link(), if available (pythonGH-132517) pythongh-122559: Synchronize C and Python implementation of the io module about pickling (pythonGH-122628) pythongh-69605: Add PyREPL import autocomplete feature to 'What's New' (python#133358) bpo-44172: Keep reference to original window in curses subwindow objects (pythonGH-26226) pythonGH-133231: Changes to executor management to support proposed `sys._jit` module (pythonGH-133287) pythongh-133363: Fix Cmd completion for lines beginning with `! ` (python#133364) pythongh-132983: Introduce `_zstd` bindings module (pythonGH-133027) pythonGH-91048: Add utils for printing the call stack for asyncio tasks (python#133284) ...

ned-deily · 2025-05-05T20:53:50Z

It looks like there are a couple of problems with the configure.ac changes to discover and check libzstd.

Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Gregory P. Smith <[email protected]> Co-authored-by: Tomas R. <[email protected]> Co-authored-by: Rogdham <[email protected]>

AA-Turner · 2025-05-06T00:45:07Z

Windows build support, autoconf fix, and the Python wrapper now all merged. I think that should be everything required for b1, but we could use this issue to track the outstanding items from #133365.

Thanks!

A

emmatyping · 2025-05-06T01:15:57Z

Windows build support, autoconf fix, and the Python wrapper now all merged. I think that should be everything required for b1, but we could use this issue to track the outstanding items from #133365.

I went ahead and added checkboxes for all of the TODOs left after review of both #133365 and #133027, as well as any TODO comments not discussed there. If I missed anything please leave a comment or edit the OP.

…) (#133788) GH-132983: remove empty_bytes from _zstd module state (GH-133785) (cherry picked from commit 98e2c3a) Co-authored-by: Adam Turner <[email protected]>

Rogdham · 2025-05-09T21:36:45Z

re the race in compression

Could it be the following?

By sharing the ZstdCompressor instance between the threads, we share the compression context of type ZSTD_CCtx
From a comment in zstd.h:
- “since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing”
- “For parallel execution, use one separate ZSTD_CStream per thread.”

Details

From below, it seems that ZSTD_compressStream_generic line 6165 is writing data at the location of zcs->inBuff (zcs being the compression context), so if another threads does the same or set it to null, we have the issue.

ThreadSanitizer:DEADLYSIGNAL
    #0 <null> <null> (libc.so.6+0x16c5c7) (BuildId: d056ce83eebe65ce7e52ecfa5af5363e4863d283)
    #1 memcpy <null> (python+0xe8c98) (BuildId: dea998379ba126e7933ccc59c8ddb0390562739e)
    #2 ZSTD_limitCopy /usr/src/debug/zstd/zstd-1.5.7/lib/compress/../common/zstd_internal.h:252:9 (libzstd.so.1+0x26eae) (BuildId: 9723b93a8052010d25908aaa6174df6de760859a)
    #3 ZSTD_compressStream_generic /usr/src/debug/zstd/zstd-1.5.7/lib/compress/zstd_compress.c:6165:39 (libzstd.so.1+0x26eae)
    #4 ZSTD_compressStream2 /usr/src/debug/zstd/zstd-1.5.7/lib/compress/zstd_compress.c:6540:5 (libzstd.so.1+0x26eae)
    #5 compress_impl /redacted/./Modules/_zstd/compressor.c:453:20 (_zstd.cpython-315t-x86_64-linux-gnu.so+0x77e0) (BuildId: 74b337c2def9ea3fa795cafd5a7f41ce493b69f9)
    #6 _zstd_ZstdCompressor_compress_impl /redacted/./Modules/_zstd/compressor.c:591:15 (_zstd.cpython-315t-x86_64-linux-gnu.so+0x8cbf) (BuildId: 74b337c2def9ea3fa795cafd5a7f41ce493b69f9)
    #7 _zstd_ZstdCompressor_compress /redacted/./Modules/_zstd/clinic/compressor.c.h:171:20 (_zstd.cpython-315t-x86_64-linux-gnu.so+0x8cbf)
<redacted>

…-133791) (cherry picked from commit 1978904) Co-authored-by: Adam Turner <[email protected]>

…) (#133792) GH-132983: PEP 7 and Argument Clinic changes for zstd (GH-133791) (cherry picked from commit 1978904) Co-authored-by: Adam Turner <[email protected]>

(cherry picked from commit 50b5370) Co-authored-by: Rogdham <[email protected]>

…133799) gh-132983: Don't allow trailer data in ZstdFile (GH-133736) (cherry picked from commit 50b5370) Co-authored-by: Rogdham <[email protected]>

(cherry picked from commit 1a548c0) Co-authored-by: Adam Turner <[email protected]>

…133854) gh-132983: Reduce the size of ``_zstdmodule.h`` (GH-133793) (cherry picked from commit 1a548c0) Co-authored-by: Adam Turner <[email protected]>

(cherry picked from commit 1a87b6e) Co-authored-by: Adam Turner <[email protected]>

gh-132983: Make zstd types immutable (GH-133784) (cherry picked from commit 1a87b6e) Co-authored-by: Adam Turner <[email protected]>

Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

…nGH-133856) (cherry picked from commit 878e0fb) Co-authored-by: Rogdham <[email protected]> Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

…33856) (#133859) gh-132983: Remove leftovers from EndlessZstdDecompressor (GH-133856) (cherry picked from commit 878e0fb) Co-authored-by: Rogdham <[email protected]> Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

sobolevn · 2025-05-11T11:21:17Z

Please, see #133885

emmatyping added extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Apr 26, 2025

emmatyping self-assigned this Apr 26, 2025

emmatyping mentioned this issue Apr 26, 2025

gh-132983: Introduce compression package and move _compression module #133018

Merged

picnixz added this to Compression issues Apr 26, 2025

bedevere-app bot mentioned this issue Apr 26, 2025

gh-132983: Introduce _zstd bindings module #133027

Merged

4 tasks

bedevere-app bot mentioned this issue Apr 27, 2025

gh-132983: Install compression package contents #133063

Merged

freakboy3742 pushed a commit that referenced this issue Apr 28, 2025

gh-132983: Install compression package contents (#133063)

632978f

Include compression package contents as part of installs.

bedevere-app bot mentioned this issue Apr 28, 2025

gh-132983: make compression._common a full-fledged package #133076

Closed

vstinner added a commit to vstinner/cpython that referenced this issue Apr 28, 2025

pythongh-132983: Convert compression._common to a package

0789ab8

Add an __init__ file. Fix test_tools.test_freeze.

bedevere-app bot mentioned this issue Apr 28, 2025

gh-132983: Convert compression._common to a package #133086

Merged

vstinner added a commit that referenced this issue Apr 28, 2025

gh-132983: Convert compression._common to a package (#133086)

fe8ad53

Add an "__init__.py" file. Fix test_tools.test_freeze.

This was referenced Apr 30, 2025

gh-132983: [DRAFT] Test _zstd code with Python test suite #133185

Closed

gh-132983: [Draft] Run refleak tests against _zstd with tests #133282

Closed

This was referenced May 4, 2025

gh-132983: Add compression.zstd and Python tests #133365

Merged

GH-132983: Build _zstd on Windows #133366

Merged

bedevere-app bot mentioned this issue May 5, 2025

GH-132983: Use a standard autoconf version check for zstd #133479

Merged

AA-Turner added a commit that referenced this issue May 5, 2025

GH-132983: Build _zstd on Windows (#133366)

e6f8e0a

AA-Turner added a commit that referenced this issue May 6, 2025

GH-132983: Use a standard autoconf version check for zstd (#133479)

793402e

bedevere-app bot mentioned this issue May 9, 2025

[3.14] GH-132983: remove empty_bytes from _zstd module state (GH-133785) #133788

Merged

bedevere-app bot mentioned this issue May 9, 2025

GH-132983: PEP 7 and Argument Clinic changes for zstd #133791

Merged

AA-Turner added a commit that referenced this issue May 10, 2025

GH-132983: PEP 7 and Argument Clinic changes for zstd (#133791)

1978904

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 10, 2025

pythonGH-132983: PEP 7 and Argument Clinic changes for zstd (pythonGH…

807082d

…-133791) (cherry picked from commit 1978904) Co-authored-by: Adam Turner <[email protected]>

This was referenced May 10, 2025

[3.14] GH-132983: PEP 7 and Argument Clinic changes for zstd (GH-133791) #133792

Merged

gh-132983: Reduce the size of _zstdmodule.h #133793

Merged

AA-Turner pushed a commit that referenced this issue May 10, 2025

gh-132983: Don't allow trailer data in ZstdFile (#133736)

50b5370

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 10, 2025

pythongh-132983: Don't allow trailer data in ZstdFile (pythonGH-133736)

0c55b02

(cherry picked from commit 50b5370) Co-authored-by: Rogdham <[email protected]>

bedevere-app bot mentioned this issue May 10, 2025

[3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) #133799

Merged

picnixz mentioned this issue May 10, 2025

gh-133810: remove http.server.CGIHTTPRequestHandler and --cgi flag #133811

Open

AA-Turner added a commit that referenced this issue May 10, 2025

gh-132983: Reduce the size of _zstdmodule.h (#133793)

1a548c0

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 10, 2025

pythongh-132983: Reduce the size of _zstdmodule.h (pythonGH-133793)

f29ced5

(cherry picked from commit 1a548c0) Co-authored-by: Adam Turner <[email protected]>

This was referenced May 10, 2025

[3.14] gh-132983: Reduce the size of _zstdmodule.h (GH-133793) #133854

Merged

gh-132983: Remove EndlessZstdDecompressor remains #133856

Merged

AA-Turner added a commit that referenced this issue May 10, 2025

gh-132983: Make zstd types immutable (#133784)

1a87b6e

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 10, 2025

pythongh-132983: Make zstd types immutable (pythonGH-133784)

4146ed7

(cherry picked from commit 1a87b6e) Co-authored-by: Adam Turner <[email protected]>

bedevere-app bot mentioned this issue May 10, 2025

[3.14] gh-132983: Make zstd types immutable (GH-133784) #133857

Merged

AA-Turner added a commit that referenced this issue May 10, 2025

[3.14] gh-132983: Make zstd types immutable (GH-133784) (#133857)

1e8a534

gh-132983: Make zstd types immutable (GH-133784) (cherry picked from commit 1a87b6e) Co-authored-by: Adam Turner <[email protected]>

AA-Turner added a commit that referenced this issue May 11, 2025

gh-132983: Remove leftovers from EndlessZstdDecompressor (#133856)

878e0fb

Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Peter Bierma <[email protected]>

bedevere-app bot mentioned this issue May 11, 2025

[3.14] gh-132983: Remove leftovers from EndlessZstdDecompressor (GH-133856) #133859

Merged

bedevere-app bot mentioned this issue May 11, 2025

gh-132983: Convert zstd __new__ methods to Argument Clinic #133860

Open

Rogdham mentioned this issue May 11, 2025

test_zstd failed on ubuntu with free-threading #133885

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement PEP 784 - Adding Zstandard to the Python standard library #132983

Implement PEP 784 - Adding Zstandard to the Python standard library #132983

emmatyping commented Apr 26, 2025 •

edited by bedevere-app bot

Loading

picnixz commented Apr 26, 2025

emmatyping commented Apr 26, 2025

AA-Turner commented May 4, 2025

ned-deily commented May 5, 2025

AA-Turner commented May 6, 2025

emmatyping commented May 6, 2025

Rogdham commented May 9, 2025 •

edited

Loading

sobolevn commented May 11, 2025

Implement PEP 784 - Adding Zstandard to the Python standard library #132983

Implement PEP 784 - Adding Zstandard to the Python standard library #132983

Comments

emmatyping commented Apr 26, 2025 • edited by bedevere-app bot Loading

Feature or enhancement

Implementation Plan:

Linked PRs

picnixz commented Apr 26, 2025

emmatyping commented Apr 26, 2025

AA-Turner commented May 4, 2025

ned-deily commented May 5, 2025

AA-Turner commented May 6, 2025

emmatyping commented May 6, 2025

Rogdham commented May 9, 2025 • edited Loading

sobolevn commented May 11, 2025

emmatyping commented Apr 26, 2025 •

edited by bedevere-app bot

Loading

Rogdham commented May 9, 2025 •

edited

Loading