Blosc fails to import using PyPy

[willsthompson]

Minimal, reproducible code sample, a copy-pastable example if possible

from numcodecs import Blosc # executed via PyPy

Problem description

The latest numcodecs package (0.6.4) installs successfully via pip in a pypy-based venv. I see no errors or warnings when installing zarr (or when installing numcodecs independently) using -vvv. However, importing Blosc throws the exception: ImportError: cannot import name 'Blosc' (similar errors are thrown from Zarr when opening a Blosc-compressed file).

I assume this is related to an incompatibility with PyPy, but I couldn't find any documentation about using PyPy with Zarr.

Version and installation information

Please provide the following:

• Value of numcodecs.__version__: 0.6.4
• Version of Python interpreter:

    Python 3.6.9 (1608da62bfc7, Dec 23 2019, 10:50:17)
    [PyPy 7.3.0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]

• Operating system (Linux/Windows/Mac): macOS 10.15.5
• How NumCodecs was installed (e.g., "using pip into virtual environment", or "using conda"): pip into venv

[jakirkham]

I don't know that any explicit work has been done on PyPy support. So this might not work.

That said, components like Blosc depend on a working compiler to be available. On macOS this means installing Xcode and Xcode Command Line Tools.

It's unfortunate that pip lacks a way to communicate a warning (like noting some component won't be built) to the user more clearly. Have mentioned this to a few pip developers earlier this year.

[willsthompson]

Okay, thanks. After some digging it seems like this could be related to issues with the build in PyPy and/or c-blosc needs a separate CFFI-based wrapper (likely) to work with PyPy. I think it might be helpful to add a note about PyPy compatibility to the docs for anyone else scratching their heads about this.

[jakirkham]

Going to reopen as I think this is still worth tracking.

[jakirkham]

Just to add we are disabling the C extensions for PyPy on these lines. This goes back to the first commit of Numcodecs ( 4e9ee6b ) and appears to have been originally disabled in Zarr with PR ( zarr-developers/zarr-python#22 ).

numcodecs/setup.py

Lines 348 to 349 in b7c5a81

	is_pypy = hasattr(sys, 'pypy_translation_info')
	with_extensions = not is_pypy and 'DISABLE_NUMCODECS_CEXT' not in os.environ

@alimanfoo, do you recall what issues you ran into when trying to build Numcodecs on PyPy?

[jakirkham]

FWIW I tried just dropping those lines and building with PyPy locally on macOS and it seemed to work. Submitted PR ( #246 ) to make that change and add an item to the CI matrix.

[dstansby]

Since we don't support PyPy at the moment, I'm going to close this. I think we would welcome a more general request issue for PyPy support if there is demand however, and someone is willing to create and maintain some PyPy continuous integration.

[jakirkham]

Would add the small exception that we do support pure Python compressors for PyPy

We just disable all compiled extensions with PyPy (as noted below). This includes Blosc (hence the error above is expected currently)

numcodecs/setup.py

Lines 383 to 385 in 230bc01

	is_pypy = hasattr(sys, 'pypy_translation_info')
	with_extensions = not is_pypy and 'DISABLE_NUMCODECS_CEXT' not in os.environ
	run_setup(with_extensions)

There have been a few attempts to build compiled extensions for PyPy, but ultimately those have been abandoned. A fresh attempt would be welcome

[martindurant]

Note that cramjam (including some form of blosc) is built for pypy. Also wasm.