Current Packaging Plan Does Not Support Gradual Adoption from Libraries

[zprobst]

Hello 👋

I am the maintainer of nodestream which is a declarative framework for Building and Maintaining graph data. As a result, a core workflow that nodestream integrates with neo4j is ETL-ing data into the graph.

I was considering allowing users to opt-in to the rust extensions. For a lot of reasons, including the possibility of needing rust build tools, users may prefer the pure python driver over the rust extensions. However, the current packaging pattern does not provide a clean way to do that. Here are the options I've considered

• Require the user to explicitly install which package they want instead of depending on neo4j or the neo4j-rust-ext directly. This is clunky because it adds a step for all users of the package, including those that don't know what the rust driver extensions do.
• Have extras that allow the user to install the right dependency. However, you can't enforce either a default or a mutually exclusive dependency.

Ideally I'd like to be able to give our users the ability to opt-into the rust extensions with an extra that installs a certain version of the driver. For instance, pip install nodestream-plugin-neo4j[rust] extra pulls in neo4j-rust-ext as opposed to just neo4j.

[robsdedude]

Hi and thanks for reaching out.

Sorry for the late reply; I was on leave last week and was very deep in the process of rethinking the packaging of the Rust extensions. I was hoping to find a clean way to give users the Rust extensions when installing neo4j but without forcing them to have a Rust tool chain available by gracefully falling back to the pure Python implementation if the extensions were not available. Your issue came just in time to make me reconsider my first approach. So thanks for that.

We've settled on ditching my plans for now as with the limitations of Python's packaging system that you mentioned plus others I couldn't come up with a clean solution. So for now the packing will stay as it is.

As a matter of fact, you can install the driver together with the Rust extensions and it will work. I just deliberately chose not to document it that way as it's way too easy to end up with an incompatible version combination. Further, I wanted to leave myself more options for packaging. However, I now see that this is a really bad idea for downstream libraries like yours.

Would adding the following to the README of this project (and of course also committing to supporting this) be a solution to you problem?

ℹ️ Since the driver is a simple Python dependency of this package, you can also manually install/specify both packages at the same time without issues.
However, make sure the versions match if you do so or leave the version of one of the two unspecified to let the package manager pick a compatible version for you (resolution might be slow, however).

That way, you'd be able to specify optional dependencies in your pyproject.toml or wherever. Along the lines of

[project]
name = "nodestream-plugin-neo4"
# ...
dependencies = [
    "neo4j == X.Y.Z"
]

[project.optional-dependencies]
rust = ["neo4j-rust-ext == X.Y.Z.*"]

and users could just either pip install nodestream-plugin-neo4j[rust] or pip install nodestream-plugin-neo4j.