Add arrow datafusion python dependency #1085
Conversation
…hon bindings for DataFusion and bare minimum refactoring we need in our repo to accomodate them
jdye64
requested review from
andygrove,
ayushdg,
charlesbluca and
galipremsagar
as code owners
…t in every build stage
|
Waiting on oxidecomputer/typify#221 to merge so that conda builds can work with new setup. |
…sql into add_arrow_datafusion_python
… needed from updating datafusion version to 20.0.0
…ccidentally deleted it earlier
There was a problem hiding this comment.
Thanks @jdye64 🎉 a few comments around how we include protoc in CI/packaging and some follow up around the timedelta changes
…sql into add_arrow_datafusion_python
…t to arrow-datafusion-python repo
tests/integration/test_rex.py
Outdated
| @@ -926,8 +926,8 @@ def test_timestampdiff(c): | |||
| "hours": [46, 6648, 591744], | |||
| "days": [1, 277, 24656], | |||
| "weeks": [0, 39, 3522], | |||
| "months": [0, 9, 810], | |||
| "quarters": [0, 3, 270], | |||
| "months": [0, 9, 821], | |||
There was a problem hiding this comment.
Since this behavior is different from the results we get from mysql and a few implementations that support timestampdiff, I'm not sure whether we want to keep the existing logic or modify it. I didn't find an easy way to get a timestampdiff with datafusion/postgresql
| [build-system] | ||
| requires = ["setuptools", "wheel", "setuptools-rust"] | ||
|
|
||
| [tool.isort] | ||
| profile = "black" | ||
|
|
||
| [tool.maturin] | ||
| include = [ | ||
| { path = "Cargo.lock", format = "sdist" } | ||
| ] | ||
| exclude = [".github/**", "ci/**", ".asf.yaml"] | ||
| # Require Cargo.lock is up to date | ||
| locked = true |
There was a problem hiding this comment.
Noting that I've been thinking about consolidating as much of our project metadata into pyproject.toml as possible (similar to other RAPIDS/Dask projects) which would make this file less stubby
There was a problem hiding this comment.
+1 to this. I've also been thinking of simplifying the setup.py scripts and moving as much as possible into the pyproject.toml file
Closed with #1084