Add arrow datafusion python dependency (#1085)

* Introduce Arrow-DataFusion-Python, the new Apache Arrow community Python bindings for DataFusion and bare minimum refactoring we need in our repo to accomodate them

* point to test upstream branch with custom build command

* update conda build recipe to include rust compiler and setuptools-rust in every build stage

* update cargo lock

* lower setuptools-rust version for a test

* Add actions for Rust environment setup

* fix workflow syntax issue

* Add RUSTFMT which output

* Set rustfmt environment variable and validate it sticks

* Updated Cargo.lock

* Add RUSTFMT to the export script_env variables for the conda build

* Set default to stable

* Point to typify version with rustfmt fix, and update conda build file

* Point to version of arrow-datafusion-python with the typify rustfmt fix

* Refactoring: not from arrow-datafusion-python but general refactoring needed from updating datafusion version to 20.0.0

* Add setuptools-rust back to conda meta.yaml recipe from where I had accidentally deleted it earlier

* Remove rust install step and also don't use container amd64/rust

* Add setup-builder back to conda workflow action

* Run apt-get commands as root

* Add Rust installation to test.yml

* Add Rust install to Rust.yml file because protoc is now needed when running cargo check to work with the substrait dependecy

* Install protoc

* tests

* Change github action files

* remove commented out code

* Add protoc action to Rust CI stage

* Add protoc action to Rust CI stage

* Add protoc to other CI build steps so that substrait support can now be built

* Add protoc to conda workflow file

* Add Utf8 parsing support for WindowRanges

* Get flat_name() from Column. This will include the table name in the new datafusion refactoring

* Add protoc to github action step Test importing with bare requirements

* Fix cargo tests

* Add build.rs and target linker flags to ensure binary is linked against masOS python bindings

* Updates for testing

* Adjustments to fix failing pytests

* More Pytest fixes and Rust test fixes

* Remove mistakenly left inplace breakpoint() calls

* Return np.timedelta instead of pd.timedelta for interval_month_day_nano

* protoc changes and small formatting review suggestions

* Include protoc back in rust.yml for cargo check command

* set avg_days_in_month -> 30. Adjust update_depedencies script to point to arrow-datafusion-python repo

* Add back logic for avg_days_in_month

* Set average days per month to 30

* Adjust PyTest expected results to align with new avg_days_in_month of 30 days

* Revert timestampdiff changes

---------

Co-authored-by: Charles Blackmon-Luca <[email protected]>
Co-authored-by: Ayush Dattagupta <[email protected]>

Author: 3 people

.github/workflows/rust.yml

@@ -30,6 +30,11 @@ jobs:
         id: detect-trigger
         with:
           keyword: "[test-df-upstream]"
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v1
+        with:
+          version: '3.x'
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
 
   # Check crate compiles
   linux-build-lib:
@@ -48,6 +53,11 @@ jobs:
         run: |
           cd dask_planner
           bash update-dependencies.sh
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v1
+        with:
+          version: '3.x'
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Check workspace in debug mode
         run: |
           cd dask_planner
@@ -76,6 +86,11 @@ jobs:
         run: |
           cd dask_planner
           bash update-dependencies.sh
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v1
+        with:
+          version: '3.x'
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Run tests
         run: |
           cd dask_planner

.github/workflows/style.yml

@@ -15,6 +15,11 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v1
+        with:
+          version: '3.x'
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
       - uses: actions-rs/toolchain@v1
         with:
             toolchain: nightly

.github/workflows/test.yml

@@ -109,6 +109,11 @@ jobs:
           use-mamba: true
           python-version: "3.8"
           channel-priority: strict
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v1
+        with:
+          version: '3.x'
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Install dependencies and nothing else
         run: |
           mamba install "setuptools-rust>=1.5.2"

continuous_integration/environment-3.10-dev.yaml

@@ -32,3 +32,4 @@ dependencies:
 - tpot
 - tzlocal>=2.1
 - uvicorn>=0.13.4
+- libprotobuf=3

continuous_integration/environment-3.8-dev.yaml

@@ -31,3 +31,4 @@ dependencies:
 - tpot
 - tzlocal=2.1
 - uvicorn=0.13.4
+- libprotobuf=3

continuous_integration/environment-3.9-dev.yaml

@@ -32,3 +32,4 @@ dependencies:
 - tpot
 - tzlocal>=2.1
 - uvicorn>=0.13.4
+- libprotobuf=3

continuous_integration/recipe/meta.yaml

@@ -28,6 +28,7 @@ requirements:
     - pip
     - python
     - setuptools-rust >=1.5.2
+    - libprotobuf =3
   run:
     - python
     - dask >=2022.3.0

dask_planner/.cargo/config.toml

@@ -0,0 +1,11 @@
+[target.x86_64-apple-darwin]
+rustflags = [
+  "-C", "link-arg=-undefined",
+  "-C", "link-arg=dynamic_lookup",
+]
+
+[target.aarch64-apple-darwin]
+rustflags = [
+  "-C", "link-arg=-undefined",
+  "-C", "link-arg=dynamic_lookup",
+]