Skip to content

Commit f0d5659

Browse files
francis-dufrancis-du
authored
[DOCS] - Fix sample code and python api docs (#71)
* fix: fix sample code and python api docs * fix: readme code sample * fix: python lint * fix: repo name in docs & url link * fix: repo name in docs & url link * fix: remove useless dependency * fix: remove .DS_Store
1 parent 72f0600 commit f0d5659

29 files changed

+590
-219
lines changed

‎.DS_Store

6 KB
Binary file not shown.

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@ target
22
Cargo.lock
33
/venv
44
.idea
5+
/docs/temp
6+
/docs/build
7+
.DS_Store
58

69
# Byte-compiled / optimized / DLL files
710
__pycache__/

‎README.md

+21-2
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,6 @@ Simple usage:
4040

4141
```python
4242
import datafusion
43-
from datafusion import functions as f
4443
from datafusion import col
4544
import pyarrow
4645

@@ -70,16 +69,27 @@ assert result.column(1) == pyarrow.array([-3, -3, -3])
7069
### UDFs
7170

7271
```python
72+
import pyarrow
7373
from datafusion import udf
7474

7575
def is_null(array: pyarrow.Array) -> pyarrow.Array:
7676
return array.is_null()
7777

7878
is_null_arr = udf(is_null, [pyarrow.int64()], pyarrow.bool_(), 'stable')
7979

80+
# create a context
81+
ctx = datafusion.SessionContext()
82+
83+
# create a RecordBatch and a new DataFrame from it
84+
batch = pyarrow.RecordBatch.from_arrays(
85+
[pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
86+
names=["a", "b"],
87+
)
88+
df = ctx.create_dataframe([[batch]])
89+
8090
df = df.select(is_null_arr(col("a")))
8191

82-
result = df.collect()
92+
result = df.collect()[0]
8393

8494
assert result.column(0) == pyarrow.array([False] * 3)
8595
```
@@ -89,7 +99,9 @@ assert result.column(0) == pyarrow.array([False] * 3)
8999
```python
90100
import pyarrow
91101
import pyarrow.compute
102+
import datafusion
92103
from datafusion import udaf, Accumulator
104+
from datafusion import col
93105

94106

95107
class MyAccumulator(Accumulator):
@@ -113,7 +125,14 @@ class MyAccumulator(Accumulator):
113125
def evaluate(self) -> pyarrow.Scalar:
114126
return self._sum
115127

128+
# create a context
129+
ctx = datafusion.SessionContext()
116130

131+
# create a RecordBatch and a new DataFrame from it
132+
batch = pyarrow.RecordBatch.from_arrays(
133+
[pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
134+
names=["a", "b"],
135+
)
117136
df = ctx.create_dataframe([[batch]])
118137

119138
my_udaf = udaf(MyAccumulator, pyarrow.float64(), pyarrow.float64(), [pyarrow.float64()], 'stable')

‎docs/Makefile

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
#
19+
# Minimal makefile for Sphinx documentation
20+
#
21+
22+
# You can set these variables from the command line, and also
23+
# from the environment for the first two.
24+
SPHINXOPTS ?=
25+
SPHINXBUILD ?= sphinx-build
26+
SOURCEDIR = source
27+
BUILDDIR = build
28+
29+
# Put it first so that "make" without argument is like "make help".
30+
help:
31+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
32+
33+
.PHONY: help Makefile
34+
35+
# Catch-all target: route all unknown targets to Sphinx using the new
36+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
37+
%: Makefile
38+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

‎docs/README.md

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# DataFusion Documentation
21+
22+
This folder contains the source content of the [python api](./source/api).
23+
These are both published to https://arrow.apache.org/datafusion/
24+
as part of the release process.
25+
26+
## Dependencies
27+
28+
It's recommended to install build dependencies and build the documentation
29+
inside a Python virtualenv.
30+
31+
- Python
32+
- `pip install -r requirements.txt`
33+
34+
## Build & Preview
35+
36+
Run the provided script to build the HTML pages.
37+
38+
```bash
39+
./build.sh
40+
```
41+
42+
The HTML will be generated into a `build` directory.
43+
44+
Preview the site on Linux by running this command.
45+
46+
```bash
47+
firefox build/html/index.html
48+
```
49+
50+
## Release Process
51+
52+
The documentation is served through the
53+
[arrow-site](https://github.com/apache/arrow-site/) repo. To release a new
54+
version of the docs, follow these steps:
55+
56+
1. Run `./build.sh` inside `docs` folder to generate the docs website inside the `build/html` folder.
57+
2. Clone the arrow-site repo
58+
3. Checkout to the `asf-site` branch (NOT `master`)
59+
4. Copy build artifacts into `arrow-site` repo's `datafusion` folder with a command such as
60+
61+
- `cp -rT ./build/html/ ../../arrow-site/datafusion/` (doesn't work on mac)
62+
- `rsync -avzr ./build/html/ ../../arrow-site/datafusion/`
63+
64+
5. Commit changes in `arrow-site` and send a PR.

‎docs/build.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
set -e
22+
rm -rf build 2> /dev/null
23+
rm -rf temp 2> /dev/null
24+
mkdir temp
25+
cp -rf source/* temp/
26+
# replace relative URLs with absolute URLs
27+
#sed -i 's/\.\.\/\.\.\/\.\.\//https:\/\/github.com\/apache\/arrow-datafusion\/blob\/master\//g' temp/contributor-guide/index.md
28+
make SOURCEDIR=`pwd`/temp html

‎docs/make.bat

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
@rem Licensed to the Apache Software Foundation (ASF) under one
2+
@rem or more contributor license agreements. See the NOTICE file
3+
@rem distributed with this work for additional information
4+
@rem regarding copyright ownership. The ASF licenses this file
5+
@rem to you under the Apache License, Version 2.0 (the
6+
@rem "License"); you may not use this file except in compliance
7+
@rem with the License. You may obtain a copy of the License at
8+
@rem
9+
@rem http://www.apache.org/licenses/LICENSE-2.0
10+
@rem
11+
@rem Unless required by applicable law or agreed to in writing,
12+
@rem software distributed under the License is distributed on an
13+
@rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
@rem KIND, either express or implied. See the License for the
15+
@rem specific language governing permissions and limitations
16+
@rem under the License.
17+
18+
@ECHO OFF
19+
20+
pushd %~dp0
21+
22+
REM Command file for Sphinx documentation
23+
24+
if "%SPHINXBUILD%" == "" (
25+
set SPHINXBUILD=sphinx-build
26+
)
27+
set SOURCEDIR=source
28+
set BUILDDIR=build
29+
30+
if "%1" == "" goto help
31+
32+
%SPHINXBUILD% >NUL 2>NUL
33+
if errorlevel 9009 (
34+
echo.
35+
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
36+
echo.installed, then set the SPHINXBUILD environment variable to point
37+
echo.to the full path of the 'sphinx-build' executable. Alternatively you
38+
echo.may add the Sphinx directory to PATH.
39+
echo.
40+
echo.If you don't have Sphinx installed, grab it from
41+
echo.http://sphinx-doc.org/
42+
exit /b 1
43+
)
44+
45+
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
46+
goto end
47+
48+
:help
49+
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
50+
51+
:end
52+
popd

‎docs/requirements.txt

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
sphinx
19+
pydata-sphinx-theme==0.8.0
20+
myst-parser
21+
maturin
22+
jinja2

‎docs/source/_static/images/DataFusion-Logo-Background-White.png

12.1 KB
Loading

‎docs/source/_static/images/DataFusion-Logo-Background-White.svg

+1
Loading

‎docs/source/_static/images/DataFusion-Logo-Dark.png

19.7 KB
Loading

0 commit comments

Comments
 (0)