Skip to content

Commit 44f5f3f

Browse files
dcheriandcherian
committed
Merge branch 'main' into topk
* main: Revert "Support first, last with datetime, timedelta (#402)" (#404) Support first, last with datetime, timedelta (#402) Bump codecov/codecov-action from 4.5.0 to 4.6.0 (#401) Bump mamba-org/setup-micromamba from 1 to 2 (#400) Revert "[revert] test with Xarray PR branch" (#393) [pre-commit.ci] pre-commit autoupdate (#399) Faster subsetting for cohorts (#397) Fix default int on windows, numpy<2 (#395) Avoid rechunking when preferred_method="blockwise" (#394) Preserve dtype better when specified. (#389) Drop python 3.9, use ruff (#392) silence warning (#390) Expand groupby_reduce property tests (#385) Fix bug with NaNs in `by` and method='blockwise' (#384) Avoid explicit np.nan, np.inf (#383)
2 parents 7056d18 + 3853101 commit 44f5f3f

32 files changed

+607
-346
lines changed

‎.github/workflows/benchmarks.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
fetch-depth: 0
2323

2424
- name: Set up conda environment
25-
uses: mamba-org/setup-micromamba@v1
25+
uses: mamba-org/setup-micromamba@v2
2626
with:
2727
environment-name: flox-bench
2828
create-args: >-

‎.github/workflows/ci-additional.yaml

+4-4
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ jobs:
5353
echo "TODAY=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
5454
5555
- name: Setup micromamba
56-
uses: mamba-org/setup-micromamba@v1
56+
uses: mamba-org/setup-micromamba@v2
5757
with:
5858
environment-file: ${{env.CONDA_ENV_FILE}}
5959
environment-name: flox-tests
@@ -77,7 +77,7 @@ jobs:
7777
--ignore flox/tests \
7878
--cov=./ --cov-report=xml
7979
- name: Upload code coverage to Codecov
80-
uses: codecov/codecov-action@v4.5.0
80+
uses: codecov/codecov-action@v4.6.0
8181
with:
8282
file: ./coverage.xml
8383
flags: unittests
@@ -106,7 +106,7 @@ jobs:
106106
run: |
107107
echo "TODAY=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
108108
- name: Setup micromamba
109-
uses: mamba-org/setup-micromamba@v1
109+
uses: mamba-org/setup-micromamba@v2
110110
with:
111111
environment-file: ${{env.CONDA_ENV_FILE}}
112112
environment-name: flox-tests
@@ -132,7 +132,7 @@ jobs:
132132
python -m mypy --install-types --non-interactive --cache-dir=.mypy_cache/ --cobertura-xml-report mypy_report
133133
134134
- name: Upload mypy coverage to Codecov
135-
uses: codecov/codecov-action@v4.5.0
135+
uses: codecov/codecov-action@v4.6.0
136136
with:
137137
file: mypy_report/cobertura.xml
138138
flags: mypy

‎.github/workflows/ci.yaml

+9-5
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ jobs:
2626
matrix:
2727
os: ["ubuntu-latest"]
2828
env: ["environment"]
29-
python-version: ["3.9", "3.12"]
29+
python-version: ["3.10", "3.12"]
3030
include:
3131
- os: "windows-latest"
3232
env: "environment"
@@ -36,7 +36,10 @@ jobs:
3636
python-version: "3.12"
3737
- os: "ubuntu-latest"
3838
env: "minimal-requirements"
39-
python-version: "3.9"
39+
python-version: "3.10"
40+
- os: "windows-latest"
41+
env: "env-numpy1"
42+
python-version: "3.10"
4043
steps:
4144
- uses: actions/checkout@v4
4245
with:
@@ -45,7 +48,7 @@ jobs:
4548
run: |
4649
echo "PYTHON_VERSION=${{ matrix.python-version }}" >> $GITHUB_ENV
4750
- name: Set up conda environment
48-
uses: mamba-org/setup-micromamba@v1
51+
uses: mamba-org/setup-micromamba@v2
4952
with:
5053
environment-file: ci/${{ matrix.env }}.yml
5154
environment-name: flox-tests
@@ -70,9 +73,10 @@ jobs:
7073
- name: Run Tests
7174
id: status
7275
run: |
76+
python -c "import xarray; xarray.show_versions()"
7377
pytest --durations=20 --durations-min=0.5 -n auto --cov=./ --cov-report=xml --hypothesis-profile ci
7478
- name: Upload code coverage to Codecov
75-
uses: codecov/codecov-action@v4.5.0
79+
uses: codecov/codecov-action@v4.6.0
7680
with:
7781
file: ./coverage.xml
7882
flags: unittests
@@ -101,7 +105,7 @@ jobs:
101105
repository: "pydata/xarray"
102106
fetch-depth: 0 # Fetch all history for all branches and tags.
103107
- name: Set up conda environment
104-
uses: mamba-org/setup-micromamba@v1
108+
uses: mamba-org/setup-micromamba@v2
105109
with:
106110
environment-file: ci/requirements/environment.yml
107111
environment-name: xarray-tests

‎.github/workflows/upstream-dev-ci.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ jobs:
4343
run: |
4444
echo "PYTHON_VERSION=${{ matrix.python-version }}" >> $GITHUB_ENV
4545
- name: Set up conda environment
46-
uses: mamba-org/setup-micromamba@v1
46+
uses: mamba-org/setup-micromamba@v2
4747
with:
4848
environment-name: flox-tests
4949
init-shell: bash

‎.pre-commit-config.yaml

+5-16
Original file line numberDiff line numberDiff line change
@@ -4,29 +4,25 @@ ci:
44
repos:
55
- repo: https://github.com/astral-sh/ruff-pre-commit
66
# Ruff version.
7-
rev: "v0.5.0"
7+
rev: "v0.6.9"
88
hooks:
99
- id: ruff
1010
args: ["--fix", "--show-fixes"]
11+
- id: ruff-format
1112

1213
- repo: https://github.com/pre-commit/mirrors-prettier
1314
rev: "v4.0.0-alpha.8"
1415
hooks:
1516
- id: prettier
1617

1718
- repo: https://github.com/pre-commit/pre-commit-hooks
18-
rev: v4.6.0
19+
rev: v5.0.0
1920
hooks:
2021
- id: check-yaml
2122
- id: trailing-whitespace
2223
- id: end-of-file-fixer
2324
- id: check-docstring-first
2425

25-
- repo: https://github.com/psf/black-pre-commit-mirror
26-
rev: 24.4.2
27-
hooks:
28-
- id: black
29-
3026
- repo: https://github.com/executablebooks/mdformat
3127
rev: 0.7.17
3228
hooks:
@@ -35,13 +31,6 @@ repos:
3531
- mdformat-black
3632
- mdformat-myst
3733

38-
- repo: https://github.com/nbQA-dev/nbQA
39-
rev: 1.8.5
40-
hooks:
41-
- id: nbqa-black
42-
- id: nbqa-ruff
43-
args: [--fix]
44-
4534
- repo: https://github.com/kynan/nbstripout
4635
rev: 0.7.1
4736
hooks:
@@ -56,12 +45,12 @@ repos:
5645
- tomli
5746

5847
- repo: https://github.com/abravalheri/validate-pyproject
59-
rev: v0.18
48+
rev: v0.20.2
6049
hooks:
6150
- id: validate-pyproject
6251

6352
- repo: https://github.com/rhysd/actionlint
64-
rev: v1.7.1
53+
rev: v1.7.3
6554
hooks:
6655
- id: actionlint
6756
files: ".github/workflows/"

‎asv_bench/benchmarks/__init__.py

-1
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ def _skip_slow():
2121
>>> from . import _skip_slow
2222
>>> def time_something_slow():
2323
... pass
24-
...
2524
>>> time_something.setup = _skip_slow
2625
"""
2726
if os.environ.get("ASV_SKIP_SLOW", "0") == "1":

‎asv_bench/benchmarks/cohorts.py

+19-6
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,12 @@ def track_num_layers(self):
6767
track_num_tasks.unit = "tasks" # type: ignore[attr-defined] # Lazy
6868
track_num_tasks_optimized.unit = "tasks" # type: ignore[attr-defined] # Lazy
6969
track_num_layers.unit = "layers" # type: ignore[attr-defined] # Lazy
70-
for f in [track_num_tasks, track_num_tasks_optimized, track_num_layers, track_num_cohorts]:
70+
for f in [
71+
track_num_tasks,
72+
track_num_tasks_optimized,
73+
track_num_layers,
74+
track_num_cohorts,
75+
]:
7176
f.repeat = 1 # type: ignore[attr-defined] # Lazy
7277
f.rounds = 1 # type: ignore[attr-defined] # Lazy
7378
f.number = 1 # type: ignore[attr-defined] # Lazy
@@ -82,9 +87,7 @@ def setup(self, *args, **kwargs):
8287
y = np.repeat(np.arange(30), 60)
8388
by = x[np.newaxis, :] * y[:, np.newaxis]
8489

85-
self.by = flox.core._factorize_multiple((by,), expected_groups=(None,), any_by_dask=False)[
86-
0
87-
][0]
90+
self.by = flox.core._factorize_multiple((by,), expected_groups=(None,), any_by_dask=False)[0][0]
8891

8992
self.array = dask.array.ones(self.by.shape, chunks=(350, 350))
9093
self.axis = (-2, -1)
@@ -101,7 +104,12 @@ def __init__(self, *args, **kwargs):
101104

102105
def rechunk(self):
103106
self.array = flox.core.rechunk_for_cohorts(
104-
self.array, -1, self.by, force_new_chunk_at=[1], chunksize=48, ignore_old_chunks=True
107+
self.array,
108+
-1,
109+
self.by,
110+
force_new_chunk_at=[1],
111+
chunksize=48,
112+
ignore_old_chunks=True,
105113
)
106114

107115

@@ -151,7 +159,12 @@ def setup(self, *args, **kwargs):
151159

152160
def rechunk(self):
153161
self.array = flox.core.rechunk_for_cohorts(
154-
self.array, -1, self.by, force_new_chunk_at=[1], chunksize=4, ignore_old_chunks=True
162+
self.array,
163+
-1,
164+
self.by,
165+
force_new_chunk_at=[1],
166+
chunksize=4,
167+
ignore_old_chunks=True,
155168
)
156169

157170

‎asv_bench/benchmarks/combine.py

+2-6
Original file line numberDiff line numberDiff line change
@@ -65,12 +65,8 @@ def construct_member(groups) -> dict[str, Any]:
6565
* 2
6666
]
6767

68-
self.x_chunk_reindexed = [
69-
construct_member(groups) for groups in [np.array((1, 2, 3, 4))] * 4
70-
]
68+
self.x_chunk_reindexed = [construct_member(groups) for groups in [np.array((1, 2, 3, 4))] * 4]
7169
self.kwargs = {
72-
"agg": flox.aggregations._initialize_aggregation(
73-
"sum", "float64", np.float64, 0, 0, {}
74-
),
70+
"agg": flox.aggregations._initialize_aggregation("sum", "float64", np.float64, 0, 0, {}),
7571
"axis": (3,),
7672
}

‎asv_bench/benchmarks/reduce.py

+6-4
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,11 @@
77

88
N = 3000
99
funcs = ["sum", "nansum", "mean", "nanmean", "max", "nanmax", "count"]
10-
engines = [None, "flox", "numpy"] # numbagg is disabled for now since it takes ages in CI
10+
engines = [
11+
None,
12+
"flox",
13+
"numpy",
14+
] # numbagg is disabled for now since it takes ages in CI
1115
expected_groups = {
1216
"None": None,
1317
"bins": pd.IntervalIndex.from_breaks([1, 2, 4]),
@@ -17,9 +21,7 @@
1721
NUMBAGG_FUNCS = ["nansum", "nanmean", "nanmax", "count", "all"]
1822
numbagg_skip = []
1923
for name in expected_names:
20-
numbagg_skip.extend(
21-
list((func, name, "numbagg") for func in funcs if func not in NUMBAGG_FUNCS)
22-
)
24+
numbagg_skip.extend(list((func, name, "numbagg") for func in funcs if func not in NUMBAGG_FUNCS))
2325

2426

2527
def setup_jit():

‎ci/benchmark.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ dependencies:
66
- build
77
- cachey
88
- dask-core
9-
- numpy>=1.22
9+
- numpy<2
1010
- mamba
1111
- pip
1212
- python=3.10

‎ci/docs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ dependencies:
1616
- myst-parser
1717
- myst-nb
1818
- sphinx
19+
- sphinx-remove-toctrees
1920
- furo>=2024.08
2021
- ipykernel
2122
- jupyter

‎ci/env-numpy1.yml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: flox-tests
2+
channels:
3+
- conda-forge
4+
dependencies:
5+
- asv
6+
- cachey
7+
- cftime
8+
- codecov
9+
- cubed>=0.14.3
10+
- dask-core
11+
- pandas
12+
- numpy<2
13+
- scipy
14+
- lxml # for mypy coverage report
15+
- matplotlib
16+
- pip
17+
- pytest
18+
- pytest-cov
19+
- pytest-pretty
20+
- pytest-xdist
21+
- syrupy
22+
- pre-commit
23+
- numpy_groupies>=0.9.19
24+
- pooch
25+
- toolz
26+
- numba
27+
- numbagg>=0.3
28+
- hypothesis
29+
- pip:
30+
- git+https://github.com/dcherian/xarray.git

‎ci/environment.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,12 @@ dependencies:
1919
- pytest-pretty
2020
- pytest-xdist
2121
- syrupy
22-
- xarray
2322
- pre-commit
2423
- numpy_groupies>=0.9.19
2524
- pooch
2625
- toolz
2726
- numba
2827
- numbagg>=0.3
2928
- hypothesis
29+
- pip:
30+
- git+https://github.com/dcherian/xarray.git

‎ci/no-dask.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,12 @@ dependencies:
1414
- pytest-pretty
1515
- pytest-xdist
1616
- syrupy
17-
- xarray
1817
- numpydoc
1918
- pre-commit
2019
- numpy_groupies>=0.9.19
2120
- pooch
2221
- toolz
2322
- numba
2423
- numbagg>=0.3
24+
- pip:
25+
- git+https://github.com/dcherian/xarray.git

‎ci/no-numba.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,9 @@ dependencies:
1919
- pytest-pretty
2020
- pytest-xdist
2121
- syrupy
22-
- xarray
2322
- pre-commit
2423
- numpy_groupies>=0.9.19
2524
- pooch
2625
- toolz
26+
- pip:
27+
- git+https://github.com/dcherian/xarray.git

‎docs/source/conf.py

+3
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"sphinx.ext.napoleon",
4141
"myst_nb",
4242
"sphinx_codeautolink",
43+
"sphinx_remove_toctrees",
4344
]
4445

4546
codeautolink_concat_default = True
@@ -54,6 +55,8 @@
5455
master_doc = "index"
5556
language = "en"
5657

58+
remove_from_toctrees = ["generated/*"]
59+
5760
# General information about the project.
5861
project = "flox"
5962
current_year = datetime.datetime.now().year

‎docs/source/user-stories/climatology-hourly.ipynb

-1
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,6 @@
9292
"%load_ext watermark\n",
9393
"\n",
9494
"\n",
95-
"\n",
9695
"%watermark -iv"
9796
]
9897
},

‎flox/__init__.py

+7-1
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,15 @@
11
#!/usr/bin/env python
22
# flake8: noqa
33
"""Top-level module for flox ."""
4+
45
from . import cache
56
from .aggregations import Aggregation, Scan # noqa
6-
from .core import groupby_reduce, groupby_scan, rechunk_for_blockwise, rechunk_for_cohorts # noqa
7+
from .core import (
8+
groupby_reduce,
9+
groupby_scan,
10+
rechunk_for_blockwise,
11+
rechunk_for_cohorts,
12+
) # noqa
713

814

915
def _get_version():

0 commit comments

Comments
 (0)