Skip to content

Automatic Dask-Zarr chunk alignment #10336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

josephnowak
Copy link
Contributor

@josephnowak josephnowak commented May 19, 2025

I'm still working on some details of this PR, but as it stands, it works and demonstrates an implementation that avoids breaking changes.

At this stage, my main concerns are:

  1. Chunk Handling Utilities – It seems there are no utility functions for managing chunks, which results in possible duplicated code across the various backends. To address this, I created a class with several static methods, though I’m not convinced this is the best solution. I’m uncertain whether these functions should be placed in a separate file or incorporated into the existing common.py file for backends. Alternatively, should there be an external package that provides these chunk utilities? It’s possible the Dask team already has something similar implemented, but I haven't confirmed this yet.

  2. Backend Test Organization – The test file for backends has grown significantly in size, making it harder to manage. It might be beneficial to start adding new tests in a separate file to improve maintainability.

I’d appreciate any feedback or suggestions regarding these concerns!

@josephnowak josephnowak marked this pull request as draft May 19, 2025 17:57
@josephnowak josephnowak changed the title Automatic chunk alignment Automatic Dask-Zarr chunk alignment May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatic Dask-Zarr chunk alignment on the to_zarr method
1 participant