Skip to content

Faster dask unstack #6150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

Faster dask unstack #6150

wants to merge 4 commits into from

Conversation

Illviljan
Copy link
Contributor

@Illviljan Illviljan commented Jan 10, 2022

ref #5582

  • Closes #xxxx
  • Tests added
  • User visible changes (including notable bug fixes) are documented in whats-new.rst
  • New functions/methods are listed in api.rst

@Illviljan Illviljan added the run-benchmark Run the ASV benchmark workflow label Jan 10, 2022
@Illviljan
Copy link
Contributor Author

Hmm, this way doesn't seem faster on first glance?

[ 72.78%] ··· unstacking.Unstacking.time_unstack_fast                1.91±0.01ms
[ 72.96%] ··· unstacking.Unstacking.time_unstack_pandas_slow          13.9±0.1ms
[ 73.15%] ··· unstacking.Unstacking.time_unstack_slow                1.91±0.02ms
[ 73.33%] ··· unstacking.UnstackingDask.time_unstack_fast            21.8±0.08ms
[ 73.52%] ··· ...king.UnstackingDask.time_unstack_pandas_slow         14.6±0.1ms
[ 73.70%] ··· unstacking.UnstackingDask.time_unstack_slow            1.94±0.02ms
[ 73.89%] ··· ...nstackingSparse.peakmem_unstack_to_sparse_2d               204M
[ 74.07%] ··· ...nstackingSparse.peakmem_unstack_to_sparse_3d               204M
[ 74.26%] ··· unstacking.UnstackingSparse.time_unstack_fast           16.4±0.2ms
[ 74.44%] ··· ...ng.UnstackingSparse.time_unstack_pandas_slow         58.3±0.2ns
[ 74.63%] ··· unstacking.UnstackingSparse.time_unstack_slow              1.20±0s
[ 74.81%] ··· ...g.UnstackingSparse.time_unstack_to_sparse_2d            444±5μs
[ 75.00%] ··· ...g.UnstackingSparse.time_unstack_to_sparse_3d        4.36±0.02ms

[ 97.78%] ··· unstacking.Unstacking.time_unstack_fast                1.85±0.09ms
[ 97.96%] ··· unstacking.Unstacking.time_unstack_pandas_slow          13.2±0.6ms
[ 98.15%] ··· unstacking.Unstacking.time_unstack_slow                   3.80±2ms
[ 98.33%] ··· unstacking.UnstackingDask.time_unstack_fast              40.7±20ms
[ 98.52%] ··· ...king.UnstackingDask.time_unstack_pandas_slow          25.2±10ms
[ 98.70%] ··· unstacking.UnstackingDask.time_unstack_slow               4.00±2ms
[ 98.89%] ··· ...nstackingSparse.peakmem_unstack_to_sparse_2d               204M
[ 99.07%] ··· ...nstackingSparse.peakmem_unstack_to_sparse_3d               204M
[ 99.26%] ··· unstacking.UnstackingSparse.time_unstack_fast            28.1±10ms
[ 99.44%] ··· ...ng.UnstackingSparse.time_unstack_pandas_slow           113±60ns
+ grep 'Traceback \|failed\|PERFORMANCE DECREASED' benchmarks.log
[ 99.63%] ··· unstacking.UnstackingSparse.time_unstack_slow            1.09±0.6s
[ 99.81%] ··· ...g.UnstackingSparse.time_unstack_to_sparse_2d          864±400μs
[100.00%] ··· ...g.UnstackingSparse.time_unstack_to_sparse_3d           8.56±4ms

@dcherian
Copy link
Contributor

ravel_multi_index can be slow. Maybe that's it?

@Illviljan Illviljan closed this Aug 12, 2022
@Illviljan Illviljan deleted the unstack_dask branch August 12, 2022 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-benchmark Run the ASV benchmark workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants