Toggle numba caching by environment variable

[timothymillar]

Fixes #371 and #869

This is just a suggestion and I'm not sure of the best way to test this in CI unless we create an additional runner?

[codecov-commenter]

Codecov Report

Merging #870 (01e9763) into main (c13acb1) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #870   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           37        38    +1     
  Lines         3163      3170    +7     
=========================================
+ Hits          3163      3170    +7     

Impacted Files	Coverage Δ
sgkit/accelerate.py	100.00% <100.00%> (ø)
sgkit/distance/metrics.py	100.00% <100.00%> (ø)
sgkit/stats/aggregation.py	100.00% <100.00%> (ø)
sgkit/stats/conversion.py	100.00% <100.00%> (ø)
sgkit/stats/hwe.py	100.00% <100.00%> (ø)
sgkit/stats/ld.py	100.00% <100.00%> (ø)
sgkit/stats/pedigree.py	100.00% <100.00%> (ø)
sgkit/stats/popgen.py	100.00% <100.00%> (ø)
sgkit/stats/utils.py	100.00% <100.00%> (ø)
sgkit/utils.py	100.00% <100.00%> (ø)

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[jeromekelleher]

Totally agree we should do this, but I wonder if it'd be better if we just made our own local jit decorator? This would allow us to put in library-wide defaults for things like cache, as well as give us more fine-grained control of when jitting is done.

[tomwhite]

Totally agree we should do this, but I wonder if it'd be better if we just made our own local jit decorator? This would allow us to put in library-wide defaults for things like cache, as well as give us more fine-grained control of when jitting is done.

Agree - see #371.

[timothymillar]

Ah, thanks I hadn't noticed that issue. I can make a start on custom jit, vectorise and guvectorise decorators.

[jeromekelleher]

FWIW, here's what I did for an experimental project downstream of sgkit. You might find parts of it familar 😉

[tomwhite]

Nice! I'm not sure how easy it would be to completely disable numba for vectorize and guvectorize since these wrappers change the type signature in a way that is non-trivial to do in pure python. We don't need that here though (but I'll note that it would be useful for #803) - just a way to enforce numba caching consistently.

[timothymillar]

I've started with some simple wrappers for jit and guvectorize. I ran into this issue numba/numba#8231 when wrapping vectorize, but then realized that the only use of it (mine) was completely unnecessary 🤷. As @tomwhite pointed out, we can't really disable numba when using guvectorize so I'm ignoring that for now. I'm also leaving out cuda for now as I don't have convenient means to test it.

[jeromekelleher]

LGTM!

[timothymillar]

@tomwhite are you happy that this satisfies #371? Or would you prefer to also handle @cuda now?

[tomwhite]

@timothymillar sorry I was away last week on vacation and have only just got back. I'm fine leaving cuda for now.

[timothymillar]

No worries, thanks @tomwhite