Upgrade Aesara version pin and unpin SciPy upper limit #5474

michaelosthege · 2022-02-15T13:44:32Z

I'm opening the PR so the CI tells me which tests break.

Closes #5446
Closes #5448

codecov · 2022-02-15T13:55:39Z

Codecov Report

Merging #5474 (bdc6e85) into main (da2030e) will increase coverage by 5.77%.
The diff coverage is 95.23%.

@@            Coverage Diff             @@
##             main    #5474      +/-   ##
==========================================
+ Coverage   81.46%   87.23%   +5.77%     
==========================================
  Files          81       81              
  Lines       14213    14247      +34     
==========================================
+ Hits        11579    12429     +850     
+ Misses       2634     1818     -816

Impacted Files	Coverage Δ
pymc/distributions/distribution.py	`90.80% <71.42%> (-0.64%)`	⬇️
pymc/aesaraf.py	`91.18% <100.00%> (+1.00%)`	⬆️
pymc/distributions/censored.py	`95.00% <100.00%> (+1.77%)`	⬆️
pymc/distributions/multivariate.py	`92.16% <100.00%> (+0.42%)`	⬆️
pymc/distributions/shape_utils.py	`97.00% <100.00%> (+0.26%)`	⬆️
pymc/util.py	`77.50% <100.00%> (+1.87%)`	⬆️
pymc/sampling_jax.py	`96.89% <0.00%> (-1.41%)`	⬇️
pymc/bart/pgbart.py	`94.92% <0.00%> (-0.32%)`	⬇️
pymc/backends/arviz.py	`90.28% <0.00%> (ø)`
pymc/distributions/continuous.py	`97.23% <0.00%> (ø)`
... and 19 more

michaelosthege

I ran only the tests that were red, so maybe I broke something. (We'll see.)

But at least the following tests are still failing, because the RV implementations need to be fixed:

FAILED pymc/tests/test_distributions.py::TestMatchesScipy::test_dirichlet_vectorized[2-a2] - ValueError: input operand has more dimensions than allowed by the axis remapping
FAILED pymc/tests/test_distributions.py::TestMatchesScipy::test_dirichlet_vectorized[size1-a2] - ValueError: shape mismatch: objects cannot be broadcast to a single shape
FAILED pymc/tests/test_distributions.py::TestMatchesScipy::test_dirichlet_vectorized[size2-a1] - ValueError: shape mismatch: objects cannot be broadcast to a single shape
FAILED pymc/tests/test_distributions.py::TestMatchesScipy::test_dirichlet_vectorized[size2-a2] - ValueError: shape mismatch: objects cannot be broadcast to a single shape
FAILED pymc/tests/test_distributions_moments.py::test_dirichlet_moment[a2-7-expected2] - TypeError: Cannot convert Type TensorType(float64, (None, None, None)) (of Variable Elemwise{true_div,no_inplace}.0) ...
FAILED pymc/tests/test_distributions_moments.py::test_dirichlet_moment[a3-size3-expected3] - TypeError: Cannot convert Type TensorType(float64, (None, None, None, None)) (of Variable Elemwise{true_div,no_in...
FAILED pymc/tests/test_distributions_moments.py::test_mv_normal_moment[mu7-cov7-size7-expected7] - ValueError: could not broadcast input array from shape (2,2) into shape (4,5,2)
FAILED pymc/tests/test_distributions_moments.py::test_mvstudentt_moment[2-mu5-cov5-2-expected5] - assert (2, 3) == (2, 2, 3)
FAILED pymc/tests/test_distributions_moments.py::test_mvstudentt_moment[2-mu6-cov6-size6-expected6] - ValueError: operands could not be broadcast together with shapes (2,5,2,3) (2,5,2,1,1)
FAILED pymc/tests/test_distributions_moments.py::test_multinomial_moment[p6-n6-2-expected6] - TypeError: Cannot convert Type TensorType(int64, (None, None, None)) (of Variable Elemwise{Cast{int64}}.0) into ...
FAILED pymc/tests/test_distributions_random.py::TestDirichletMultinomial_1D_n_2D_a::test_distribution - assert (1, 2, 4) == (1, 4)

For MvNormal and MvStudentT I think this is just the rng_fn (and get_moment).

I gotta jump now, so anybody please feel free to take over.

pymc/distributions/multivariate.py

michaelosthege · 2022-02-15T16:58:30Z

pymc/distributions/shape_utils.py

-            rv_out = change_rv_size_fn(rv_var=rv_out, new_size=shape[:-1])
+            # Into added dims 👇 plus implied dims 👇 but excluding support 👇 dims.
+            new_size = (*shape[:-1], *tuple(rv_out.shape[:ndim_actual - ndim_supp]))
+            rv_out = change_rv_size_fn(rv_var=rv_out, new_size=new_size)


The else: below may no longer be relevant, but we should first add type hints to the signature, because when I tried to un-indent the ShapeError it turned out that some of the int parameters are actually Optional[int].

@ricardoV94 it looks like this whole block in shape_utils.py is no longer covered by any tests. It is covered on main (line 661).
Can we remove it, or is there an edge case that could still trigger these branches?

is there an edge case that could still trigger these branches?

None that I know of

Looks like I messed up the ShapeWarning. I'll take a look at it in the morning.

pymc/tests/test_distributions_moments.py

pymc/tests/test_shape_handling.py

OriolAbril · 2022-02-15T20:14:12Z

You should also check https://pymc--5474.org.readthedocs.build/en/5474/learn/examples/pymc_overview.html#model-specification to make sure not only the installation finishes but that it is actually working and pymc can be imported and used (same url will work as new commits come in)

pymc/tests/test_shape_handling.py

ricardoV94 · 2022-02-22T11:05:08Z

pymc/distributions/multivariate.py

        mv_samples = multivariate_normal.rng_fn(rng=rng, mean=np.zeros_like(mu), cov=cov, size=size)

-        size = tuple(size or ())
+        # Take chi2 draws and add an axis of length 1 to the right for correct broadcasting below
+        chi2_samples = np.sqrt(rng.chisquare(nu, size=size) / nu)[..., None]


@Sayam753 can you have a second look if the new Multivariate rng_fn looks correct?

These tests were introduced between pre-existing Dirichlet/Multinomial/DirichletMultionmial tests that belong to conceptually related distributions

Closes pymc-devs#5446 Closes pymc-devs#5448

ricardoV94 · 2022-02-22T13:43:49Z

Tests are passing, and I doubled check the following suspicious (initially) non-failing distributions:

MatrixNormal
KroneckerNormal
CAR

Closes pymc-devs#5446

…tions

michaelosthege

With the last commit mypy -p pymc has zero complaints about shape_utils.py :)

pymc/util.py

pymc/distributions/shape_utils.py

pymc/distributions/distribution.py

michaelosthege added the pytensor label Feb 15, 2022

michaelosthege self-assigned this Feb 15, 2022

michaelosthege commented Feb 15, 2022

View reviewed changes

michaelosthege added the help wanted label Feb 15, 2022

ricardoV94 reviewed Feb 16, 2022

View reviewed changes

pymc/tests/test_shape_handling.py Outdated Show resolved Hide resolved

michaelosthege added the shape problem label Feb 16, 2022

ricardoV94 force-pushed the upgrade-aesara branch 4 times, most recently from c50e0fa to 4767081 Compare February 22, 2022 11:04

ricardoV94 reviewed Feb 22, 2022

View reviewed changes

ricardoV94 force-pushed the upgrade-aesara branch 3 times, most recently from 5dd186e to 211e547 Compare February 22, 2022 11:38

ricardoV94 and others added 3 commits February 22, 2022 13:04

Move stickbreaking tests

98ddca9

These tests were introduced between pre-existing Dirichlet/Multinomial/DirichletMultionmial tests that belong to conceptually related distributions

Separate transitive shape checks into separate asserts

6dbe443

Upgrade Aesara version pin and unpin SciPy upper limit

0a7d966

Closes pymc-devs#5446 Closes pymc-devs#5448

ricardoV94 force-pushed the upgrade-aesara branch from 211e547 to 3dd9e2f Compare February 22, 2022 12:54

Adapt to new size kwarg interpretation

7bb32cf

Closes pymc-devs#5446

ricardoV94 force-pushed the upgrade-aesara branch from dfaafb2 to 0d7f0c3 Compare February 22, 2022 15:03

ricardoV94 added 3 commits February 22, 2022 16:08

Fix Wishart shape inference when there are batched parameters

0dfe7f9

Update MatrixNormal shape_inference and remove shape and size restric…

0d86bdd

…tions

Fix KroneckerNormal ndim_supp

ef5653f

ricardoV94 force-pushed the upgrade-aesara branch from 0d7f0c3 to ef5653f Compare February 22, 2022 15:08

ricardoV94 marked this pull request as ready for review February 22, 2022 15:12

ricardoV94 removed the help wanted label Feb 22, 2022

michaelosthege commented Feb 22, 2022

View reviewed changes

pymc/util.py Show resolved Hide resolved

pymc/distributions/shape_utils.py Show resolved Hide resolved

michaelosthege force-pushed the upgrade-aesara branch 2 times, most recently from 4c2c095 to 2fde55e Compare February 23, 2022 20:42

ricardoV94 reviewed Feb 23, 2022

View reviewed changes

pymc/distributions/distribution.py Show resolved Hide resolved

michaelosthege mentioned this pull request Feb 23, 2022

Make VI work on v4 #4582

Merged

michaelosthege requested a review from ricardoV94 February 25, 2022 23:27

ricardoV94 reviewed Feb 26, 2022

View reviewed changes

pymc/distributions/distribution.py Outdated Show resolved Hide resolved

michaelosthege and others added 2 commits February 27, 2022 17:33

Remove maybe_resize helper function and fix some type hints

9055b1e

Add expand kwarg to Censored.change_size

bdc6e85

ricardoV94 force-pushed the upgrade-aesara branch from 2fde55e to bdc6e85 Compare February 27, 2022 16:37

ricardoV94 approved these changes Feb 27, 2022

View reviewed changes

michaelosthege merged commit 2c6abf6 into pymc-devs:main Feb 27, 2022

michaelosthege deleted the upgrade-aesara branch February 27, 2022 18:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Aesara version pin and unpin SciPy upper limit #5474

Upgrade Aesara version pin and unpin SciPy upper limit #5474

michaelosthege commented Feb 15, 2022

codecov bot commented Feb 15, 2022 •

edited

Loading

michaelosthege left a comment •

edited

Loading

michaelosthege Feb 15, 2022

michaelosthege Feb 22, 2022 •

edited

Loading

ricardoV94 Feb 22, 2022 •

edited

Loading

michaelosthege Feb 22, 2022

OriolAbril commented Feb 15, 2022 •

edited

Loading

ricardoV94 Feb 22, 2022

ricardoV94 commented Feb 22, 2022 •

edited

Loading

michaelosthege left a comment

Upgrade Aesara version pin and unpin SciPy upper limit #5474

Upgrade Aesara version pin and unpin SciPy upper limit #5474

Conversation

michaelosthege commented Feb 15, 2022

codecov bot commented Feb 15, 2022 • edited Loading

Codecov Report

michaelosthege left a comment • edited Loading

Choose a reason for hiding this comment

michaelosthege Feb 15, 2022

Choose a reason for hiding this comment

michaelosthege Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

ricardoV94 Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

michaelosthege Feb 22, 2022

Choose a reason for hiding this comment

OriolAbril commented Feb 15, 2022 • edited Loading

ricardoV94 Feb 22, 2022

Choose a reason for hiding this comment

ricardoV94 commented Feb 22, 2022 • edited Loading

michaelosthege left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 15, 2022 •

edited

Loading

michaelosthege left a comment •

edited

Loading

michaelosthege Feb 22, 2022 •

edited

Loading

ricardoV94 Feb 22, 2022 •

edited

Loading

OriolAbril commented Feb 15, 2022 •

edited

Loading

ricardoV94 commented Feb 22, 2022 •

edited

Loading