Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict clipping of DataFrame.corr only when cov=False #61214

Merged
merged 17 commits into from
Apr 3, 2025
Merged

Restrict clipping of DataFrame.corr only when cov=False #61214

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d63760a
clip correlation coefficient between -1 and 1
jhendricks487 Mar 20, 2025
1ffedbe
Added test to check if corr within bounds
jhendricks487 Mar 20, 2025
a997140
Added tuple to mistyped parameter
jhendricks487 Mar 20, 2025
7f2fdaa
Transfered np.clip to algos.nancorr
jhendricks487 Mar 20, 2025
34eb701
Clip covxy / divsor instead of result
jhendricks487 Mar 20, 2025
fc5ccae
Clip covxy / divsor within nogil
jhendricks487 Mar 20, 2025
904c892
Merge branch 'main' into clip-corr-edge-cases
j-hendricks Mar 21, 2025
c83ccaf
Merge branch 'main' into clip-corr-edge-cases
j-hendricks Mar 24, 2025
2a2186d
Merge remote-tracking branch 'upstream/main' into clip-corr-edge-cases
jhendricks487 Mar 25, 2025
ff2adc6
Added whatsnew note
jhendricks487 Mar 25, 2025
f0573e1
Replaced long entry with single entry
jhendricks487 Mar 25, 2025
7c93140
restricted clipping to when cov=False
jhendricks487 Apr 2, 2025
cf07e2c
Merge branch 'main' into clip-corr-edge-cases
j-hendricks Apr 2, 2025
23eb852
Added test for covariance
jhendricks487 Apr 3, 2025
3d873a0
Added test for covariance
jhendricks487 Apr 3, 2025
f0767f0
Changed name of test variables
jhendricks487 Apr 3, 2025
4a3b605
fixed test case for cov
jhendricks487 Apr 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions pandas/_libs/algos.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,10 +391,11 @@ def nancorr(const float64_t[:, :] mat, bint cov=False, minp=None):
# clip `covxy / divisor` to ensure coeff is within bounds
if divisor != 0:
val = covxy / divisor
if val > 1.0:
val = 1.0
elif val < -1.0:
val = -1.0
if not cov:
if val > 1.0:
val = 1.0
elif val < -1.0:
val = -1.0
result[xi, yi] = result[yi, xi] = val
else:
result[xi, yi] = result[yi, xi] = NaN
Expand Down
6 changes: 6 additions & 0 deletions pandas/tests/frame/methods/test_cov_corr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -497,3 +497,9 @@ def test_corr_within_bounds(self):
corr_matrix = df2.corr()
assert corr_matrix.min().min() >= -1.0
assert corr_matrix.max().max() <= 1.0

def test_cov_with_missing_values(self):
df = DataFrame({"A": [1, 2, None, 4], "B": [2, 4, None, 9]})
expected = DataFrame({"A": [1.0, 1.0], "B": [1.0, 1.0]})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the the expected dataframe needs index=["A", "B"]: https://github.com/pandas-dev/pandas/actions/runs/14250783260/job/39942795933?pr=61214#step:5:45

And can you confirm that 1 is the expected value? If the 2.2.3 behavior is correct, then the values in #61154 (comment) were different:

Out[68]:
          A     B
A  2.333333   5.5
B  5.500000  13.0

I don't know whether it matters, but it might be worth testing both df.cov() and df.dropna().cov()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing now. Thanks!

result = df.dropna().cov()
tm.assert_frame_equal(result, expected)
Loading