SparseSeries accepts scipy.sparse.spmatrix in constructor #16617

kernc · 2017-06-06T20:22:39Z

closes API/DEPR: deprecate SparseSeries.from_coo and accept in constructor #15634
tests added / passed
passes git diff upstream/master --name-only -- '*.py' | flake8 --diff
whatsnew entry

jreback

just a quick look. this is probably going to be tricky to make work.

jreback · 2017-06-06T22:19:35Z

doc/source/whatsnew/v0.20.2.txt

@@ -25,6 +25,9 @@ Enhancements
  has been added to return the group order (:issue:`11642`); see
  :ref:`here <groupby.ngroup>`.

+
+- ``SparseSeries`` and ``SparseArray`` now support 1d ``scipy.sparse.spmatrix`` in constructor. Additionally, ``SparseDataFrame`` can be assigned columns of ``scipy.sparse.spmatrix``; see :ref:`here <sparse.scipysparse_series>`. (:issue:`15634`)


will be for 0.21.0

jreback · 2017-06-06T22:20:08Z

pandas/core/sparse/frame.py

+            else:
+                # 2d; make it iterable
+                value = list(value.tocsc().T)
+        super().__setitem__(key, value)


use the fully qualified call

jreback · 2017-06-06T22:20:30Z

pandas/core/sparse/series.py

@@ -722,6 +726,9 @@ def combine_first(self, other):

    def to_coo(self, row_levels=(0, ), column_levels=(1, ), sort_labels=False):
        """
+        DEPRECATED; instead, make a SparseSeries with a two-level index,
+        unstack it, then use .to_coo() on the resulting SparseDataFrame.


use the deprecated sphinx directive (I think we are changing these all over)

jreback · 2017-06-06T22:20:33Z

pandas/core/sparse/series.py

@@ -779,6 +786,9 @@ def to_coo(self, row_levels=(0, ), column_levels=(1, ), sort_labels=False):
    @classmethod
    def from_coo(cls, A, dense_index=False):
        """
+        DEPRECATED; instead, pass 1d scipy.sparse matrices directly into
+        SparseSeries constructor, and 2d into SparseDataFrame constructor.


…structor

kernc · 2017-06-06T23:48:01Z

this is probably going to be tricky to make work.

Why do you think so? It does seem to work for the moment. 😃

codecov · 2017-06-07T11:26:05Z

Codecov Report

Merging #16617 into master will decrease coverage by <.01%.
The diff coverage is 84.84%.

@@            Coverage Diff             @@
##           master   #16617      +/-   ##
==========================================
- Coverage   90.96%   90.95%   -0.01%     
==========================================
  Files         161      161              
  Lines       49263    49287      +24     
==========================================
+ Hits        44810    44827      +17     
- Misses       4453     4460       +7

Flag	Coverage Δ
#multiple	`88.71% <84.84%> (-0.01%)`	⬇️
#single	`40.22% <30.3%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/sparse/series.py	`95.1% <100%> (+0.03%)`	⬆️
pandas/core/indexing.py	`94.01% <100%> (ø)`	⬆️
pandas/core/sparse/frame.py	`94.77% <100%> (+0.51%)`	⬆️
pandas/core/sparse/array.py	`91.62% <100%> (+0.16%)`	⬆️
pandas/core/internals.py	`93.56% <54.54%> (+0.13%)`	⬆️
pandas/plotting/_converter.py	`63.23% <0%> (-1.82%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10c17d4...293bb47. Read the comment docs.

codecov · 2017-06-07T11:26:07Z

Codecov Report

Merging #16617 into master will decrease coverage by <.01%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master   #16617      +/-   ##
==========================================
- Coverage   90.96%   90.95%   -0.01%     
==========================================
  Files         161      161              
  Lines       49263    49287      +24     
==========================================
+ Hits        44810    44827      +17     
- Misses       4453     4460       +7

Flag	Coverage Δ
#multiple	`88.71% <85.71%> (-0.01%)`	⬇️
#single	`40.22% <28.57%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/sparse/frame.py	`94.77% <100%> (+0.51%)`	⬆️
pandas/core/sparse/array.py	`91.62% <100%> (+0.16%)`	⬆️
pandas/core/sparse/series.py	`95.1% <100%> (+0.03%)`	⬆️
pandas/core/indexing.py	`94.01% <100%> (ø)`	⬆️
pandas/core/internals.py	`93.56% <61.53%> (+0.13%)`	⬆️
pandas/plotting/_converter.py	`63.23% <0%> (-1.82%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10c17d4...ef03e73. Read the comment docs.

In Python 2, inner-block anonymous exception seems to overwrite the outer-block anonymous exception. We're supposed to re-raise the latter.

jreback

looks pretty good. a couple of comments.

jreback · 2017-06-08T10:50:19Z

doc/source/sparse.rst

+   sdf[['z', 'w']] = sp_arr[:, [7, 8]]
+   sdf.iloc[:, -5:]
+
+Below interface is deprecated.


say that this is deprecated in 0.21.0

jreback · 2017-06-08T10:50:32Z

doc/source/whatsnew/v0.21.0.txt

@@ -25,6 +25,10 @@ New features
 - Added `__fspath__` method to :class`:pandas.HDFStore`, :class:`pandas.ExcelFile`,
  and :class:`pandas.ExcelWriter` to work properly with the file system path protocol (:issue:`13823`)

+- ``SparseSeries`` and ``SparseArray`` now support 1d ``scipy.sparse.spmatrix`` in constructor.


in the constructor

jreback · 2017-06-08T10:51:05Z

doc/source/whatsnew/v0.21.0.txt

@@ -25,6 +25,10 @@ New features
 - Added `__fspath__` method to :class`:pandas.HDFStore`, :class:`pandas.ExcelFile`,
  and :class:`pandas.ExcelWriter` to work properly with the file system path protocol (:issue:`13823`)

+- ``SparseSeries`` and ``SparseArray`` now support 1d ``scipy.sparse.spmatrix`` in constructor.
+  Additionally, ``SparseDataFrame`` can be assigned columns of ``scipy.sparse.spmatrix``;


make this 2nd sentence a separate bullet point (you can use same issue on both of them, or 2nd one should be the PR number maybe)

jreback · 2017-06-08T10:52:27Z

pandas/core/sparse/frame.py

+                                    kind=self._default_kind)
+            else:
+                # 2d; make it iterable
+                value = list(value.tocsc().T)


does this materialize?

jreback · 2017-06-08T10:53:41Z

pandas/tests/sparse/test_frame.py

+        spm = csr_matrix(np.arange(len(sdf))).T
+        sdf['X'] = spm
+        assert _equal(sdf[['X']].to_coo(), spm)
+


this comparision on the scipy side is fine, but also let's compare with assert_sparse_series/frame_equal

jreback · 2017-06-08T10:54:37Z

pandas/tests/sparse/test_frame.py

+
+        # 1d row -- changing series contents not yet supported
+        spm = csr_matrix(np.arange(sdf.shape[1], dtype=float))
+        idx = np.zeros(sdf.shape[0], dtype=bool)


can you test with .loc/.iloc as well (might already be another issue about this, if not and its too complicated for here, then create a new issue)

jreback · 2017-07-19T10:32:57Z

can you rebase / update

jreback · 2017-09-10T14:48:56Z

@kernc if you have time can you rebase / update

kernc · 2017-09-10T17:11:23Z

For some reason (probably assert_raises in test_setitem_spmatrix()), I had a while ago decided to first implement setitem on SparseSeries that I have 80% done waiting in a feature branch somewhere. If you insist, I can push this first, however.

jreback · 2017-09-10T17:21:49Z

oh we can do the other first
was just looking thru open prs

what was that number?

kernc · 2017-10-04T16:45:31Z

Right, it's #17785.

jreback · 2018-01-21T18:20:45Z

closing as stale. if you want to continue working, pls ping.

kernc added 5 commits June 6, 2017 21:34

PERF: avoid unnecessary array copy

639fc6f

ENH: SparseArray constructor supports 1d scipy.sparse.spmatrix input

690a09f

ENH: Series constructor supports 1d scipy.sparse.spmatrix input

9d6d2fe

ENH: SparseDataFrame supports scipy.sparse.spmatrix in setitem

3a12685

DOC: Document scipy sparse matrix accepted in SparseSeries constructor

6bc8c8a

jreback requested changes Jun 6, 2017

View reviewed changes

jreback added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Sparse Sparse Data Type labels Jun 6, 2017

fixup! DOC: Document scipy sparse matrix accepted in SparseSeries con…

97da8bd

…structor

kernc force-pushed the scipy-sparse branch from e8b4e07 to a91f6ea Compare June 7, 2017 10:45

fixup! ENH: SparseDataFrame supports scipy.sparse.spmatrix in setitem

293bb47

kernc force-pushed the scipy-sparse branch from a91f6ea to 293bb47 Compare June 7, 2017 11:25

kernc added 2 commits June 7, 2017 18:17

Fix tests.sparse.test_frame.TestSparseDataFrame.test_setitem_spmatrix

47ef68a

In Python 2, inner-block anonymous exception seems to overwrite the outer-block anonymous exception. We're supposed to re-raise the latter.

Fix CirleCI test on ancient SciPy

ef03e73

jreback requested changes Jun 8, 2017

View reviewed changes

jsexauer mentioned this pull request Jun 8, 2017

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

TomAugspurger added this to the 0.21.0 milestone Jun 30, 2017

jreback removed this from the 0.21.0 milestone Sep 23, 2017

jreback closed this Jan 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SparseSeries accepts scipy.sparse.spmatrix in constructor #16617

SparseSeries accepts scipy.sparse.spmatrix in constructor #16617

kernc commented Jun 6, 2017

jreback left a comment

jreback Jun 6, 2017

jreback Jun 6, 2017

jreback Jun 6, 2017

jreback Jun 6, 2017

kernc commented Jun 6, 2017 •

edited

Loading

codecov bot commented Jun 7, 2017

codecov bot commented Jun 7, 2017 •

edited

Loading

jreback left a comment

jreback Jun 8, 2017

jreback Jun 8, 2017

jreback Jun 8, 2017

jreback Jun 8, 2017

jreback Jun 8, 2017

jreback Jun 8, 2017

jreback commented Jul 19, 2017

jreback commented Sep 10, 2017

kernc commented Sep 10, 2017

jreback commented Sep 10, 2017

kernc commented Oct 4, 2017

jreback commented Jan 21, 2018

SparseSeries accepts scipy.sparse.spmatrix in constructor #16617

SparseSeries accepts scipy.sparse.spmatrix in constructor #16617

Conversation

kernc commented Jun 6, 2017

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kernc commented Jun 6, 2017 • edited Loading

codecov bot commented Jun 7, 2017

Codecov Report

codecov bot commented Jun 7, 2017 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jul 19, 2017

jreback commented Sep 10, 2017

kernc commented Sep 10, 2017

jreback commented Sep 10, 2017

kernc commented Oct 4, 2017

jreback commented Jan 21, 2018

kernc commented Jun 6, 2017 •

edited

Loading

codecov bot commented Jun 7, 2017 •

edited

Loading