ENH partial sorting for mi in sortlevel

[hayd]

fixes #3984.

Still need to tweak the docstrings and stuff. But I think this is working.

cc @jtratner

[ghost]

Would it be more general to overload level using types rather then adding a keyword
in 3 places? Perhaps that way supporting sorting on a "slice" of levels rather then
introducing a kw to choose between "that one" and "that one and all those that follow"?

[hayd]

I was just concerned about keeping old (strange?) behavior, I don't follow how to do this with types, but definitely the kwargs are inelegant.

[ghost]

What I meant was you can express the same operation and more
by accepting slice(3,None) or slice(3,5), or correspondingly (3,) and (3,5)
as values for level. Using types as in, rather then integers, accept tuples/slices
to express a range of levels to sort.

[hayd]

That's a nice idea. The issue is the current behaviour is to sort all when you pass a single level e.g. sortlevel('A')... maybe this shouldn't be the default and we could do something more logical (like what you suggest)?

Actually this current behaviour is a little deranged... is sortlevels the only way to sort a mi (sort doesn't work!)...

[ghost]

You'll have to give examples, I'm not sure what you mean by deranged.
It doesn't look to me like it sorts all levels at all.

You probably tried the following already:

mi=pd.MultiIndex.from_product([['A0','A1','A2'],['B0','B1','B2'],['C0','C1','C2']])
list(mi.get_values())
list(mi.sortlevel(0)[0].get_values())
list(mi.sortlevel(1)[0].get_values())
list(mi.sortlevel(2)[0].get_values())

That sorts by one and only one level, and it looks stable wrt to the other levels doing so.
mi.lexsort_depth seems to be set to 0 when sorting on inner levels, which is right.
all seems to be in order.

What are you after, preferably with explicit example?

[hayd]

"That sorts by one and only one level," here's an example, let's reverse:

In [8]: mi=pd.MultiIndex.from_product([['A2','A1','A0'],['B2','B1','B0'],['C2','C1','C0']])

In [10]: print mi.sortlevel(1)[0].get_values()
[('A0', 'B0', 'C0') ('A0', 'B0', 'C1') ('A0', 'B0', 'C2')
 ('A1', 'B0', 'C0') ('A1', 'B0', 'C1') ('A1', 'B0', 'C2')
 ('A2', 'B0', 'C0') ('A2', 'B0', 'C1') ('A2', 'B0', 'C2')
 ('A0', 'B1', 'C0') ('A0', 'B1', 'C1') ('A0', 'B1', 'C2')
 ('A1', 'B1', 'C0') ('A1', 'B1', 'C1') ('A1', 'B1', 'C2')
 ('A2', 'B1', 'C0') ('A2', 'B1', 'C1') ('A2', 'B1', 'C2')
 ('A0', 'B2', 'C0') ('A0', 'B2', 'C1') ('A0', 'B2', 'C2')
 ('A1', 'B2', 'C0') ('A1', 'B2', 'C1') ('A1', 'B2', 'C2')
 ('A2', 'B2', 'C0') ('A2', 'B2', 'C1') ('A2', 'B2', 'C2')]

[ghost]

now I see what you mean.

[hayd]

This is kindof documented in the DataFrame.sortlevel, but tbh this is more of an observation than a spec:

Sort multilevel index by chosen axis and primary level. Data will be
lexicographically sorted by the chosen level followed by the other
levels (in order)

... :s

[ghost]

Yeah, it's a tossup on which docstring is more abstruse, df.sortlevel or mi.sortlevel.

You could have level=(1,2) can sort just level 1, level=(1,3) means sort levels [1,2) ,
and level=(1,) can mean sort levels 1..end.

If you want to sort on multiple single levels, just do multiple invocations.

[ghost]

But now I understand why you chose to have the default True keyword. makes sense.

[jorisvandenbossche]

@hayd It's maybe also an option to fix this in sort/sort_index by providing there a level keyword? It does already work, but it just sorts all levels. And then you can do there the more 'logical' thing to only sort the level you specified?

[hayd]

@jorisvandenbossche I wasn't sure if this was part of @jtratner's work on refactoring index - and making the common methods.

+1 a level kwarg makes sense!

I was confused about what sort_index actually does, I thought it would sort by the index, but it appears to accept columns... i.e is just sort with an axis arg?

[jreback]

@hayd related is #5190

[jreback]

hows this coming?

[hayd]

@jreback Is the remaining thing here to add a level (and sort_remaining?) arg to sort and sort_index ?

[jreback]

I think this is fine...wouldn't really change sort/sort_index ATM.

ping me to look at this in a day or 2

[hayd]

ping

[jreback]

this was nice.... any other possibilites besides sort_remaining?

[hayd]

Not sure, happy to change, what's your issue with sort_remaining?

(sort_remaining_levels ?)

[jreback]

just sounds odd - though can't think of anything better

you are defaulting this to true right?

[hayd]

yeah, defaults to true so doesn't break backwards compat... (though perhaps we will later)

[jreback]

maybe I think it's a perf issue to completely sort

[hayd]

@jreback shall we get this in?

[jreback]

looks fine

release note / v0.14.0 mention

merge when ready

I don't think docs are really necessary as the do string covers

[jreback]

looks fine....just check release notes / v0.14

[jtratner]

Would this be more clear as:

Lexicographically sort index of dataframe on specified axis, starting with the specified level and then sorting by other levels in the order they're defined on the multilevel index (sort_remaining can optionally disable sorting on other levels).

It's not perfect, but maybe clearer?

[jtratner]

👍 from me as well. In addition to the minor suggestion on the docstring, should we add a warning about the performance hit from non-lexsorted MultiIndex? (either via level != 0 or sort_remaining=False)

[jreback]

ping!

[jreback]

ping

[jreback]

@hayd ping....need to get this in ASAP

[jreback]

ping!