Allow Panel to expand along major and minor indexes

[kghose]

related #4971

Currently Panel will allow us to expand along the items dimension and will align cells in added DataFrames to the current major and minor indexes. However, if our DataFrame has a major or minor indexes not present in the original Panel, our append will silently fail (NaN s will populate the whole item)

I feel that a more natural behavior is for the Panel to expand along its major and minor axes to accomodate the new data.

I have to add here that I am really enjoying Panel and I find the logical progression from Series -> DataFrame -> Panel very well thought out.

Thanks.

[jreback]

you are asking for 2 different things, arithmetic alignment (see issue #4971), and arbitrary indexing, implemented here (for 0.13): http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#indexing-api-changes

can you show a use case (with code). can incoporate as tests.

[kghose]

import pandas as pd, numpy as np

#Full dimensions of a slice of our panel
index = ['1','2','3','4'] #major_index
columns = ['a','b','c'] #minor_index

df = pd.DataFrame(np.arange(12).reshape(4,3),columns=columns,index=index) #A full slice of the panel
df1 = pd.DataFrame(np.arange(4).reshape(2,2),columns=['a','b'],index=['2','4']) #A partial slice
df2 = pd.DataFrame(np.arange(4).reshape(2,2),columns=['d','e'],index=['5','6']) #Partial slice with a new column and index


pn = pd.Panel({'A': df})
pn['B'] = df1
pn['C'] = df2

for key in pn.items:
  print pn[key]

Output ->
   a   b   c   d   e
1   0   1   2 NaN NaN 
2   3   4   5 NaN NaN
3   6   7   8 NaN NaN
4   9  10  11 NaN NaN
5 NaN NaN NaN NaN NaN
6 NaN NaN NaN NaN NaN

    a   b   c   d   e
1 NaN NaN NaN NaN NaN
2   0   1 NaN NaN NaN
3 NaN NaN NaN NaN NaN
4   2   3 NaN NaN NaN
5 NaN NaN NaN NaN NaN
6 NaN NaN NaN NaN NaN

    a   b   c   d   e
1 NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN
4 NaN NaN NaN NaN NaN
5 NaN NaN NaN  0   1
6 NaN NaN NaN  2   3

I think I got the formatting right

Thanks! I should start using the dev version, instead of release!

[jreback]

fyi...you should use np.arange; pylab prob imports it for convience, but this is not a typical way of doing arange

[kghose]

Yep. I've gotten into the habit of using pylab as my swissknife, because it imports a bunch of other stuff automatically!

[jreback]

this is enabled from this issue (in 0.13): #2578

[kghose]

Thanks!

[JohnSmizz]

Hey maybe I've misread the above but to me it seems this still isn't working on v0.13.1 ? Running the code above given by kghose and displaying 'pn['C']' (to view df2) yields

    a   b   c
1 NaN NaN NaN
2 NaN NaN NaN
3 NaN NaN NaN
4 NaN NaN NaN

even though expected would be

    a   b   c   d   e
1 NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN
4 NaN NaN NaN NaN NaN
5 NaN NaN NaN  0   1
6 NaN NaN NaN  2   3

if I understand this issue correctly?

[jreback]

@JohnSmizz

no when you do an assignment (this applies to Series,DataFrame, or Panel). The object being assigned is aligned with the existing, meaning it will have the same indicies. Any additional indicies of the object being aligned (e.g. all of them in df2) are essentially discarded.

You could do what you want say with a concat which makes a union of the indicies. (Assignment with alignment is kind of like a left join)