BUG: .date and timezones

[jreback]

from SO

In [1]: s = Series(pd.to_datetime([1396670200, 1405733045, 1448248441],unit='s')).dt.tz_localize('UTC').dt.tz_convert('US/Eastern')

In [2]: s
Out[2]: 
0   2014-04-04 23:56:40-04:00
1   2014-07-18 21:24:05-04:00
2   2015-11-22 22:14:01-05:00
dtype: datetime64[ns, US/Eastern]

These are correct as they are in the local timezone, but we are not propogating the tz here and just returning a naive date (in local zone)

In [3]: s.dt.date
Out[3]: 
0    2014-04-04
1    2014-07-18
2    2015-11-22
dtype: object

In [4]: s.dt.date[0]
Out[4]: datetime.date(2014, 4, 4)

These are converted to UTC, then returning a naive date
These should be local & with a timezone attached

In [5]: s.apply(lambda x: x.date())
Out[5]: 
0    2014-04-05
1    2014-07-19
2    2015-11-23
dtype: object

further, test

s.apply(lambda x: x)

[rockg]

Python datetime.date is naive so I think that is the right thing. However, clearly the second one needs to be localized.

[rockg]

This I believe is the same as #11800. I have the tests ready to go once apply is changed as mentioned there.

[jreback]

@rockg did you have a PR for this?

[rockg]

I have not pushed it...will do so by tomorrow morning.

[jreback]

gr8 thanks

[rockg]

I thought this would be fixed by #11564 but it is still around. Was that your understanding?

>>> s.apply(lambda x: x.date())
Out[1]: 
0    2014-04-05
1    2014-07-19
2    2015-11-23
dtype: object

[jreback]

that was much more focused on timeseltas
this is a separate (smaller issue) I think

[rockg]

All right...I need some direction then. This is the problem spot. Basically it gets the datetime64s out and converts those into Timestamps all of which are tz-unaware. One solution is to remove the tz like tz_localize(None) and then get values from that (there doesn't seem to be tz_localize on a datetime64 series). Other thoughts?

[jreback]

oh, that's non-conformant code, maybe something like:

        if needs_i8_conversion(values.dtype):
            f = lambda v: i8_boxer(v)
        mapped = lib.map_infer(values, f, convert=convert_dtype)

[jorisvandenbossche]

@rockg Did you have something?
Otherwise will push this to the next release

[rockg]

I think this was actually fixed by some other pull requests (perhaps the referenced issues), but I cannot test for a couple days. If not fixed, then you can push.

[jorisvandenbossche]

Hmm, not really clear what was actually the issue. This example from @jreback:

In [3]: s.dt.date
Out[3]: 
0    2014-04-04
1    2014-07-18
2    2015-11-22
dtype: object

Isn't this the correct and expected output? (master still gives the same)

The second issues was the apply version:

In [5]: s.apply(lambda x: x.date())
Out[5]: 
0    2014-04-05
1    2014-07-19
2    2015-11-23
dtype: object

where the UTC dates are given. This seems to be fixed in master:

In [15]: pd.__version__
Out[15]: '0.18.1+387.gae4ffac'

In [16]: s.apply(lambda x: x.date())
Out[16]: 
0    2014-04-04
1    2014-07-18
2    2015-11-22
dtype: object

[rockg]

Is s equal to [1] above and in US/Eastern? If so, then master is now correct. Before it was converting under the hood to UTC and then taking date of that which was wrong.

[jorisvandenbossche]

Yes, I used the s from [1] at top of this issue. But note that the Out[3] of my previous post was just a copy of the the top post, so it was already like that when this issue was opened. So therefore I am somewhat confused (as this seems the correct output to me)

[rockg]

Yes, [3] is right. There cannot be timezones attached to date objects. For me the real issue was that apply was not returning the correct date in the local timezone (instead it was using the UTC date).

[rockg]

And that issue is now fixed so I think this can be closed.

[jreback]

I don't think this is fixed
the timestamps are converted to naive in
s.dt.date but should have the original timezone (as datetime.datetime)

[rockg]

I always understood date to return python date objects. normalizereturns midnight timestamps so I think they are meant to be fundamentally different.

[jreback]

that's right they should be datetime.datetime

WITH a tzinfo

[jreback]

they r missing the tzinfo

[rockg]

But I'm saying they should by python date objects so without a timezone.

[rockg]

Just like if you have a datetime.datetime and call date on it, you get the date object back. There are other ways to get midnight such as normalize. This is a change and I don't really see the purpose to it.

[jreback]

no that doesn't make sense - u have lost the tz for no reason (this is of course for a tz aware series - obviously a tz naive one this is already correct)

[jreback]

try date on a tz-aware datetime.datetime

[rockg]

>>> ts = pd.Timestamp("2016-1-1", tz="US/Eastern")
>>> ts
Timestamp('2016-01-01 00:00:00-0500', tz='US/Eastern')
>>> ts.date()
datetime.date(2016, 1, 1)

[rockg]

And raw datetime:

>>> import pytz
>>> from datetime import datetime
>>> d = datetime(2016, 1, 1)
>>> tz = pytz.timezone("US/Eastern")
>>> dtz = tz.localize(d)
>>> dtz
datetime.datetime(2016, 1, 1, 0, 0, tzinfo=<DstTzInfo 'US/Eastern' EST-1 day, 19:00:00 STD>)
>>> dtz.date()
datetime.date(2016, 1, 1)
>>> 

[jreback]

so the issue is that we alway have Timestamps so we always have tz, and thus can preserve it. Ok I guess just need tests to validate that we are returning the local tz date then.

[jorisvandenbossche]

@jreback So you propose that s.dt.date does not any longer return datetime.date objects, but rather a Timestamp (at midnight) with a tz?
In any case it is not a bug currently, as for now, date() is supposed to give you a date (which by definition no longer has a timezone) and not a datetime/timestamp.

Of course such a change could be discussed, but I agree with @rockg that I also don't see the purpose of this change.

If we do not change it, this issue indeed only needs some tests to confirm the fixed apply behaviour to get closed.

[jreback]

no this is fine

though maybe should doc it somehow/where
naively I would have though we would preserve the tz meta data but apparently .date doesn't (which IMHO is odd)

[rockg]

I'll add tests/doc mention tonight and then we should be good.