concat(..., copy=False) with datetime tz-aware data raises ValueError: cannot create a DatetimeTZBlock without a tz

[karldw]

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.DataFrame({
    'timestamp': [pd.Timestamp('2017-08-27 01:00:00.709949+0000', tz='UTC')],
})

pd.concat([df])      # works
pd.concat([df, df])  # works
pd.concat([df], copy=False)  # fails
# Edit: added full traceback
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File ".../pandas/core/reshape/concat.py", line 229, in concat
#     return op.get_result()
#   File ".../pandas/core/reshape/concat.py", line 426, in get_result
#     copy=self.copy)
#   File ".../pandas/core/internals/managers.py", line 2055, in concatenate_block_managers
#     b = b.make_block_same_class(values, placement=placement)
#   File ".../pandas/core/internals/blocks.py", line 235, in make_block_same_class
#     klass=self.__class__, dtype=dtype)
#   File ".../pandas/core/internals/blocks.py", line 3095, in make_block
#     return klass(values, ndim=ndim, placement=placement)
#   File ".../pandas/core/internals/blocks.py", line 1680, in __init__
#     values = self._maybe_coerce_values(values)
#   File ".../pandas/core/internals/blocks.py", line 2266, in _maybe_coerce_values
#     raise ValueError("cannot create a DatetimeTZBlock without a tz")
# ValueError: cannot create a DatetimeTZBlock without a tz

Problem description

Running pd.concat with a single dataframe and copy=False fails, giving an error about cannot create a DatetimeTZBlock without a tz.

Closely related:

• DF.stack() on single column datetime with timezone causes loss of timezone #19420
• https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3AReshaping+label%3ATimezones

Expected Output

The original dataframe, since I'm only concat-ing a single dataframe.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-14-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.1
pytest: 4.2.0
pip: 19.0.1
setuptools: 40.6.3
Cython: 0.29.4
numpy: 1.15.4
scipy: 1.2.0
pyarrow: 0.11.1
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: 2.4.10
xlrd: 1.2.0
xlwt: None
xlsxwriter: 1.1.2
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: 1.2.17
pymysql: None
psycopg2: 2.7.6.1 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: 0.2.1
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[TomAugspurger]

Can you post the full traceback?

Any interest in working on this?

[karldw]

I edited to add the full traceback.
I don't think I have the time to solve this.

[mroeschke]

The core issue is here:

pandas/pandas/core/internals/managers.py

Line 2062 in 3ab9318

values = values.view()

Taking a view of a tz aware DatetimeArray removes tz information and is passing the result to a DatetimeTZBlock.

2 options

1. Remove the view or don't view for DatetimeArray?
2. Pass in dtype (which contains the tz) to make_block_same_class, but it looks like we are deprecating this argument

pandas/pandas/core/internals/blocks.py

Lines 229 to 232 in 3ab9318

	if dtype is not None:
	# issue 19431 fastparquet is passing this
	warnings.warn("dtype argument is deprecated, will be removed "
	"in a future release.", DeprecationWarning)

cc @jreback

[jreback]

i think .view() for DTA with tz is broken
we don’t use .view much at all generally as this is mainly for numpy compat

[TomAugspurger]

Pushing to 0.25.0. I'm hoping to clean up our concat internals before then. But if someone wants to put up a dedicated fix for this before then, that'd be great too.

[HowManyOliversAreThere]

I'm unsure whether this was actively resolved or just happens to have been resolved, but I was hitting this bug on one machine with older packages and not hitting the issue (functioning as expected) on another machine with newer packages.

• Failing machine:
  • Python 3.8.0
  • Pandas 0.25.3
  • Numpy 1.17.4
• Succeeding machine:
  • Python 3.7.5
  • Pandas 1.0.2
  • Numpy 1.18.1

[mroeschke]

Was probably fixed in a recent release. Could use a regression test.

[SurajH1]

take

[luckyvs1]

@SurajH1 are you still working on this issue?