Whole JSON gets dumped from error in _pull_records()

[wjandrea]

This error message shows the entire JSON, then the erroring part, both of which can be very large.

pandas/pandas/io/json/_normalize.py

Lines 429 to 432 in 68c1af5

	raise TypeError(
	f"{js} has non list value {result} for path {spec}. "
	"Must be list or null."
	)

Example:

url = 'https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=IBM&interval=5min&apikey=demo'
r = requests.get(url)
data = r.json()

meta = pd.json_normalize(data, 'Meta Data')

Error (clipped for readability):

Traceback (most recent call last):

  Cell In[50], line 1
    meta = pd.json_normalize(data, 'Meta Data')

  File pandas/io/json/_normalize.py:519 in json_normalize
    _recursive_extract(data, record_path, {}, level=0)

  File pandas/io/json/_normalize.py:501 in _recursive_extract
    recs = _pull_records(obj, path[0])

  File pandas/io/json/_normalize.py:431 in _pull_records
    raise TypeError(

TypeError: {'Meta Data': {'1. Information': 'Intraday (5min) open, high, low, close prices and volume', '2. Symbol': 'IBM', '3. Last Refreshed': '2023-12-07 19:55:00', '4. Interval': '5min', '5. Output Size': 'Compact', '6. Time Zone': 'US/Eastern'}, 'Time Series (5min)': ...} has non list value {'1. Information': 'Intraday (5min) open, high, low, close prices and volume', '2. Symbol': 'IBM', '3. Last Refreshed': '2023-12-07 19:55:00', '4. Interval': '5min', '5. Output Size': 'Compact', '6. Time Zone': 'US/Eastern'} for path Meta Data. Must be list or null.

(Here I've clipped the 'Time Series (5min)' value, but it takes up an order of magnitude more space.)

I believe the error could be changed to simply:

raise TypeError(f"non list value for path {spec}. Must be list or null.")

Then if you need more information, you can inspect data[spec].

Although, that wouldn't be ideal for cases where you've inlined the data retrieval, like pd.json_normalize(requests.get(url).json(), 'Meta Data').

[mroeschke]

Thanks for the report. Maybe

raise TypeError(f"{spec} contains {type(result).__name__}. Must be list or null.")

to at least inform of the type present that isn't a list or null

[wjandrea]

Yes, great idea. I'd add back the word "path" since "spec" isn't mentioned in the docs.

And while we're here, spec should probably be printed as repr instead of str, so:

raise TypeError(f"path {spec!r} contains {type(result).__name__}. Must be list or null.")

which would be in this case:

TypeError: path 'Meta Data' contains dict. Must be list or null.

[wjandrea]

Or you might want to move the path to the end in case it too is bulky:

raise TypeError(f"path must contain list or null, but got {type(result).__name__} at {spec!r}")

[mroeschke]

I think your suggestion in #56405 (comment) looks good. Happy to have a PR!

[rmhowe425]

take