Description
The flusurv data acquisition steps have begun failing with the following stack trace:
current issue: 202040
fetching data for CA (2, 1)
[fetching flusurv data...]
[extracting values...]
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 189, in <module>
main()
File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 185, in main
update(issue, args.location, args.test)
File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 99, in update
data = flusurv.get_data(location_code)
File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv.py", line 194, in get_data
data_out = extract_from_object(data_in)
File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv.py", line 162, in extract_from_object
prev_rate = data_out[epiweek][age]
IndexError: list index out of range
Tip: reproduce locally by following the dev guide (i.e. bringing up database and web servers) and then running:
docker run --rm --network delphi-net delphi_python python3 -m delphi.epidata.acquisition.flusurv.flusurv_update CA
The cause appears to be that the data now, as of 2020w40 (start of the 2020 flu season), includes 3 new age groups. This is great news in general. However, the definition of the new groups is unknown as CDC's webapp doesn't appear to have been updated yet.
Once the site above is updated with definitions of the new age groups, we should capture them. This will require fairly straightforward updates to the acquisition script, the database table, the API server, and the API docs.
In the meantime, as a stop-gap, I'll submit a fix which simply ignores the new, unknown, age groups.
TODO: revisit a week from today -- 2020-09-16