prototype datasets tests are failing

[pmeier]

CI failures are unrelated:

FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[imagenet-val-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[cub200-train-2011-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[cub200-test-2011-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[imagenet-val-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[imagenet-val-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-train-2011-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-train-2011-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-test-2011-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-test-2011-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-cub200-test-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-cub200-test-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[cub200-test-2011]

Given that the failures are only happening for imagenet and cub200, my gut says this has something to do with #6076. cc @YosuaMichael @bjuncek @NicolasHug @ejguan @NivekT

Originally posted by @pmeier in #6514 (comment)

[pmeier]

@NivekT identified pytorch/pytorch#84079 as offender. It was already reverted so I'm guessing we only need to wait for the fresh nightly.

[pmeier]

As expected, the problem is resolved with today's nightly: https://github.com/pytorch/vision/runs/8091700701

[NivekT]

Thanks @pmeier for looking into and tracking this!