Handle Stale(ish) Index Pages

[dstufft]

Currently accessing pages like /simple/foo/ will cache the responses for up to 10 minutes, if someone releases a version in that 10 minutes this can lead to some confusing results. We can communicate to people that they are getting a cached result better (see #2901) but it would be nice if we can think of a way to remove this small window of time when an update won't be visible to these users.

In #2904 @rbtcollins came up with:

The second I think a last-invalidated RSS-or-similar feed from PyPI itself would be appropriate (e.g. lists the API pages invalidated in the last say 12 minutes and the timestamp of their invalidation) - we can compare that to the Date header on the pages as we access them providing cheap (1 small HTTP request-response per pip install/wheel invocation) invalidation across arbitrary pages. Basically the same strategy as has been used to make Squid ultra responsive for accelerated sites for ages.

[dstufft]

Closing this, the cache is useful particularly for tox runs and people with high latency internet. Any possible solution here is going to be more trouble than it is worth.