Implement ArchiveVersionDownloads background job

[Turbo87]

This PR implements a solution for #3479. The new background job can be used to export all version download data to S3 and then remove it from the database. This should allow us to shrink the database considerably, which hopefully might have a positive effect on our database performance.

[codecov]

Codecov Report

Attention: Patch coverage is 84.11215% with 51 lines in your changes are missing coverage. Please review.

Project coverage is 88.43%. Comparing base (180c7b2) to head (3e730b9).

Files	Patch %	Lines
src/worker/jobs/archive_version_downloads.rs	85.89%	44 Missing ⚠️
src/admin/enqueue_job.rs	0.00%	5 Missing ⚠️
src/bin/background-worker.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8596      +/-   ##
==========================================
- Coverage   88.50%   88.43%   -0.07%     
==========================================
  Files         274      275       +1     
  Lines       26938    27257     +319     
==========================================
+ Hits        23841    24105     +264     
- Misses       3097     3152      +55     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eth3lbert]

Do you think generating a series of dates from date_start to current_date - 90 and then looping through them to export as CSV could be a possible solution? This approach came to mind as it's relatively simple. It would only require keeping track of date_start and wouldn't necessitate splitting the CSV file.

[Turbo87]

Do you think generating a series of dates from date_start to current_date - 90 and then looping through them to export as CSV could be a possible solution?

it is a possible solution, but a) figuring out that date_start value and b) exporting per date are quite slow queries and it seemed faster just export a single file and perform the post-processing on the Rust side instead. admittedly the post-processing step isn't particularly fast either, but from the limited amount of testing I've performed it seemed like the one export query + post-process approach would be faster than the date_start query + N export queries approach.

[eth3lbert]

Thanks for breaking it down!

it is a possible solution, but a) figuring out that date_start value

The simplest solution would be to simply start from the date that crates.io service was activated if there is no date_start recorded. The date_start should then be updated and recorded every time the job is processed.

b) exporting per date are quite slow queries and it seemed faster just export a single file and perform the post-processing on the Rust side instead. admittedly the post-processing step isn't particularly fast either, but from the limited amount of testing I've performed it seemed like the one export query + post-process approach would be faster than the date_start query + N export queries approach.

I can't definitively conclude the cause of the slowness without knowing the query's execution plan. However, I suspect it's because the query isn't utilizing the index.

I'm unsure whether using the following query structure would be beneficial:

where version_id in (select id from versions) and date = 'some_date'

If performance remains poor when filtering with a single date (although this should improve once unneeded records are archived), then the proposed splitting CSV file approach makes perfect sense to me.

[LawnGnome]

OK, so as implemented, this LGTM.

Did you test shelling out to psql against running the query and writing the CSV files directly from within the background worker? To me that feels like the simpler approach — and would give us some potential benefits around being able to order and/or chunk the dates that we're extracting, which would limit how many open files we'd need — which makes me suspect that there's a reason for \copy that I'm not seeing.

Approving to unblock this, but I'm curious about the above. (And have probably just missed some context while I was in the land of the 🦘!)

[LawnGnome]

Looks like standard Heroku dynos have a 10k open file descriptor limit.

This shouldn't be an issue for many years, but having one file descriptor open per historical day plus whatever other work is happening in the background worker is a potential footgun.

[Turbo87]

it would be an issue if the historical days would stay in the database, but once we have exported the historical data to S3 we will no longer have this issue, because at most we would be exporting a couple of days and most likely only a single day.

[Turbo87]

Did you test shelling out to psql against running the query and writing the CSV files directly from within the background worker?

yeah, I tried a native COPY command via diesel::sql_query(), but that complained about not being able to access the file and the copy functionality in diesel itself is not yet released AFAIK. since we were already using something roughly similar for the database dump I figured this should work well enough.

one potential alternative would be COPY ... TO stdout and then handling the stdout stream for the child process, but I'm not convinced that that would be less complex than the current solution 😅