Skip to content

Add incomplete download support #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

ardaatahan
Copy link
Contributor

This PR:

  • Expands on Make downloads resumable across app sessions #187 which downloads incomplete files directly into the target directory, which could lead to runtime errors while trying to read those files in the case a download did not fully complete. This is resolved by downloading incomplete files first to the cache directory then moving them to their final locations. This increases swift-transformers' download logic parity with the huggingface_hub library as laid out here
  • Also adds download cancellation support. Previously, if you were downloading a file inside a task and cancelled the task, the download would continue to run even if the task was canceled. This is done via adding withTaskCancellationHandler inside HubFileDownloader's Download method, which would immediately invalidate the url session and updates progress as cancelled.
  • Renames urlSession to session in Downloader class to resolve naming conflict with URLSessionDownloadDelegate methods with the same name.
  • Creates target directory structure lazily (as opposed to eagerly before) when moving the incomplete files to their final locations.
  • Adds/improves test cases to verify new functionality.

@ardaatahan
Copy link
Contributor Author

@pcuenca, @FL33TW00D FYI - I changed the session member variable (renamed from urlSession due to naming conflict) in Downloader to internal from private. This is because the DownloaderTests file extends Downloader with a test only interruptDownload method that directly accesses the session to simulate network interruptions at precise progress checkpoints (50% and 75%). This allows verification of the resumable download functionality I added with #155. Since this access remains module internal and doesn't expand the public API surface, it still maintains proper encapsulation.

@FL33TW00D
Copy link
Collaborator

Looks really good, nice work @ardaatahan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants