rustup: add support for caching of old nightlies, download resumption and verification

[erickt]

This allows rustup to be used for bisecting bugs across the nightlies, which can really help to speed up tracking down bugs. In addition, this PR changes rustup to use curl's download resumption, and also adds hash verification of the downloads to make sure they are not corrupted. We still need to add GPG signing in order to protect against someone tampering with the downloads though.

[sfackler]

Tidy issue

[alexcrichton]

I still get an error when running this :(

$ curl -O https://raw.githubusercontent.com/erickt/rust/rustup/src/etc/rustup.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13759  100 13759    0     0  60259      0 --:--:-- --:--:-- --:--:-- 60346
$ sh rustup.sh
rustup: CFG_CURL             := /usr/bin/curl (7.35.0)
rustup: CFG_TAR              := /bin/tar (1.27.1)
rustup: CFG_SHASUM           := /usr/bin/shasum (5.84)
rustup: 
rustup: processing rustup.sh args
rustup: 
rustup: CFG_PREFIX           :=  
rustup: CFG_DATE             :=  
rustup: CFG_RUST_DATE        :=  
rustup: CFG_CARGO_DATE       :=  
rustup: 
rustup: validating rustup.sh args
rustup: 
rustup: host triple: x86_64-unknown-linux-gnu
rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  111M  100  111M    0     0  25.4M      0  0:00:04  0:00:04 --:--:-- 26.8M
rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.sha256
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   139  100   139    0     0   3987      0 --:--:-- --:--:-- --:--:--  4088
rustup: Verifying hash
rustup: Downloading https://static.rust-lang.org/cargo-dist/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz to ./rustup-tmp-install/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4127k  100 4127k    0     0  24.8M      0 --:--:-- --:--:-- --:--:-- 25.0M
rustup: Downloading https://static.rust-lang.org/cargo-dist/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz.sha256
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   141  100   141    0     0   3943      0 --:--:-- --:--:-- --:--:--  4028
rustup: Verifying hash
rustup: Extracting ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
/bin/tar: ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now
rustup: error: failed to unpack installer

[erickt]

@alexcrichton: want to try again?

[alexcrichton]

No dice :(

$ curl -O https://raw.githubusercontent.com/erickt/rust/rustup/src/etc/rustup.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13806  100 13806    0     0   464k      0 --:--:-- --:--:-- --:--:--  481k
$ sh rustup.sh
rustup: CFG_CURL             := /usr/bin/curl (7.35.0)
rustup: CFG_TAR              := /bin/tar (1.27.1)
rustup: CFG_FILE             := /usr/bin/file (5.14)
rustup: CFG_SHASUM           := /usr/bin/shasum (5.84)
rustup: 
rustup: processing rustup.sh args
rustup: 
rustup: CFG_PREFIX           :=  
rustup: CFG_DATE             :=  
rustup: CFG_RUST_DATE        :=  
rustup: CFG_CARGO_DATE       :=  
rustup: 
rustup: validating rustup.sh args
rustup: 
rustup: host triple: x86_64-unknown-linux-gnu
mktemp: too few X's in template ‘rustup-tmp-install’
mkdir: cannot create directory ‘’: No such file or directory
rustup: error: failed to create create temporary installation directory

[erickt]

@alexcrichton: want to try again? I think #19227 might have fixed the mktemp issue you hit.

[erickt]

Shucks, looks like linux has a sha256sum, not a shasum like on osx. I'll update it to probe for both binaries.

[erickt]

The PR has been patched to support sha256sum on linux, and shasum on BSD and OS X.

[alexcrichton]

This works for me and looks ok to me, r? @brson

[flaper87]

Tried and works for me as well. LGTM

[brson]

Superseded by #20452

[alexcrichton]

Closing in favor of #20452