rustbuild: Implement `make dist` #32237

alexcrichton · 2016-03-13T23:56:44Z

This commit implements the make dist command in the new rustbuild build
system, porting over dist.mk and prepare.mk into Rust. There's a huge amount
of complexity between those two files, not all of which is likely justified, so
the Rust implementation is much smaller.

Currently the implementation still shells out to rust-installer as well as some
python scripts, but ideally we'd rewrite it all in the future to not shell out
and be in Rust proper.

rust-highfive · 2016-03-13T23:56:48Z

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2016-03-13T23:56:50Z

r? @brson

alexcrichton · 2016-03-13T23:57:53Z

src/bootstrap/build/dist.rs

+    // installed
+    t!(fs::create_dir_all(&overlay));
+    let cp = |file: &str| {
+        t!(fs::copy(build.src.join(file), overlay.join(file)));


So the install script uses the install command all over the place instead of just normal file copies. I'm... not really sure why, but maybe you know @brson? Was it to just explicitly control the permissions? I wouldn't mind adding a little wrapper to this module to do that, but I'm mainly just curious if there are reasons beyond permissions that install is used.

@alexcrichton Just permissions.

And I think this code should change to set the permissions. The installer will actually set them all itself itself during install, but that's a hack because we've historically produced seemingly-broken tarballs that don't retain their permissions. The installer should not be doing that and the permissions should be configured correctly before packaging. Here's the permissions rust-installer expects.

alexcrichton · 2016-03-14T00:01:30Z

For now this is also rebased on #32206, but the only relevant commit to this PR is the last one.

This commit fixes our support for cross compiling a compiler to run on FreeBSD. Over the weekend I managed to get a cross compiler from Linux to FreeBSD [1] which I hope to soon use to start producing FreeBSD nightly compilers. With the `make dist` support added in rust-lang#32237 we should be able to produce standard rustc/rust-std packages for FreeBSD through a new slave with this cross compiler. Currently, however, we don't "Just Work" when cross compiling FreeBSD and a number of changes were required (part of this PR). They include: * A few build fixes were needed in LLVM. Our own branch has been rebased on the actual 3.8 release and I applied one extra commit [2] which contains two fixes: 1. The LLVM CMake build system passes the `-Wl,-z,defs` flag on many platforms, but *not* when `CMAKE_SYSTEM_NAME` is "FreeBSD". Unfortunately this doesn't take into account when we're cross compiling, and as predicted the build will fail if `-Wl,-z,defs` is passed (see [3] for more info). To fix this we test `TARGET_TRIPLE` instead of the `CMAKE_SYSTEM_NAME` which is what we're compiling for which fixes the problem. 2. The `PATH_MAX` constant is apparently defined in a different location than many other Unix systems, so a file which required this just needed some help to keep compiling. * Support for compiling compiler-rt with CMake has been added to rustbuild. It looks like it just emulates Linux in what it compiles as it didn't seem to naturally produce anything else... At least the architecture is right, so seems good for now at least! [1]: https://github.com/alexcrichton/port-of-rust/blob/master/prebuilt/freebsd/Dockerfile [2]: rust-lang/llvm@be89e4b5 [3]: https://bugs.webkit.org/show_bug.cgi?id=138420

bors · 2016-03-15T07:13:54Z

☔ The latest upstream changes (presumably #32206) made this pull request unmergeable. Please resolve the merge conflicts.

rustbuild: Fix cross compiling to FreeBSD This commit fixes our support for cross compiling a compiler to run on FreeBSD. Over the weekend I managed to get a cross compiler from Linux to FreeBSD [1] which I hope to soon use to start producing FreeBSD nightly compilers. With the `make dist` support added in #32237 we should be able to produce standard rustc/rust-std packages for FreeBSD through a new slave with this cross compiler. Currently, however, we don't "Just Work" when cross compiling FreeBSD and a number of changes were required (part of this PR). They include: * A few build fixes were needed in LLVM. Our own branch has been rebased on the actual 3.8 release and I applied one extra commit [2] which contains two fixes: 1. The LLVM CMake build system passes the `-Wl,-z,defs` flag on many platforms, but *not* when `CMAKE_SYSTEM_NAME` is "FreeBSD". Unfortunately this doesn't take into account when we're cross compiling, and as predicted the build will fail if `-Wl,-z,defs` is passed (see [3] for more info). To fix this we test `TARGET_TRIPLE` instead of the `CMAKE_SYSTEM_NAME` which is what we're compiling for which fixes the problem. 2. The `PATH_MAX` constant is apparently defined in a different location than many other Unix systems, so a file which required this just needed some help to keep compiling. * Support for compiling compiler-rt with CMake has been added to rustbuild. It looks like it just emulates Linux in what it compiles as it didn't seem to naturally produce anything else... At least the architecture is right, so seems good for now at least! [1]: https://github.com/alexcrichton/port-of-rust/blob/master/prebuilt/freebsd/Dockerfile [2]: rust-lang/llvm@be89e4b5 [3]: https://bugs.webkit.org/show_bug.cgi?id=138420

japaric · 2016-03-16T00:32:01Z

I tried configure --enable-rustbuild --host=arm-unknown-linux-gnueabihf --enable-dist-host-only && make && make dist with this PR and rustbuild generated these tarballs:

rust-docs-1.9.0-dev.tar.gz
rust-std-1.9.0-dev-arm-unknown-linux-gnueabihf.tar.gz
rust-std-1.9.0-dev-x86_64-unknown-linux-gnu.tar.gz
rustc-1.9.0-dev-x86_64-unknown-linux-gnu.tar.gz

Note that rustc-arm is missing. @alexcrichton suggested running python bootstrap.py --stage 2 --step dist --host arm-unknown-linux-gnueabihf to generate the missing tarball and I got this error:

$ python ../rust/src/bootstrap/bootstrap.py --stage 2 --step dist --host arm-unknown-linux-gnueabihf
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> arm-unknown-linux-gnueabihf)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> arm-unknown-linux-gnueabihf)
Building stage2 std artifacts (arm-unknown-linux-gnueabihf -> arm-unknown-linux-gnueabihf)
failed to run `rustc` to learn about target-specific information

To learn more, run the command again with --verbose.


command did not execute successfully: "/buildslave/build/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "-j" "8" "--target" "arm-unknown-linux-gnueabihf" "--release" "--features" " jemalloc" "--manifest-path" "/buildslave/rust/src/rustc/std_shim/Cargo.toml"
expected success, got: exit code: 101

Which seems to indicate that rustbuild is trying to run the rustc ARM binary.

Then @alexcrichton suggested this patch. I applied and after calling make && make dist I got:

(...)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> arm-unknown-linux-gnueabihf)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> arm-unknown-linux-gnueabihf)
Building stage2 std artifacts (arm-unknown-linux-gnueabihf -> arm-unknown-linux-gnueabihf)
failed to run `rustc` to learn about target-specific information

To learn more, run the command again with --verbose.


command did not execute successfully: "/buildslave/build/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "-j" "8" "--target" "arm-unknown-linux-gnueabihf" "--release" "--features" " jemalloc" "--manifest-path" "/buildslave/rust/src/rustc/std_shim/Cargo.toml"
expected success, got: exit code: 101

I hope this info helps to debug the problem!

alexcrichton · 2016-03-16T17:52:25Z

Ok, I've pushed a commit which I believe fixes that problem. I... am not entirely sure how it fixes it but it seems to work locally.

alexcrichton · 2016-03-16T18:32:19Z

Confirmed to work:

$ ./configure --host=i686-unknown-linux-gnu,arm-unknown-linux-gnueabihf --enable-rustbuild
$ make dist
[..]
$ ls build/dist
rustc-1.9.0-dev-arm-unknown-linux-gnueabihf.tar.gz      
rustc-1.9.0-dev-i686-unknown-linux-gnu.tar.gz           
rustc-1.9.0-dev-x86_64-unknown-linux-gnu.tar.gz         
rust-docs-1.9.0-dev.tar.gz                              
rust-std-1.9.0-dev-arm-unknown-linux-gnueabihf.tar.gz   
rust-std-1.9.0-dev-i686-unknown-linux-gnu.tar.gz        
rust-std-1.9.0-dev-x86_64-unknown-linux-gnu.tar.gz

r? @brson

brson · 2016-03-16T20:50:37Z

rust-buildbot expects that as part of dist the docs will be copied to dist/doc/$CFG_PACKAGE_VERS. This is where the docs are uploaded from, and also critically how rust-buildbot figures out the name of the package being built!

brson · 2016-03-16T21:02:38Z

src/bootstrap/build/dist.rs

+    cp("LICENSE-MIT");
+    cp("README.md");
+    // tiny morsel of metadata is used by rust-packaging
+    let version = &build.version;


I think this value is incorrect. It should be something like rustc 1.9.0-nightly (c66d2380a 2016-03-15), exactly what rustc prints for --version.

Oh looks like I actually managed to accomodate for this!

This commit implements the `make dist` command in the new rustbuild build system, porting over `dist.mk` and `prepare.mk` into Rust. There's a huge amount of complexity between those two files, not all of which is likely justified, so the Rust implementation is *much* smaller. Currently the implementation still shells out to rust-installer as well as some python scripts, but ideally we'd rewrite it all in the future to not shell out and be in Rust proper.

alexcrichton · 2016-03-17T00:16:18Z

re-r? @brson

Added an install function which sets permissions and also added the doc directory to make dist.

brson · 2016-03-17T00:21:00Z

@bors r+

bors · 2016-03-17T00:21:01Z

📌 Commit 6cc06b3 has been approved by brson

bors · 2016-03-17T01:28:33Z

⌛ Testing commit 6cc06b3 with merge 0d746bc...

rustbuild: Implement `make dist` This commit implements the `make dist` command in the new rustbuild build system, porting over `dist.mk` and `prepare.mk` into Rust. There's a huge amount of complexity between those two files, not all of which is likely justified, so the Rust implementation is *much* smaller. Currently the implementation still shells out to rust-installer as well as some python scripts, but ideally we'd rewrite it all in the future to not shell out and be in Rust proper.

bors · 2016-03-17T02:44:29Z

💔 Test failed - auto-win-gnu-64-nopt-t

alexcrichton · 2016-03-17T05:16:28Z

@bors: retry

On Wed, Mar 16, 2016 at 7:44 PM, bors [email protected] wrote:

[image: 💔] Test failed - auto-win-gnu-64-nopt-t
http://buildbot.rust-lang.org/builders/auto-win-gnu-64-nopt-t/builds/3459

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#32237 (comment)

bors · 2016-03-17T09:35:33Z

⌛ Testing commit 6cc06b3 with merge abb1515...

rustbuild: Implement `make dist` This commit implements the `make dist` command in the new rustbuild build system, porting over `dist.mk` and `prepare.mk` into Rust. There's a huge amount of complexity between those two files, not all of which is likely justified, so the Rust implementation is *much* smaller. Currently the implementation still shells out to rust-installer as well as some python scripts, but ideally we'd rewrite it all in the future to not shell out and be in Rust proper.

bors · 2016-03-17T14:52:28Z

rust-highfive assigned aturon Mar 13, 2016

rust-highfive assigned brson and unassigned aturon Mar 13, 2016

alexcrichton reviewed Mar 13, 2016
View reviewed changes

alexcrichton force-pushed the rustbuild-make-dist branch from f5fb248 to 689e6bb Compare March 14, 2016 00:00

alexcrichton force-pushed the rustbuild-make-dist branch 5 times, most recently from 2a01d58 to c00fef1 Compare March 14, 2016 05:10

alexcrichton mentioned this pull request Mar 14, 2016

rustbuild: Fix cross compiling to FreeBSD #32239

Merged

alexcrichton force-pushed the rustbuild-make-dist branch from c00fef1 to 40449c4 Compare March 15, 2016 14:58

alexcrichton force-pushed the rustbuild-make-dist branch from 40449c4 to 942d609 Compare March 16, 2016 17:51

alexcrichton mentioned this pull request Mar 16, 2016

Tracking issue for rustbuild, the alternate build system for rustc #31590

Closed

48 tasks

brson reviewed Mar 16, 2016
View reviewed changes

alexcrichton force-pushed the rustbuild-make-dist branch from 942d609 to cff833f Compare March 17, 2016 00:08

alexcrichton force-pushed the rustbuild-make-dist branch from cff833f to 6cc06b3 Compare March 17, 2016 00:15

japaric mentioned this pull request Mar 17, 2016

new binary releases for mips and arm rust-lang-deprecated/rust-buildbot#78

Closed

3 tasks

bors mentioned this pull request Mar 17, 2016

move const_eval and check_match out of librustc into their own crate #32259

Merged

bors merged commit 6cc06b3 into rust-lang:master Mar 17, 2016

alexcrichton deleted the rustbuild-make-dist branch March 27, 2016 17:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustbuild: Implement `make dist` #32237

rustbuild: Implement `make dist` #32237

alexcrichton commented Mar 13, 2016

rust-highfive commented Mar 13, 2016

alexcrichton commented Mar 13, 2016

alexcrichton Mar 13, 2016

brson Mar 16, 2016

brson Mar 16, 2016

alexcrichton commented Mar 14, 2016

bors commented Mar 15, 2016

japaric commented Mar 16, 2016

alexcrichton commented Mar 16, 2016

alexcrichton commented Mar 16, 2016

brson commented Mar 16, 2016

brson Mar 16, 2016

alexcrichton Mar 17, 2016

alexcrichton commented Mar 17, 2016

brson commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

alexcrichton commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

rustbuild: Implement make dist #32237

rustbuild: Implement make dist #32237

Conversation

alexcrichton commented Mar 13, 2016

rust-highfive commented Mar 13, 2016

alexcrichton commented Mar 13, 2016

alexcrichton Mar 13, 2016

Choose a reason for hiding this comment

brson Mar 16, 2016

Choose a reason for hiding this comment

brson Mar 16, 2016

Choose a reason for hiding this comment

alexcrichton commented Mar 14, 2016

bors commented Mar 15, 2016

japaric commented Mar 16, 2016

alexcrichton commented Mar 16, 2016

alexcrichton commented Mar 16, 2016

brson commented Mar 16, 2016

brson Mar 16, 2016

Choose a reason for hiding this comment

alexcrichton Mar 17, 2016

Choose a reason for hiding this comment

alexcrichton commented Mar 17, 2016

brson commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

alexcrichton commented Mar 17, 2016

bors commented Mar 17, 2016

bors commented Mar 17, 2016

rustbuild: Implement `make dist` #32237

rustbuild: Implement `make dist` #32237