error_index_generator cannot find -lLLVM-3.9

[cuviper]

I'm trying to build with an external LLVM that lives in a non-standard path, but I'm getting this error:

Building stage0 tool error_index_generator (x86_64-unknown-linux-gnu)
   Compiling error_index_generator v0.0.0 (file:///builddir/rust/src/tools/error_index_generator)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" [... posted in full below]  "-l" "LLVM-3.9" [...]
  = note: /usr/bin/ld: cannot find -lLLVM-3.9
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `error_index_generator`.

I'm using --llvm-root=/usr/lib64/llvm3.9, and its llvm-config reports the correct search path:

# /usr/lib64/llvm3.9/bin/llvm-config --ldflags --libs
-L/usr/lib64/llvm3.9/lib
-lLLVM-3.9

I don't know why error_index_generator is linking LLVM, but it seems to have picked up the library requirement without the library search path. I guess one of its crate dependencies is indirectly leading to rustc_llvm. However, I can see that librustc_llvm-*.so is correctly linked to libLLVM-3.9.so, so it must have used the right library path at that point.

I do see that the build script rustc_llvm-*/output has both of these expected lines:

• cargo:rustc-link-lib=dylib=LLVM-3.9
• cargo:rustc-link-search=native=/usr/lib64/llvm3.9/lib

Normally I use a more typical --llvm-root=/usr and it works fine, but of course the search path isn't an issue if the library is in the standard path already.

Here's the verbose build output for error_index_generator:

Building stage0 tool error_index_generator (x86_64-unknown-linux-gnu)
running: "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "-j" "8" "--target" "x86_64-unknown-linux-gnu" "-v" "--release" "--manifest-path" "/builddir/rust/src/tools/error_index_generator/Cargo.toml"
   Compiling error_index_generator v0.0.0 (file:///builddir/rust/src/tools/error_index_generator)
     Running `/builddir/rust/build/bootstrap/debug/rustc --crate-name error_index_generator src/tools/error_index_generator/main.rs --crate-type bin --emit=dep-info,link -C opt-level=2 -C metadata=e97dafd8c29cb5c4 -C extra-filename=-e97dafd8c29cb5c4 --out-dir /builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/release/deps`
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/error_index_generator-e97dafd8c29cb5c4.0.o" "-o" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/error_index_generator-e97dafd8c29cb5c4" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/release/deps" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustdoc-8cfbe926b71e58fb" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_driver-3448470125485f1b" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_typeck-a32db09fad57f156" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_save_analysis-b46d330be0de3abb" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_privacy-08ea86aa4c29a66f" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_plugin-d3708a435dcec728" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_metadata-072faec0de75595d" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_passes-639d94ec9d02120c" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_trans-d31713e7b6eb3ea9" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_platform_intrinsics-b197b69430172bb1" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "flate-fd68d342189f2d80" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_incremental-9233eef3716e80cd" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_borrowck-c0291043cb8fa3bf" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_mir-9035a2564e719c3c" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_lint-74087fae485d91e9" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_const_eval-2ed75a90a13c1055" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "syntax_ext-7c7f1d757a07082d" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "proc_macro-5fa53ecb0c049609" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_resolve-d6a1f980925721c1" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc-2d36eb2c186c1638" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_const_math-94eba3a8c2e0e2ae" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_back-5d73d9391cf0e484" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_llvm-d6deafbca12aedf2" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "graphviz-b65d21519489f87a" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "fmt_macros-136fc4505ca2d430" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "arena-5ae1748aa150585f" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "syntax-5d1005fe2d2fbc36" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_data_structures-0bc584c878f3912a" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "rustc_errors-4cfcb74bccbb01e2" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "syntax_pos-c920172bef839037" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "serialize-df95e1966fd3f831" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "log-184a405a51092326" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "test-53682a11e61be1e7" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "term-d2650bbd90536ace" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "getopts-0640b5f955e4b186" "-L" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "std-4d81be0f21c9fba1" "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-1fc384286702beba.rlib" "-l" "LLVM-3.9" "-l" "rt" "-l" "dl" "-l" "tinfo" "-l" "pthread" "-l" "z" "-l" "m" "-l" "stdc++" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-Wl,-rpath,$ORIGIN/../lib"
  = note: /usr/bin/ld: cannot find -lLLVM-3.9
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `error_index_generator`.

Caused by:
  process didn't exit successfully: `/builddir/rust/build/bootstrap/debug/rustc --crate-name error_index_generator src/tools/error_index_generator/main.rs --crate-type bin --emit=dep-info,link -C opt-level=2 -C metadata=e97dafd8c29cb5c4 -C extra-filename=-e97dafd8c29cb5c4 --out-dir /builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/rust/build/x86_64-unknown-linux-gnu/stage0-tools/release/deps` (exit code: 101)


command did not execute successfully: "/builddir/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "-j" "8" "--target" "x86_64-unknown-linux-gnu" "-v" "--release" "--manifest-path" "/builddir/rust/src/tools/error_index_generator/Cargo.toml"
expected success, got: exit code: 101

[cuviper]

FWIW, everything compiles fine if I kludge in RUSTFLAGS=-Clink-arg=-L/usr/lib64/llvm3.9/lib.

[alexcrichton]

Hm yes I think this is an unfortunate side effect of the organization of rustbuild. Cargo propagates these flags amongst crates but these are two separate Cargo projects, so the error index generator is not receiving the appropriate -L flags that rustc_llvm's build script is printing out.

I... unfortunately don't know the greatest way to solve this, other than just vendoring rustc_llvm's build script or something like that (which would be awful).

[cuviper]

How does it pass the -lLLVM3.9 through? Couldn't that also include the -L?

[alexcrichton]

The -l comes through rustc itself as it's encoded into the metadata that a library needs to be linked to, currently the metadata doesn't encode paths, however.

[cuviper]

Is that through #[link]? Could we also add the paths through #[link_args]?

[alexcrichton]

Nah I think it's all through the build script. I forget, but I don't think #[link_args] will work.

[cuviper]

I just discovered this affects all proc-macro use too, even outside the compiler build itself, as those plugins link indirectly to rustc_llvm and end up with the -lLLVM-3.9 too.

[cuviper]

So, I understand it's the build script which generates those lines:

cargo:rustc-link-lib=dylib=LLVM-3.9
cargo:rustc-link-search=native=/usr/lib64/llvm3.9/lib

But it seems the link-lib ends up encoded in the metadata and carried forward, but the link-search does not. That's what I meant could be encoded somehow too, maybe not literally #[link_args], but some similar effect in the metadata.

[cuviper]

I just discovered this affects all proc-macro use too, even outside the compiler build itself, as those plugins link indirectly to rustc_llvm and end up with the -lLLVM-3.9 too.

I take that back. I saw the error while testing num-derive, but that LLVM dependency came from using compiletest_rs (with unstable features), not proc-macro itself. Plain use of num-derive looks ok.

[devurandom]

I ran into the same issue building rustdoc_tool_binary of Rust 1.23.0 (LLVM 4.0.1 in /usr/lib/llvm/4): rust-1.23.0.build.log

The work-around from #40717 (comment) got me past this issue: devurandom/gentoo-overlay@11651be#diff-d8799a17f743865c5a79e6380cddca82R148

[cuviper]

FWIW, I currently deal with this by setting LIBRARY_PATH for gcc as the linker, instead of RUSTFLAGS.

[steveklabnik]

@cuviper are you still running into this today?

[cuviper]

I hadn't noticed because I've always done those builds with LIBRARY_PATH since, but it now seems OK without. I suspect this was fixed when codegen (née trans) started loading dynamically in #47671.