rustc randomly fail to link with C program

[xclaesse]

Here is a minimal test case to reproduce: https://github.com/xclaesse/rustc-bug.

I believe this is the same issue than #55931 initially discovered this issue while working on this Meson PR: mesonbuild/meson#11714.

I suspect the problem is when compiling both cdylib and staticlib for the same stuff.rs with the same crate name, they use some common intermediary files.

[bjorn3]

Are you using two separate rustc invocations or a single with two --crate-type arguments?

[xclaesse]

2 separate innovations running in parallel, see the makefile in my test case above.

[xclaesse]

If I'm right and rustc write temporary files (probably .o files) that could clash between both invocations, is there a way to tell rustc where to write its temp files? That way I can easily give a different dir to each invocations.

[bjorn3]

You could add a different -Cmetadata argument between the two invocations. This should affect the name of the temporary files. However if possible I did recommend merging both rustc invocations. This cuts the required amount of cpu time almost in half as pretty much the only difference between the cdylib and staticlib crate types is the step in which the dynamic library is linked or the archive assembled. The rest of the compilation process is reused when you specify both crate types.

[xclaesse]

Ideally we should have a single invocation indeed, but that would require a bit more refactoring in Meson to work like that. I'll try -Cmetadata=value argument to see if that can work around the problem for now, thanks.

[xclaesse]

@bjorn3 I can confirm that passing -C metadata=foo to one command and -C metadata=bar to the other seems to fix the issue. I can also see that if I add --emit metadata both writes the same liba.rmeta file. However, I was expecting to get foo.rmeta and bar.rmeta, so I probably did not understand what -C metadata=foo does, is foo a temporary directory where it writes some files? Is there a way to tell rustc to not delete intermediary files?

[bjorn3]

-Cmetadata doesn't affect the output name and location of the final artifact. It affects the "stable crate id" which is a hash that determines the identity of a crate. One of the things that determines the names of temporary artifacts is the "stable crate id". You can tell rustc to not delete temporary files using -Csave-temps. Most of them are stored in the directory passed to --out-dir (which defaults to the current working directory and is the location where the final cdylib/staticlib will be stored unless you explicitly give a different path), but some temporary files are stored in a random directory in /tmp.

[xclaesse]

Is there a downside in changing that "stable crate id" by adding the name of the final artifact to avoid conflicts? Like -Cmetadata=liba.a and -Cmetadata=liba.so?

Using different --out-dir also seems to fix the issue, but is there a way to only put temporary files there and not the final artifacts?

One thing I noticed with -Csave-temps is the conflicting liba.rmeta are automatically put in randimized temporary directory: rmetaN0c8S7/lib.rmeta and rmetapQ1cYv/lib.rmeta which makes sense, all other temporary files should be automatically moved to such directory IMHO. Or let CLI define that directory, Meson always create a per target private directory for similar cases when building C libraries.

[bjorn3]

Is there a downside in changing that "stable crate id" by adding the name of the final artifact to avoid conflicts? Like -Cmetadata=liba.a and -Cmetadata=liba.so?

There shouldn't be. The "stable crate id" is only relevant when linking against a rust crate. The cdylib and staticlib crate types don't allow linking against it as rust crate anyway.

Using different --out-dir also seems to fix the issue, but is there a way to only put temporary files there and not the final artifacts?

You can combine --out-dir with --emit link=path/to/libfoo.a.

One thing I noticed with -Csave-temps is the conflicting liba.rmeta are automatically put in randimized temporary directory: rmetaN0c8S7/lib.rmeta and rmetapQ1cYv/lib.rmeta which makes sense, all other temporary files should be automatically moved to such directory IMHO.

Makes sense.

[xclaesse]

You can combine --out-dir with --emit link=path/to/libfoo.a.

Awesome, that's a workable solution for me I think. Thanks.

I still think this should be fixed at rustc level, even if it's not the indented usage, by ensuring temp files are put in randomized temp directories.