Apply Profile-guided optimization to improve performance

[FilipAndersson245]

Profile-guided optimization (PGO) shows great promise in improving the speed of software, last year tests where made on applying it on Rust itself improving build time by ~15%.
Would it be feasible to do something similar for Rust analyzer to improve its speed?
As i understand it the difficulties would be

• gathering runtime statistics
• longer compilation times

[matklad]

It would be feasibly, but the tradeoff between additional perf boost and additional burden of maintaining a more complex build process is not worth it at this stage. It’s more impactful to spend the effort on making rust-analyzer more performant directly. The primary blocker for that work is understanding rust-analyzer’s heap structure: #9309

[lnicola]

I just tried this, it's an ~8% improvement in analysis-stats, especially in the type inference:

baseline:

Database loaded:     651.08ms, 278minstr
  crates: 36, mods: 715, decls: 14906, fns: 11073
Item Collection:     9.72s, 74ginstr
  exprs: 301267, ??ty: 515 (0%), ?ty: 582 (0%), !ty: 220
Inference:           15.01s, 110ginstr
Total:               24.73s, 185ginstr

pgo:

Database loaded:     638.59ms, 273minstr
  crates: 36, mods: 715, decls: 14906, fns: 11073
Item Collection:     9.43s, 73ginstr
  exprs: 301267, ??ty: 515 (0%), ?ty: 582 (0%), !ty: 220
Inference:           13.28s, 107ginstr
Total:               22.71s, 180ginstr

Steps for posterity, since they weren't obvious:

RUSTFLAGS="-C profile-generate" cargo build --release
target/release/rust-analyzer analysis-stats .
llvm-profdata merge *.profraw --output merged.profdata
RUSTFLAGS="-C profile-use=$PWD/merged.profdata" cargo build --release

Probably not worth the hassle for now.

[FilipAndersson245]

Was some time since this issue and RA is maturing quite nicely, maybe we should look over this again?
saw Kobzol made this https://github.com/Kobzol/cargo-pgo to simplify the process of getting a PGO binary.

[lnicola]

cargo pgo bolt build --with-pgo appears to crash BOLT on my system (LLVM 14.0.6, BOLT 14.0.6), and BOLT doesn't seem to be packaged with LLVM 14 in the distros I've tried, making it a bit hard to acquire. I still don't think it's worth bothering with that. As for plain PGO, on self:

# baseline
Database loaded:     508.15ms, 256minstr (metadata 317.41ms, 23minstr; build 102.87ms, 9210kinstr)
  crates: 43, mods: 869, decls: 19387, fns: 14479
Item Collection:     10.34s, 89ginstr
  exprs: 411733, ??ty: 45 (0%), ?ty: 130 (0%), !ty: 1                                                                                                                
Inference:           37.58s, 265ginstr
Total:               47.92s, 355ginstr

# pgo
Database loaded:     489.57ms, 248minstr (metadata 304.50ms, 21minstr; build 104.55ms, 8241kinstr)
  crates: 43, mods: 869, decls: 19387, fns: 14479
Item Collection:     8.99s, 76ginstr
  exprs: 411733, ??ty: 45 (0%), ?ty: 130 (0%), !ty: 1                                                                                                                
Inference:           29.43s, 227ginstr
Total:               38.42s, 304ginstr

for a speed-up of 20%.

For sysroot:

# baseline
Database loaded:     570.04ms, 132minstr (metadata 250.14ms, 2907kinstr; build 280.13ms, 31minstr)
  crates: 35, mods: 1267, decls: 39546, fns: 26480
Item Collection:     4.28s, 46ginstr
  exprs: 421064, ??ty: 42223 (10%), ?ty: 18705 (4%), !ty: 265                                                       
Inference:           29.26s, 213ginstr
Total:               33.54s, 259ginstr

# pgo
Database loaded:     547.65ms, 126minstr (metadata 253.65ms, 2687kinstr; build 255.81ms, 26minstr)
  crates: 35, mods: 1267, decls: 39546, fns: 26480
Item Collection:     3.71s, 41ginstr
  exprs: 421064, ??ty: 42223 (10%), ?ty: 18705 (4%), !ty: 265                                                       
Inference:           23.42s, 177ginstr
Total:               27.13s, 219ginstr

(19% faster)

My steps in #9412 (comment) still work, but seem to yield a smaller improvement. I'm not sure if cargo pgo is doing some extra magic, or it's just measurement noise.

In any case, 15-20% is a decent improvement.

[FilipAndersson245]

Great! This looks awesome, 15-20% is a huge improvement.

[lnicola]

15-20% is a drop in the ocean compared to the algorithmic improvements (that nobody had time/managed/knew how to make) 😅.

[zamazan4ik]

Just to history - I've applied PGO (without BOLT) to Clangd (a project similar to Rust Analyzer but for C++) and got nice improvements as well: link.

[FilipAndersson245]

@lnicola It has been quite some time since the last time PGO was evaluated, Are rust-analyzer in a better state where it may be suitable to distribute PGO-optimized builds?

[ofek]

Are PGO builds now distributed?

[ChayimFriedman2]

Are PGO builds now distributed?

No. There are still bigger wins to be gained (which nobody tries currently AFAIK).

[ofek]

Are you referring to this? #9309

[ChayimFriedman2]

@ofek Performance work is tracked in #17491.

[berkus]

@ChayimFriedman2 jfyi, i am using rust-analyzer every day, all day - every single percent shaved off of runtime of this tool is a huge win for my work day. I urge you to not ignore a 15% win that is basically coming for free.

[FilipAndersson245]

I think as of right now there is a bug with PGO + LTO in the rust compiler preventing both to be used simultaneously, so any exploration of eventual performance would probably be best after that is fixed.
rust-lang/rust#115344

[FilipAndersson245]

The problem with pgo in combination with LTO have now been fixed so PGO can be explored again.

[Veykril]

#19582 does this now

[FilipAndersson245]

#19582 is awesome, but is there a blocker from enabling it on windows?

[zamazan4ik]

If we want to provide PGO-optimized binaries to users, I think we need to reopen this issue or create another one due to this PR: #19583

[FilipAndersson245]

Seem the problem with zigbuild not working is maybe something that have to be changed in zig.
rust-cross/cargo-zigbuild#315

[Kobzol]

I extended the PGO logic and changed CI to use PGO for Windows. If someone with a Windows machine would like to test the performance effect, the instructions are in #19585.

[Kobzol]

• Use PGO on Linux x64 builds #19595 enabled PGO on x64 Linux
• Allow training PGO on a custom crate and enable it Windows on CI #19585 enabled PGO on Windows
• Build aarch64 builds on CI with PGO #19597 enabled PGO on 64-bit ARM (aarch64)

[zamazan4ik]

@Kobzol you mentioned in #19582 (comment) that PGO on macOS is quite broken. Could you please elaborate on this a bit more? Is there any open issue for that with any details? I am asking it since I don't remember issues with PGO on macOS platform.

[Kobzol]

It's possible I was being a bit too pessimistic, I remember having a lot of trouble to apply PGO on the compiler itself on macOS. That being said, it was some time ago, and it might have actually been on the LLVM side, not the rustc side.

Would you like to try PGO on macOS for rust-analyzer? It should be similar to the PRs I already sent.

[zamazan4ik]

Yeah, let's try it for macOS too. I'll create a PR for that.

[zamazan4ik]

We enabled PGO for macOS platform too. The results are available here: #19611 (comment)