Encode dep node edge count as u32 instead of usize #139636

Zoxc · 2025-04-10T15:38:43Z

This encodes the dep node edge count as u32 instead of usize. It doesn't need to be usize as the count is limited the the unique number of dep nodes which we use a 32-bit index for. It probably saves some branches for encoding / decoding, but it isn't too hot as the count gets packed in with other fields when it's low.

Benchmark	Before	After		Before	After		Before	After
Benchmark	Time	Time	%	Physical Memory	Physical Memory	%	Committed Memory	Committed Memory	%
🟣 clap:check:unchanged	0.3343s	0.3343s	-0.00%	97.02 MiB	96.89 MiB	-0.13%	167.93 MiB	167.80 MiB	-0.08%
🟣 hyper:check:unchanged	0.1353s	0.1350s	-0.19%	61.91 MiB	61.83 MiB	-0.13%	124.64 MiB	124.64 MiB	0.00%
🟣 regex:check:unchanged	0.2479s	0.2481s	0.10%	78.34 MiB	78.35 MiB	0.02%	145.20 MiB	145.31 MiB	0.08%
🟣 syn:check:unchanged	0.5347s	0.5333s	-0.26%	118.63 MiB	118.66 MiB	0.02%	193.05 MiB	193.11 MiB	0.03%
Total	1.2521s	1.2507s	-0.11%	355.90 MiB	355.74 MiB	-0.05%	630.83 MiB	630.87 MiB	0.01%
Summary	1.0000s	0.9991s	-0.09%	1 byte	1.00 bytes	-0.06%	1 byte	1.00 bytes	0.01%

rustbot · 2025-04-10T15:38:48Z

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

compiler-errors · 2025-04-10T16:00:50Z

👀

compiler-errors · 2025-04-13T16:28:20Z

@bors try @rust-timer queue

Encode dep node edge count as u32 instead of usize This encodes the dep node edge count as u32 instead of usize. It doesn't need to be usize as the count is limited the the unique number of dep nodes which we use a 32-bit index for. It probably saves some branches for encoding / decoding, but it isn't too hot as the count gets packed in with other fields when it's low. <table><tr><td rowspan="2">Benchmark</td><td colspan="1">Before</th><td colspan="2">After</th><td colspan="1">Before</th><td colspan="2">After</th><td colspan="1">Before</th><td colspan="2">After</th></tr><tr><td align="right">Time</td><td align="right">Time</td><td align="right">%</th><td align="right">Physical Memory</td><td align="right">Physical Memory</td><td align="right">%</th><td align="right">Committed Memory</td><td align="right">Committed Memory</td><td align="right">%</th></tr><tr><td>🟣 clap:check:unchanged</td><td align="right">0.3343s</td><td align="right">0.3343s</td><td align="right"> -0.00%</td><td align="right">97.02 MiB</td><td align="right">96.89 MiB</td><td align="right"> -0.13%</td><td align="right">167.93 MiB</td><td align="right">167.80 MiB</td><td align="right"> -0.08%</td></tr><tr><td>🟣 hyper:check:unchanged</td><td align="right">0.1353s</td><td align="right">0.1350s</td><td align="right"> -0.19%</td><td align="right">61.91 MiB</td><td align="right">61.83 MiB</td><td align="right"> -0.13%</td><td align="right">124.64 MiB</td><td align="right">124.64 MiB</td><td align="right"> 0.00%</td></tr><tr><td>🟣 regex:check:unchanged</td><td align="right">0.2479s</td><td align="right">0.2481s</td><td align="right"> 0.10%</td><td align="right">78.34 MiB</td><td align="right">78.35 MiB</td><td align="right"> 0.02%</td><td align="right">145.20 MiB</td><td align="right">145.31 MiB</td><td align="right"> 0.08%</td></tr><tr><td>🟣 syn:check:unchanged</td><td align="right">0.5347s</td><td align="right">0.5333s</td><td align="right"> -0.26%</td><td align="right">118.63 MiB</td><td align="right">118.66 MiB</td><td align="right"> 0.02%</td><td align="right">193.05 MiB</td><td align="right">193.11 MiB</td><td align="right"> 0.03%</td></tr><tr><td>Total</td><td align="right">1.2521s</td><td align="right">1.2507s</td><td align="right"> -0.11%</td><td align="right">355.90 MiB</td><td align="right">355.74 MiB</td><td align="right"> -0.05%</td><td align="right">630.83 MiB</td><td align="right">630.87 MiB</td><td align="right"> 0.01%</td></tr><tr><td>Summary</td><td align="right">1.0000s</td><td align="right">0.9991s</td><td align="right"> -0.09%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> -0.06%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> 0.01%</td></tr></table>

bors · 2025-04-13T16:29:31Z

⌛ Trying commit 22dd86c with merge 98ea798...

compiler-errors · 2025-04-13T17:42:14Z

Anyways r=me if perf comes back neutral or good

@bors delegate+

bors · 2025-04-13T17:42:17Z

✌️ @Zoxc, you can now approve this pull request!

If @compiler-errors told you to "r=me" after making some further change, please make that change, then do @bors r=@compiler-errors

bors · 2025-04-13T18:33:19Z

☀️ Try build successful - checks-actions
Build commit: 98ea798 (98ea798569623436da636f880c1c04430337b720)

Allow out of order dep graph node encoding This allows out of order dep graph node encoding by also encoding the index instead of using the file node order as the index. `MemEncoder` is also brought back to life and used for encoding. Both of these are done to enable thread-local encoding of dep graph nodes. This is based on rust-lang#139636.

Use thread local dep graph encoding This adds thread local encoding of dep graph nodes. Each thread has a `MemEncoder` that gets flushed to the global `FileEncoder` when it exceeds 64 kB. Each thread also has a local cache of dep indices. This means there can now be empty gaps in `SerializedDepGraph`. Indices are marked green and also allocated by the new atomic operation `DepNodeColorMap::try_mark_green` as the encoder lock is removed. This is based on rust-lang#139636 and rust-lang#139756.

rust-timer · 2025-04-13T20:37:59Z

Finished benchmarking commit (98ea798): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 2.7%, secondary -7.8%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.7%	[2.7%, 2.7%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-7.8%	[-7.8%, -7.8%]	1
All ❌✅ (primary)	2.7%	[2.7%, 2.7%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 780.32s -> 781.151s (0.11%)
Artifact size: 365.51 MiB -> 365.51 MiB (-0.00%)

compiler-errors · 2025-04-13T22:46:47Z

cool

@bors r+ rollup=maybe

bors · 2025-04-13T22:46:49Z

📌 Commit 22dd86c has been approved by compiler-errors

It is now in the queue for this repository.

Rollup of 9 pull requests Successful merges: - rust-lang#138336 (Improve `-Z crate-attr` diagnostics) - rust-lang#139636 (Encode dep node edge count as u32 instead of usize) - rust-lang#139666 (cleanup `mir_borrowck`) - rust-lang#139695 (compiletest: consistently use `camino::{Utf8Path,Utf8PathBuf}` throughout) - rust-lang#139699 (Proactively update coroutine drop shim's phase to account for later passes applied during shim query) - rust-lang#139718 (enforce unsafe attributes in pre-2024 editions by default) - rust-lang#139722 (Move some things to rustc_type_ir) - rust-lang#139760 (UI tests: migrate remaining compile time `error-pattern`s to line annotations when possible) - rust-lang#139776 (Switch attrs to `diagnostic::on_unimplemented`) r? `@ghost` `@rustbot` modify labels: rollup

Allow out of order dep graph node encoding This allows out of order dep graph node encoding by also encoding the index instead of using the file node order as the index. `MemEncoder` is also brought back to life and used for encoding. Both of these are done to enable thread-local encoding of dep graph nodes. This is based on rust-lang#139636.

Rollup merge of rust-lang#139636 - Zoxc:graph-edges-len-u32, r=compiler-errors Encode dep node edge count as u32 instead of usize This encodes the dep node edge count as u32 instead of usize. It doesn't need to be usize as the count is limited the the unique number of dep nodes which we use a 32-bit index for. It probably saves some branches for encoding / decoding, but it isn't too hot as the count gets packed in with other fields when it's low. <table><tr><td rowspan="2">Benchmark</td><td colspan="1">Before</th><td colspan="2">After</th><td colspan="1">Before</th><td colspan="2">After</th><td colspan="1">Before</th><td colspan="2">After</th></tr><tr><td align="right">Time</td><td align="right">Time</td><td align="right">%</th><td align="right">Physical Memory</td><td align="right">Physical Memory</td><td align="right">%</th><td align="right">Committed Memory</td><td align="right">Committed Memory</td><td align="right">%</th></tr><tr><td>🟣 clap:check:unchanged</td><td align="right">0.3343s</td><td align="right">0.3343s</td><td align="right"> -0.00%</td><td align="right">97.02 MiB</td><td align="right">96.89 MiB</td><td align="right"> -0.13%</td><td align="right">167.93 MiB</td><td align="right">167.80 MiB</td><td align="right"> -0.08%</td></tr><tr><td>🟣 hyper:check:unchanged</td><td align="right">0.1353s</td><td align="right">0.1350s</td><td align="right"> -0.19%</td><td align="right">61.91 MiB</td><td align="right">61.83 MiB</td><td align="right"> -0.13%</td><td align="right">124.64 MiB</td><td align="right">124.64 MiB</td><td align="right"> 0.00%</td></tr><tr><td>🟣 regex:check:unchanged</td><td align="right">0.2479s</td><td align="right">0.2481s</td><td align="right"> 0.10%</td><td align="right">78.34 MiB</td><td align="right">78.35 MiB</td><td align="right"> 0.02%</td><td align="right">145.20 MiB</td><td align="right">145.31 MiB</td><td align="right"> 0.08%</td></tr><tr><td>🟣 syn:check:unchanged</td><td align="right">0.5347s</td><td align="right">0.5333s</td><td align="right"> -0.26%</td><td align="right">118.63 MiB</td><td align="right">118.66 MiB</td><td align="right"> 0.02%</td><td align="right">193.05 MiB</td><td align="right">193.11 MiB</td><td align="right"> 0.03%</td></tr><tr><td>Total</td><td align="right">1.2521s</td><td align="right">1.2507s</td><td align="right"> -0.11%</td><td align="right">355.90 MiB</td><td align="right">355.74 MiB</td><td align="right"> -0.05%</td><td align="right">630.83 MiB</td><td align="right">630.87 MiB</td><td align="right"> 0.01%</td></tr><tr><td>Summary</td><td align="right">1.0000s</td><td align="right">0.9991s</td><td align="right"> -0.09%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> -0.06%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> 0.01%</td></tr></table>

Allow out of order dep graph node encoding This allows out of order dep graph node encoding by also encoding the index instead of using the file node order as the index. `MemEncoder` is also brought back to life and used for encoding. Both of these are done to enable thread-local encoding of dep graph nodes. This is based on rust-lang#139636.

rustbot assigned estebank Apr 10, 2025

This comment has been minimized.

Sign in to view

compiler-errors assigned compiler-errors and unassigned estebank Apr 10, 2025

Encode dep node edge count as u32 instead of usize

22dd86c

Zoxc force-pushed the graph-edges-len-u32 branch from dd7eec5 to 22dd86c Compare April 10, 2025 19:02

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 13, 2025

Zoxc mentioned this pull request Apr 13, 2025

Allow out of order dep graph node encoding #139756

Open

This comment has been minimized.

Sign in to view

Zoxc mentioned this pull request Apr 13, 2025

Use thread local dep graph encoding #139758

Open

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 13, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 13, 2025

jhpratt mentioned this pull request Apr 14, 2025

Rollup of 9 pull requests #139781

Merged

bors merged commit 5c494aa into rust-lang:master Apr 14, 2025
7 checks passed

rustbot added this to the 1.88.0 milestone Apr 14, 2025

Zoxc deleted the graph-edges-len-u32 branch April 14, 2025 13:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode dep node edge count as u32 instead of usize #139636

Encode dep node edge count as u32 instead of usize #139636

Zoxc commented Apr 10, 2025

rustbot commented Apr 10, 2025

This comment has been minimized.

compiler-errors commented Apr 10, 2025

compiler-errors commented Apr 13, 2025

This comment has been minimized.

bors commented Apr 13, 2025

compiler-errors commented Apr 13, 2025

bors commented Apr 13, 2025

bors commented Apr 13, 2025

This comment has been minimized.

rust-timer commented Apr 13, 2025

compiler-errors commented Apr 13, 2025

bors commented Apr 13, 2025

Encode dep node edge count as u32 instead of usize #139636

Encode dep node edge count as u32 instead of usize #139636

Conversation

Zoxc commented Apr 10, 2025

rustbot commented Apr 10, 2025

This comment has been minimized.

compiler-errors commented Apr 10, 2025

compiler-errors commented Apr 13, 2025

This comment has been minimized.

bors commented Apr 13, 2025

compiler-errors commented Apr 13, 2025

bors commented Apr 13, 2025

bors commented Apr 13, 2025

This comment has been minimized.

rust-timer commented Apr 13, 2025

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

compiler-errors commented Apr 13, 2025

bors commented Apr 13, 2025