Room for potential CI speedups

[AntoniosBarotsis]

Looking at a CI run, both build-linux-arm and build-linux-musl don't appear to be using a cache, is this intentional? If not, adding one should reduce the step times by a decent margin. They definitely won't be as quick as the native builds but a gain is a gain 😅

Additionally, there is a wall of deprecation warnings, most of which should be coming from actions-rs/toolchain@v1 which is not maintained anymore, a maintained alternative is https://github.com/dtolnay/rust-toolchain.

Another suggestion would be to perhaps look into rust-cache for caching instead of doing it manually as it comes with some more defaults that might be useful.

[extrawurst]

Feel free to make PRs so we can compare :)

[AntoniosBarotsis]

Will draft something in the upcoming days

[extrawurst]

looking to put this into action in v0.24.1 patch release tomorrow.