Prevent Memory Buildup by Clearing Joinsets

[carlaKC]

TIL that the memory for a task spawned in a joinset will not be released until its corresponding entry in the joinset is consumed ❗ ❗

This means that the pattern that we currently use of spawning a bunch of tasks then waiting on the joinset at the end of the simulation results in massive memory blowup over time, because we never free the memory from out completed tasks.

Created this PR while diagnosing this issue, and opening it up so that it's available if anyone needs a quickndirty fix.
This can and should be done in a slightly neater way!

[carlaKC]

I'm going to be OOO for a week (back 3 March), just opening this so that it's here!

[elnosh]

hi, I'm new here :) as discussed in the call I thought to have a background task clearing those tasks from the JoinSet.

I was playing a bit with it and added something roughly like this to wait_for_shutdown where that function could be running in the background.
`

    loop {
        tokio::select! {
            _ = self.shutdown_listener.clone() => {
                while let Some(res) = self.tasks.join_next().await {
                    if let Err(e) = res {
                        log::error!("Graph task exited with error: {e}");
                    } 
                }
                break;
            },
            _ = self.tasks.join_next() => {}
        }
    };

`

This would loop and clear tasks from the set as they are finished or clear them all if it received a shutdown signal.

I tried running this code with the scenario from this branch https://github.com/carlaKC/sim-ln/tree/review-club-memleak with Heaptrack and it seems to be clearing the tasks as finished and avoiding the memory buildup.

[carlaKC]

hi, I'm new here :)

Welcome!

I was playing a bit with it and added something roughly like this to wait_for_shutdown where that function could be running in the background.

That looks great!

Would you like to put this all together and take over the PR? I'm quite short on time at the moment, but would happily review once you've put it all together!

[elnosh]

sure!

[carlaKC]

Replaced by #229 🚀