Thank you for this. Could you please be a little more specific so it's easier to identify the performance issues? I.e. are you doing full scan or sync? What parameters are you using? How many txs per address and txs in total?

However, just off the top of my head, I think it is very likely it's because we are fetching and checking the merkle proofs per tx in the new bdk_electrum (I'm assuming the fetching part is more of an issue).

Not fetching the merkle proofs is a no go, as the electrum API only returns the height that a tx was included in (not the block hash). The old BDK was doing things incorrectly.

There are definitely things we can do to help the performance here. The most obvious thing would be to use the Electrum API properly. The Electrum API was intended as a pub-sub API. Work on this is done in an experimental client here: https://github.com/evanlinjin/experiments

We can also fetch merkle proofs asynchronously. This would be much easier if could use async rust code for the networking side (as I am doing in the aforementioned repo).

We can also cache/store/persist the merkle proofs if the tx can already be linked to the current best chain. This requires us to persist/cache headers.

For the short term, based on the way I understand the problem, I think the quickest way to speed things up is to fetch merkle proofs and headers in a batch and then do processing later on. Instead of fetch -> process -> fetch -> process as we are doing now. This will reduce round trips.

thanks for the fast answer.

the parameters are:

• full_scan stop_gap=500, batch_size=100, fetch_prev_txouts=True

Having a stop gap of 500 means you'll be scanning 500 extra addresses. Does reducing this improve the situation?

The stop gap is user defined in Bitcoin Safe, and therefore is not up to me to improve performance.
For this particular case the stop gap cannot be lowered without missing txs.

We also noticed a similar slowdown when upgrading to the latest bdk_electrum crate and determined that it was related to the Merkle proofs validation step. I'm hoping that the resolution of bitcoindevkit/bdk_wallet#56 will improve the performance in this regard.