RUST-2155 mongo failed with tokio no reactor running since 3.2.0

[Xuanwo]

Versions/Environment

1. What version of Rust are you using?

Latest stable rust.

2. What operating system are you using?

Linux (github action ubuntu 24.04)

3. What versions of the driver and its dependencies are you using? (Run
   cargo pkgid mongodb & cargo pkgid bson)

3.2.0

4. What version of MongoDB are you using? (Check with the MongoDB shell using db.version())

No related. (unable to connect since client side panic)

5. What is your MongoDB topology (standalone, replica set, sharded cluster, serverless)?

No related. (unable to connect since client side panic)

Describe the bug

It used to work correctly 3.1.1, but failed since 3.2.0

thread '<unnamed>' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mongodb-3.2.0/src/client.rs:685:13:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panicking.rs:76:14
   2: core::panicking::panic_display
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panicking.rs:269:5
   3: tokio::runtime::scheduler::Handle::current::panic_cold_display
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panic.rs:100:13
   4: tokio::runtime::scheduler::Handle::current
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/scheduler/mod.rs:108:27
   5: tokio::runtime::handle::Handle::current
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/handle.rs:141:20
   6: mongodb::runtime::join_handle::AsyncJoinHandle<T>::spawn
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mongodb-3.2.0/src/runtime/join_handle.rs:19:22
   7: mongodb::runtime::spawn
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mongodb-3.2.0/src/runtime.rs:65:5
   8: <mongodb::client::Client as core::ops::drop::Drop>::drop
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mongodb-3.2.0/src/client.rs:685:13
   9: core::ptr::drop_in_place<mongodb::client::Client>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  10: core::ptr::drop_in_place<mongodb::db::DatabaseInner>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  11: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  12: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  13: core::ptr::drop_in_place<alloc::sync::Arc<mongodb::db::DatabaseInner>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  14: core::ptr::drop_in_place<mongodb::db::Database>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  15: core::ptr::drop_in_place<mongodb::coll::CollectionInner>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  16: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  17: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  18: core::ptr::drop_in_place<alloc::sync::Arc<mongodb::coll::CollectionInner>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  19: core::ptr::drop_in_place<mongodb::coll::Collection<bson::document::Document>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  20: <tokio::sync::once_cell::OnceCell<T> as core::ops::drop::Drop>::drop::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/sync/once_cell.rs:109:37
  21: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/loom/std/unsafe_cell.rs:16:9
  22: <tokio::sync::once_cell::OnceCell<T> as core::ops::drop::Drop>::drop
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/sync/once_cell.rs:108:17
  23: core::ptr::drop_in_place<tokio::sync::once_cell::OnceCell<mongodb::coll::Collection<bson::document::Document>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  24: core::ptr::drop_in_place<opendal::services::mongodb::backend::Adapter>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  25: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  26: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  27: core::ptr::drop_in_place<alloc::sync::Arc<opendal::services::mongodb::backend::Adapter>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  28: core::ptr::drop_in_place<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  29: core::ptr::drop_in_place<opendal::layers::error_context::ErrorContextAccessor<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  30: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  31: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  32: core::ptr::drop_in_place<alloc::sync::Arc<opendal::layers::error_context::ErrorContextAccessor<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  33: core::ptr::drop_in_place<opendal::layers::complete::CompleteAccessor<opendal::layers::error_context::ErrorContextAccessor<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  34: core::ptr::drop_in_place<opendal::layers::correctness_check::CorrectnessAccessor<opendal::layers::complete::CompleteAccessor<opendal::layers::error_context::ErrorContextAccessor<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  35: core::ptr::drop_in_place<opendal::layers::type_eraser::TypeEraseAccessor<opendal::layers::correctness_check::CorrectnessAccessor<opendal::layers::complete::CompleteAccessor<opendal::layers::error_context::ErrorContextAccessor<opendal::raw::adapters::kv::backend::Backend<opendal::services::mongodb::backend::Adapter>>>>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  36: core::ptr::drop_in_place<dyn opendal::raw::accessor::AccessDyn>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  37: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  38: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  39: core::ptr::drop_in_place<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  40: core::ptr::drop_in_place<opendal::layers::blocking::BlockingAccessor<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  41: core::ptr::drop_in_place<opendal::layers::type_eraser::TypeEraseAccessor<opendal::layers::blocking::BlockingAccessor<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  42: core::ptr::drop_in_place<dyn opendal::raw::accessor::AccessDyn>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  43: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  44: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  45: core::ptr::drop_in_place<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  46: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  47: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  48: core::ptr::drop_in_place<alloc::sync::Arc<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  49: core::ptr::drop_in_place<opendal::layers::retry::RetryAccessor<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>,opendal::layers::retry::DefaultRetryInterceptor>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  50: core::ptr::drop_in_place<opendal::layers::type_eraser::TypeEraseAccessor<opendal::layers::retry::RetryAccessor<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>,opendal::layers::retry::DefaultRetryInterceptor>>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  51: core::ptr::drop_in_place<dyn opendal::raw::accessor::AccessDyn>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  52: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:1888:18
  53: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/sync.rs:2575:13
  54: core::ptr::drop_in_place<alloc::sync::Arc<dyn opendal::raw::accessor::AccessDyn>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  55: core::ptr::drop_in_place<opendal::types::operator::blocking_operator::BlockingOperator>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  56: core::ptr::drop_in_place<alloc::boxed::Box<opendal::types::operator::blocking_operator::BlockingOperator>>
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ptr/mod.rs:521:1
  57: core::mem::drop
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/mem/mod.rs:942:24
  58: Java_org_apache_opendal_Operator_disposeInternal
             at ./src/operator.rs:44:5
  59: <unknown>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
Aborted (core dumped)

To Reproduce

• Create a gloabl tokio runtime.
• Start mongo driver inside it.

[Xuanwo]

Currently, we can workaround this by exclude 3.2.0:

# mongodb has known issues on 3.2.0: https://github.com/mongodb/mongo-rust-driver/issues/1287
mongodb = { version = ">=3,<3.2.0", optional = true }
mongodb-internal-macros = { version = ">=3,<3.2.0", optional = true }

[abr-egn]

Thanks for the report!

I can't reproduce this failure with a simple client binary (below) - can you give me more information about how you're constructing and managing the client, and ideally minimal code for a reproduction?

From the error stack, it looks like the issue is happening not when the client is being constructed but when it's being dropped; the client does need to do async work to release server-side resources when it's dropped so the associated tokio runtime needs to outlive the client. (This didn't change between 3.1.x and 3.2.0, so I'm not sure why you'd be getting an error when upgrading.)

Test program:

#[tokio::main]
async fn main() {
    let client = mongodb::Client::with_uri_str("mongodb://localhost:27017")
        .await
        .unwrap();
    let names = client.list_database_names().await.unwrap();
    println!("{:?}", names);
}

Cargo.toml:

[package]
name = "minimal-client"
version = "0.1.0"
edition = "2021"

[dependencies]
mongodb = "3.2.0"
tokio = "1.43.0"

Output:

["admin", "config", "local"]

[github-actions]

There has not been any recent activity on this ticket, so we are marking it as stale. If we do not hear anything further from you, this issue will be automatically closed in one week.

[Xuanwo]

Hi, thanks for the response. I'm in holiday currently. I will provide more details and maybe a reproduction later.

[abbudao]

Hi, I'm currently having the same issue. Worked fine before the upgrade to 3.2.1. On my benchmark suites I'm now receiving:

thread 'main' panicked at /home/xxxxx/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mongodb-3.2.1/src/client.rs:685:13:

The related part seems to be the Drop for client:

impl Drop for Client {
    fn drop(&mut self) {
        if !self.inner.shutdown.executed.load(Ordering::SeqCst)
            && !self.inner.dropped.load(Ordering::SeqCst)
            && TrackingArc::strong_count(&self.inner) == 1
        {
            // We need an owned copy of the client to move into the spawned future. However, if this
            // call to drop completes before the spawned future completes, the number of strong
            // references to the inner client will again be 1 when the cloned client drops, and thus
            // end_all_sessions will be called continuously until the runtime shuts down. Storing a
            // flag indicating whether end_all_sessions has already been called breaks
            // this cycle.
            self.inner.dropped.store(true, Ordering::SeqCst);
            let client = self.clone();
            crate::runtime::spawn(async move {
                client.end_all_sessions().await;
            });
        }
    }
}

I couldn't make a reproducer yet but my benchmarks are structured like (simplified version):

use criterion::{black_box, criterion_group, criterion_main};
use tokio::runtime::Runtime;

#[allow(clippy::too_many_lines)]
pub fn criterion_benchmark(_c: &mut Criterion) {
    let tests = ["some_test_name"];
    let rt = Runtime::new().expect("runtime creation failed");

    let mut criterion= Criterion::default();

    for t in tests {
        let env = rt.block_on(async {  setup_function().await });
        criterion.bench_with_input(
            BenchmarkId::new("sample_benchmark_name", t),
            &t,
            |b, _test_name| {
                b.to_async(&rt).iter_with_large_drop(|| async {
                    black_box(
                        env.server
                            .post(&FooPostPath.to_string())
                            .json(&BarRequest{})
                            .await,
                    )
                });
            },
        );
    }
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

From what I've gathered, a change was introduced recently on 3.2.0 on the Drop implementation:
https://jira.mongodb.org/browse/RUST-1437

[abr-egn]

Aha! Thank you for your debugging work, I have a theory now on what's happening. I don't know how I missed RUST-1437 when I was looking for relevant changes here.

As you've noticed, it's hard to make a reliable reproduction case for this error. Once I have a potential fix in place, would it be possible for you to run your benchmarks against a git branch of the driver with that fix?

[abbudao]

@abr-egn thank you for the quick reply! I will gladly validate it when you have a fix in place. Let me know if you need anything else to validate your hypothesis!