Extended $out syntax fails when using Aggregation.out(…).in("dbName")

[ricardohsmello]

Hi team,

I'm using Spring Boot 3.4.4 with Spring Data MongoDB 4.4.4, and I noticed an issue when using the $out aggregation stage together with .in("new_db"). The aggregation works perfectly if I don't include .in(), meaning it defaults to writing in the current database.

Reproduction

void exportBooksWithPublishedYear() {
    Aggregation aggregation = Aggregation.newAggregation(
        Aggregation.project("title", "publishedAt")
                   .andExpression("year(publishedAt)").as("year"),

        Aggregation.out("books_year_export")
                   .in("new_db")  // <--  Only breaks when this is included
                   .mode(OutOperation.OutMode.INSERT)
    );

    AggregationOptions options = AggregationOptions.builder()
        .skipOutput()  // <-- I also tried skipOutput but didn't work
        .build();

    mongoTemplate.aggregate(
        aggregation.withOptions(options),
        Book.BOOK_COLLECTION_NAME,
        Document.class
    );
}

Running on MongoDB Atlas 8.0.9 (Cloud) gives:

com.mongodb.MongoCommandException: Command failed with error 8000 (AtlasError): '$out must contain both db and coll'

{
  "ok": 0,
  "errmsg": "Failed to retrieve database or collection name from $out, err=$out must contain both db and coll",
  "code": 8000,
  "codeName": "AtlasError"
}

Running on MongoDB AtlasLocalDev 8.0.5 (Docker)

Using the same code on the [MongoDB Atlas Local Docker image] gives:

com.mongodb.MongoCommandException: Command failed with error 40415 (IDLUnknownField): 'BSON field '$out.to' is an unknown field.'

{
  "ok": 0.0,
  "errmsg": "BSON field '$out.to' is an unknown field.",
  "code": 40415,
  "codeName": "IDLUnknownField"
}

Environment

• Spring Boot: 3.4.4
• Spring Data MongoDB: 4.4.4
• MongoDB Java Driver: 5.2.1
• Java: 21

I’m not sure if the $out stage is being constructed correctly under the hood when using .in("dbName"), but it seems to be missing expected fields for the newer MongoDB 8.x versions.

Please let me know if I can help with a sample project or further info. Thanks a lot!

[christophstrobl]

thank you @ricardohsmello for bringing this up. Seems the $out stage changed a bit in the past few years when it was originally defined as:

enums:
    WriteMode:
        description: "Possible merge mode values."
        type: string
        values:
            kModeInsert: "insert"
            kModeReplace: "replace"

structs:
    DocumentSourceOutSpec:
        description: A document used to specify the $out stage of an aggregation pipeline.
        strict: true
        fields:
            to:
                cpp_name: targetCollection
                type: string
                description: Name of the target collection. 

            db:
                cpp_name: targetDb
                type: string
                optional: true
                description: Name of the target database, defaults to the database of the
                             aggregation.

            mode:
                cpp_name: mode
                type: WriteMode
                description: The merge mode for the output operation.

            uniqueKey:
                cpp_name: uniqueKey
                type: object
                optional: true
                description: Document of fields representing the unique key.

            dropTarget:
                cpp_name: dropTarget
                type: bool
                default: false
                description: If true, the 'to' collection is atomically dropped and replaced with 
                             the results of the aggregation.

Guess we'll need to update to newer syntax.

[therepanic]

I think we should simply change the syntax from to to coll in OutOperation and in the tests for it? May I work on this?

[mp911de]

The reason the extended syntax is never shown in any reference docs is that $merge has superseded the extended $out syntax that we've captured from a driver state that has never made it to production (https://jira.mongodb.org/browse/JAVA-3278). That being said, we should backport the change as much as possible.