Skip to content

feat(NODE-6507): generate encryption configuration on mongoose connect #15320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Apr 15, 2025
Merged

feat(NODE-6507): generate encryption configuration on mongoose connect #15320

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
db8eef7
Add support for encrypted models and discriminators
baileympearson Mar 17, 2025
f1c986c
misc test cleanups
baileympearson Mar 18, 2025
682fb11
update setup script
baileympearson Mar 18, 2025
cda0e26
misc cleanup
baileympearson Mar 18, 2025
edfbdfa
documentation
baileympearson Mar 18, 2025
1a48b98
doc comments
baileympearson Mar 18, 2025
065bc99
use object fromentries
baileympearson Mar 18, 2025
a8f37eb
fix lint in markdown file
baileympearson Mar 18, 2025
53d31a6
throw errors when discriminators have duplicate keys
baileympearson Mar 19, 2025
888c0a8
Remove table testing in favor of hard-coded it blocks
baileympearson Mar 19, 2025
5d3b51f
consistent capitalization in docs
baileympearson Mar 19, 2025
2c02d45
add missing configuration
baileympearson Mar 19, 2025
ecb3f7c
Add support for encrypted discriminators
baileympearson Apr 2, 2025
b7016de
add test for default mongoose connection
baileympearson Apr 2, 2025
14a40ff
add test for default mongoose connection
baileympearson Apr 2, 2025
59af7cf
fix discriminator logic
baileympearson Apr 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 23 additions & 9 deletions lib/drivers/node-mongodb-native/connection.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,10 @@ NativeConnection.prototype.createClient = async function createClient(uri, optio

const { schemaMap, encryptedFieldsMap } = this._buildEncryptionSchemas();

if ((Object.keys(schemaMap).length > 0 || Object.keys(encryptedFieldsMap).length) && !options.autoEncryption) {
throw new Error('Must provide `autoEncryption` when connecting with encrypted schemas.');
}

if (Object.keys(schemaMap).length > 0) {
options.autoEncryption.schemaMap = schemaMap;
}
Expand Down Expand Up @@ -365,20 +369,30 @@ NativeConnection.prototype._buildEncryptionSchemas = function() {
const qeMappings = {};
const csfleMappings = {};

const encryptedModels = Object.values(this.models).filter(model => model.schema._hasEncryptedFields());

// If discriminators are configured for the collection, there might be multiple models
// pointing to the same namespace. For this scenario, we merge all the schemas for each namespace
// into a single schema.
// Notably, this doesn't allow for discriminators to declare multiple values on the same fields.
for (const model of Object.values(this.models)) {
// into a single schema and then generate a schemaMap/encryptedFieldsMap for the combined schema.
for (const model of encryptedModels) {
const { schema, collection: { collectionName } } = model;
const namespace = `${this.$dbName}.${collectionName}`;
if (schema.encryptionType() === 'csfle') {
csfleMappings[namespace] ??= new Schema({}, { encryptionType: 'csfle' });
csfleMappings[namespace].add(schema);
} else if (schema.encryptionType() === 'queryableEncryption') {
qeMappings[namespace] ??= new Schema({}, { encryptionType: 'queryableEncryption' });
qeMappings[namespace].add(schema);
const mappings = schema.encryptionType() === 'csfle' ? csfleMappings : qeMappings;

mappings[namespace] ??= new Schema({}, { encryptionType: schema.encryptionType() });

const isNonRootDiscriminator = schema.discriminatorMapping && !schema.discriminatorMapping.isRoot;
if (isNonRootDiscriminator) {
const rootSchema = schema._baseSchema;
schema.eachPath((pathname) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential edge case here: discriminator base schema defines nested path, discriminator child schema defines subdocument with same path but different options.

const schema = new Schema({
  name: {
    first: { type: String, encrypt: { keyId: [keyId], algorithm } }
  }
});

const discriminatorSchema = new Schema({
  name: new Schema({ first: Number }) // Different type, no encryption, stored as same field in MDB
});

schema.eachpath() doesn't account for subdocuments because subdocuments have a distinct schema (it does account for nested paths though because nested paths do not have their own schema)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - I wrote a test for this scenario and you were right.

I've been trying out some different approaches and I can't find a better solution than something like this:

  function* allPaths(schema, prefix) {
    for (const path of Object.keys(schema.paths)) {
      const fullPath = prefix != null ? `${prefix}.${path}` : path;
      if (schema.path(path).instance === 'Embedded') {
        yield* allPaths(schema.path(path).schema, fullPath);
      } else {
        yield fullPath;
      }
    }
  }

  const paths = new Set(allPaths(schema));

  for (const path of allPaths(model.schema)) {
    if (paths.has(path) && (model.schema._hasEncryptedField(path) || schema._hasEncryptedField(path))) {
      throw new Error(`cannot declare an encrypted field on child schema overriding base schema. key=${path}`);
    }
  }

This generates all possible paths in the schema (the above doesn't handle arrays, but encryption on fields in arrays isn't supported so that's out of scope here). Not my first choice, because this feels brittle. I'll keep looking at it but

  1. Can you think of a better solution here than recursively iterating over all paths + nested schemas?
  2. If not, does a utility that I could use like allPaths exist somewhere?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think recursively checking all paths is necessary here because you just want to find conflicts in the top-level paths. So if discriminator schema has a path pathname with an encrypted field, and root schema has a nested path with rootSchema.nested[pathname.split('.')[0]], you can already call that a conflict and throw an error. Similarly, if discriminator schema has a nested path pathname but in root schema you have rootSchema.paths[pathname] then you can also throw an error.

Copy link
Contributor Author

@baileympearson baileympearson Apr 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily, right? That would throw an error in scenarios where the child schema provides a subdocument with the same root path but doesn't modify the encrypted path. ex:

const schema = new Schema({
  name: {
    first: { type: String, encrypt: { keyId: [keyId], algorithm } }
  }
});

const discriminatorSchema = new Schema({
  name: new Schema({ age: Number }) // Different path, no encryption
});

I'd expect this to be fine, because there isn't a conflicting path for name.first.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case you described, name: new Schema({ age: Number }) would actually override name: { first: { type: String } }, so the discriminator schema would not have a name.first property at all. Discriminators merge nested paths from root schema, but subdocuments override because subdocuments can have middleware.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the approach of determining conflicting base paths, as you suggested but I ran into two complications:

  1. It isn't enough to simply detect base path collisions, you need to check that something in the schema also has an encrypted field that uses the base path as well. This is feasible but requires some logic to allow for asking "is there an encrypted field at some subpath of this path?"
  2. It felt like there were more corner cases than not - schemas with only one level of nesting, one level of subdocuments, sub documents on parent but not child and vice versa, multiple layers of nesting of subdocuments, etc.

I decided that the two above points complicated the approach. The changes in this PR were much simpler for me to work with.. This approach works by:

  1. finding the intersection of all keypaths for all fields (including subdocuments) between the parent and child
  2. for each keypath, validates that this keypath isn't encrypted on the parent and on the child

Let me know what you think - I can rework it if you would prefer a different approach.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, this is a reasonable approach. This approach is more restrictive, but simpler implementation. If it proves to be too restrictive, we can come up with a more flexible approach.

if (rootSchema.path(pathname)) return;
if (!mappings[namespace]._hasEncryptedField(pathname)) return;

throw new Error(`Cannot have duplicate keys in discriminators with encryption. key=${pathname}`);
});
}

mappings[namespace].add(schema);
}

const schemaMap = Object.fromEntries(Object.entries(csfleMappings).map(
Expand Down
7 changes: 7 additions & 0 deletions lib/helpers/model/discriminator.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,13 @@ module.exports = function discriminator(model, name, schema, tiedValue, applyPlu
const baseSchemaPaths = Object.keys(baseSchema.paths);
const conflictingPaths = [];


baseSchema.eachPath((pathname) => {
if (schema._hasEncryptedField(pathname)) {
throw new Error(`cannot declare an encrypted field on child schema overriding base schema. key=${pathname}`);
}
});

for (const path of baseSchemaPaths) {
if (schema.nested[path]) {
conflictingPaths.push(path);
Expand Down
9 changes: 8 additions & 1 deletion lib/schema.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -721,7 +721,6 @@ Schema.prototype.encryptionType = function encryptionType(encryptionType) {
Schema.prototype.add = function add(obj, prefix) {
if (obj instanceof Schema || (obj != null && obj.instanceOfSchema)) {
merge(this, obj);

return this;
}

Expand Down Expand Up @@ -914,6 +913,14 @@ Schema.prototype._hasEncryptedFields = function _hasEncryptedFields() {
return Object.keys(this.encryptedFields).length > 0;
};

/**
* @api private
*/
Schema.prototype._hasEncryptedField = function _hasEncryptedField(path) {
return path in this.encryptedFields;
};


/**
* Builds an encryptedFieldsMap for the schema.
*/
Expand Down
127 changes: 127 additions & 0 deletions test/encryption/encryption.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -797,6 +797,133 @@ describe('encryption integration tests', () => {
});
});

describe('duplicate keys in discriminators', function() {
beforeEach(async function() {
connection = createConnection();
});
describe('csfle', function() {
it('throws on duplicate keys declared on different discriminators', async function() {
const schema = new Schema({
name: {
type: String, encrypt: { keyId: [keyId], algorithm }
}
}, {
encryptionType: 'csfle'
});
model = connection.model('Schema', schema);
discrim1 = model.discriminator('Test', new Schema({
age: {
type: Int32, encrypt: { keyId: [keyId], algorithm }
}
}, {
encryptionType: 'csfle'
}));

discrim2 = model.discriminator('Test2', new Schema({
age: {
type: Int32, encrypt: { keyId: [keyId], algorithm }
}
}, {
encryptionType: 'csfle'
}));

const error = await connection.openUri(process.env.MONGOOSE_TEST_URI, {
dbName: 'db', autoEncryption: {
keyVaultNamespace: 'keyvault.datakeys',
kmsProviders: { local: { key: LOCAL_KEY } },
extraOptions: {
cryptdSharedLibRequired: true,
cryptSharedLibPath: process.env.CRYPT_SHARED_LIB_PATH
}
}
}).catch(e => e);

assert.ok(error instanceof Error);
assert.match(error.message, /Cannot have duplicate keys in discriminators with encryption/);
});
it('throws on duplicate keys declared on root and child discriminators', async function() {
const schema = new Schema({
name: {
type: String, encrypt: { keyId: [keyId], algorithm }
}
}, {
encryptionType: 'csfle'
});
model = connection.model('Schema', schema);
assert.throws(() => model.discriminator('Test', new Schema({
name: {
type: String, encrypt: { keyId: [keyId], algorithm }
}
}, {
encryptionType: 'csfle'
})),
/cannot declare an encrypted field on child schema overriding base schema\. key=name/
);
});
});

describe('queryable encryption', function() {
it('throws on duplicate keys declared on different discriminators', async function() {
const schema = new Schema({
name: {
type: String, encrypt: { keyId }
}
}, {
encryptionType: 'queryableEncryption'
});
model = connection.model('Schema', schema);
discrim1 = model.discriminator('Test', new Schema({
age: {
type: Int32, encrypt: { keyId: keyId2 }
}
}, {
encryptionType: 'queryableEncryption'
}));

discrim2 = model.discriminator('Test2', new Schema({
age: {
type: Int32, encrypt: { keyId: keyId3 }
}
}, {
encryptionType: 'queryableEncryption'
}));

const error = await connection.openUri(process.env.MONGOOSE_TEST_URI, {
dbName: 'db', autoEncryption: {
keyVaultNamespace: 'keyvault.datakeys',
kmsProviders: { local: { key: LOCAL_KEY } },
extraOptions: {
cryptdSharedLibRequired: true,
cryptSharedLibPath: process.env.CRYPT_SHARED_LIB_PATH
}
}
}).catch(e => e);

assert.ok(error instanceof Error);
assert.match(error.message, /Cannot have duplicate keys in discriminators with encryption/);
});
it('throws on duplicate keys declared on root and child discriminators', async function() {
const schema = new Schema({
name: {
type: String, encrypt: { keyId }
}
}, {
encryptionType: 'queryableEncryption'
});
model = connection.model('Schema', schema);
assert.throws(() => model.discriminator('Test', new Schema({
name: {
type: String, encrypt: { keyId: keyId2 }
}
}, {
encryptionType: 'queryableEncryption'
})),
/cannot declare an encrypted field on child schema overriding base schema\. key=name/
);
});
});
});

});
});
});