From 54bdcf6551288651637c31d5cc40b1da40227ff2 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Wed, 27 Jul 2022 15:10:21 -0500 Subject: [PATCH 1/9] DRIVERS-2333 Cache AWS Credentials Where Possible --- source/auth/auth.rst | 8 ++++++++ source/auth/tests/mongodb-aws.rst | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index c4d51cbf35..ebc2a728f7 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1098,6 +1098,14 @@ From the JSON response drivers MUST obtain the ``access_key``, ``secret_key`` and ``security_token`` which will be used during the `Signature Version 4 Signing Process `_. +Caching Credentials +___________________ +Credentials fetched by the driver using AWS endpoints MUST be cached and reused +to avoid hitting rate limitations. The "Expiration" field must be stored and +used to determine when to clear the cache. If the "Expiration" is within 5 +minutes of the current UTC time, the cache must be cleared. +If AWS authentication fails for any reason, the cache must be cleared. + ------------------------- Connection String Options ------------------------- diff --git a/source/auth/tests/mongodb-aws.rst b/source/auth/tests/mongodb-aws.rst index 43a5ed31ff..9c770fab9f 100644 --- a/source/auth/tests/mongodb-aws.rst +++ b/source/auth/tests/mongodb-aws.rst @@ -10,6 +10,7 @@ There are 6 scenarios drivers MUST test: #. ``Assume Role``: Auth via temporary credentials obtained from an STS AssumeRole request #. ``Assume Role with Web Identity``: Auth via temporary credentials obtained from an STS AssumeRoleWithWebIdentity request #. ``AWS Lambda``: Auth via environment variables ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``, and ``AWS_SESSION_TOKEN``. +#. Caching of AWS credentials fetched by the driver. For brevity, this section gives the values ````, ```` and ```` in place of a valid access key ID, secret access key and session token (also known as a security token). Note that if these values are passed into the URI they MUST be URL encoded. Sample values are below. @@ -115,3 +116,26 @@ Sample URIs both with and without optional session tokens set are shown below. D URI="mongodb://localhost/?authMechanism=MONGODB-AWS" .. note:: No username, password or session token is passed into the URI. Drivers MUST check the environment variables listed above for these values. If the session token is set Drivers MUST use it. + + +Cached Credentials +================== + +Drivers MUST ensure that they are testing the ability to cache credentials. +Drivers will need to be able to query and override the cached credentials to +verify usage. To determine whether to run the cache tests, the driver can +check for the absence of the AWS_ACCESS_KEY_ID and of credentials in the URI. + +#. Clear the cache. +#. Create a new client. +#. Ensure that a ``find`` operation adds credentials to the cache.. +#. Override the cached credentials with an "Expiration" that is within one + minute of the current UTC time. +#. Create a new client. +#. Ensure that a ``find`` operation updates the credentials in the cache. +#. Poison the cache with garbage content. +#. Create a new client. +#. Ensure that a ``find`` operation results in an error. +#. Ensure that the cache has been cleared. +#. Ensure that a subsequent ``find`` operation succeeds. +#. Ensure that the cache has been set. From 72820569e80be0f401682e71da98f3e9be7739f6 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Wed, 27 Jul 2022 15:13:13 -0500 Subject: [PATCH 2/9] update changelog and version --- source/auth/auth.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index ebc2a728f7..686786eb59 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -5,8 +5,20 @@ Driver Authentication ===================== +<<<<<<< HEAD +======= +:Spec: 100 +:Spec Version: 1.12.0 +:Title: Driver Authentication +:Author: Craig Wilson, David Golden +:Advisors: Andy Schwerin, Bernie Hacket, Jeff Yemin, David Golden +>>>>>>> a1756248 (update changelog and version) :Status: Accepted :Minimum Server Version: 2.6 +<<<<<<< HEAD +======= +:Last Modified: 2022-07-27 +>>>>>>> a1756248 (update changelog and version) .. contents:: @@ -1342,6 +1354,7 @@ Q: Should drivers support accessing Amazon EC2 instance metadata in Amazon ECS? Changelog ========= +:2022-10-07: Require caching of AWS credentials fetched by the driver. :2022-10-05: Remove spec front matter and convert version history to changelog. :2022-09-07: Add support for AWS AssumeRoleWithWebIdentity. :2022-01-20: Require that timeouts be applied per the client-side operations timeout spec. From a73b5d3f2cedd3b1369a69c879d342dee35c3c29 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Wed, 10 Aug 2022 13:13:53 -0500 Subject: [PATCH 3/9] Add note about the five minute window --- source/auth/auth.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index 686786eb59..fe17fac57a 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1118,6 +1118,9 @@ used to determine when to clear the cache. If the "Expiration" is within 5 minutes of the current UTC time, the cache must be cleared. If AWS authentication fails for any reason, the cache must be cleared. +.. note:: + Five minutes was chosen because based on the AWS documentation for `IAM roles for EC2 `_ : "We make new credentials available at least five minutes before the expiration of the old credentials". The intent is to have some buffer between when the driver fetches the credentials and when the server verifies them. + ------------------------- Connection String Options ------------------------- From 00f1985f843ac09d57add19142ce8c2110d0e890 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Mon, 22 Aug 2022 17:12:00 -0500 Subject: [PATCH 4/9] address review --- source/auth/auth.rst | 21 +++++++++++++++++---- source/auth/tests/mongodb-aws.rst | 4 ++-- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index fe17fac57a..b8b018fb23 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1113,10 +1113,23 @@ MUST obtain the ``access_key``, ``secret_key`` and ``security_token`` which will Caching Credentials ___________________ Credentials fetched by the driver using AWS endpoints MUST be cached and reused -to avoid hitting rate limitations. The "Expiration" field must be stored and -used to determine when to clear the cache. If the "Expiration" is within 5 -minutes of the current UTC time, the cache must be cleared. -If AWS authentication fails for any reason, the cache must be cleared. +to avoid hitting AWS rate limitations. The "Expiration" field MUST be stored +and used to determine when to clear the cache. Credentials are considered +valid if they are more than one minute away from expiring, to the reduce the +chance of expiration before they are validated by the server. + +If there are no current valid cached credentials, the driver MUST initiate a +credential request. To avoid adding a bottleneck that would override the +``maxConnecting`` setting, the driver MUST not place a lock on making a +request. The cache MUST be written atomically. + +Once valid credentials are received, the driver SHOULD schedule an update of +the cache within five minutes of the expiration time, unless the driver does not have the ability to schedule background tasks. The background task MUST +handle timeouts or errors and retry appropriately. The purpose of the +background task is to limit the chance of needing to initiate multiple +simultaneous requests. + +If AWS authentication fails for any reason, the cache MUST be cleared. .. note:: Five minutes was chosen because based on the AWS documentation for `IAM roles for EC2 `_ : "We make new credentials available at least five minutes before the expiration of the old credentials". The intent is to have some buffer between when the driver fetches the credentials and when the server verifies them. diff --git a/source/auth/tests/mongodb-aws.rst b/source/auth/tests/mongodb-aws.rst index 9c770fab9f..8eaedef2a6 100644 --- a/source/auth/tests/mongodb-aws.rst +++ b/source/auth/tests/mongodb-aws.rst @@ -129,8 +129,8 @@ check for the absence of the AWS_ACCESS_KEY_ID and of credentials in the URI. #. Clear the cache. #. Create a new client. #. Ensure that a ``find`` operation adds credentials to the cache.. -#. Override the cached credentials with an "Expiration" that is within one - minute of the current UTC time. +#. Override the cached credentials with an "Expiration" that is within thirty + seconds of the current UTC time. #. Create a new client. #. Ensure that a ``find`` operation updates the credentials in the cache. #. Poison the cache with garbage content. From f5e7b71f7a4f9edaa4c020d049fba8ac5649d71f Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Wed, 31 Aug 2022 12:39:45 -0500 Subject: [PATCH 5/9] update language to encourage SDK use --- source/auth/auth.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index b8b018fb23..dec12e2f25 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1113,7 +1113,12 @@ MUST obtain the ``access_key``, ``secret_key`` and ``security_token`` which will Caching Credentials ___________________ Credentials fetched by the driver using AWS endpoints MUST be cached and reused -to avoid hitting AWS rate limitations. The "Expiration" field MUST be stored +to avoid hitting AWS rate limitations. AWS recommends using a suitable +Software Development Kit (SDK) for your langauge. If that SDK supports +credential fetch and automatic refresh/caching, then that mechanism can +be used in lieu of manual caching. + +If using manual caching, the "Expiration" field MUST be stored and used to determine when to clear the cache. Credentials are considered valid if they are more than one minute away from expiring, to the reduce the chance of expiration before they are validated by the server. From ce365eba50b98e39a4aaf62e92c9ffd937d6d518 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Fri, 7 Oct 2022 11:46:25 -0500 Subject: [PATCH 6/9] Update to use 5 minutes and no background task --- source/auth/auth.rst | 8 +------- source/auth/tests/mongodb-aws.rst | 6 +++--- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index dec12e2f25..f673ed9f67 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1120,7 +1120,7 @@ be used in lieu of manual caching. If using manual caching, the "Expiration" field MUST be stored and used to determine when to clear the cache. Credentials are considered -valid if they are more than one minute away from expiring, to the reduce the +valid if they are more than five minutes away from expiring, to the reduce the chance of expiration before they are validated by the server. If there are no current valid cached credentials, the driver MUST initiate a @@ -1128,12 +1128,6 @@ credential request. To avoid adding a bottleneck that would override the ``maxConnecting`` setting, the driver MUST not place a lock on making a request. The cache MUST be written atomically. -Once valid credentials are received, the driver SHOULD schedule an update of -the cache within five minutes of the expiration time, unless the driver does not have the ability to schedule background tasks. The background task MUST -handle timeouts or errors and retry appropriately. The purpose of the -background task is to limit the chance of needing to initiate multiple -simultaneous requests. - If AWS authentication fails for any reason, the cache MUST be cleared. .. note:: diff --git a/source/auth/tests/mongodb-aws.rst b/source/auth/tests/mongodb-aws.rst index 8eaedef2a6..4917509151 100644 --- a/source/auth/tests/mongodb-aws.rst +++ b/source/auth/tests/mongodb-aws.rst @@ -129,11 +129,11 @@ check for the absence of the AWS_ACCESS_KEY_ID and of credentials in the URI. #. Clear the cache. #. Create a new client. #. Ensure that a ``find`` operation adds credentials to the cache.. -#. Override the cached credentials with an "Expiration" that is within thirty - seconds of the current UTC time. +#. Override the cached credentials with an "Expiration" that is within one + minute of the current UTC time. #. Create a new client. #. Ensure that a ``find`` operation updates the credentials in the cache. -#. Poison the cache with garbage content. +#. Poison the cache with invalid auth content. #. Create a new client. #. Ensure that a ``find`` operation results in an error. #. Ensure that the cache has been cleared. From df74cbee0134ac21415205b04e819adbeee02e34 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Fri, 7 Oct 2022 13:37:59 -0500 Subject: [PATCH 7/9] address review --- source/auth/auth.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index f673ed9f67..c1d21ba0f9 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1113,20 +1113,20 @@ MUST obtain the ``access_key``, ``secret_key`` and ``security_token`` which will Caching Credentials ___________________ Credentials fetched by the driver using AWS endpoints MUST be cached and reused -to avoid hitting AWS rate limitations. AWS recommends using a suitable -Software Development Kit (SDK) for your langauge. If that SDK supports +to avoid hitting AWS rate limitations. AWS recommends using a suitable +Software Development Kit (SDK) for your langauge. If that SDK supports credential fetch and automatic refresh/caching, then that mechanism can be used in lieu of manual caching. If using manual caching, the "Expiration" field MUST be stored -and used to determine when to clear the cache. Credentials are considered +and used to determine when to clear the cache. Credentials are considered valid if they are more than five minutes away from expiring, to the reduce the chance of expiration before they are validated by the server. If there are no current valid cached credentials, the driver MUST initiate a -credential request. To avoid adding a bottleneck that would override the +credential request. To avoid adding a bottleneck that would override the ``maxConnecting`` setting, the driver MUST not place a lock on making a -request. The cache MUST be written atomically. +request. he cache MUST be written atomically. If AWS authentication fails for any reason, the cache MUST be cleared. From 8f0fb92a2fa852c9f33f844c6599d7f60effa316 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Mon, 10 Oct 2022 10:19:28 -0500 Subject: [PATCH 8/9] address review --- source/auth/auth.rst | 18 +++--------------- source/auth/tests/mongodb-aws.rst | 4 ++-- 2 files changed, 5 insertions(+), 17 deletions(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index c1d21ba0f9..966c918274 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -5,20 +5,8 @@ Driver Authentication ===================== -<<<<<<< HEAD -======= -:Spec: 100 -:Spec Version: 1.12.0 -:Title: Driver Authentication -:Author: Craig Wilson, David Golden -:Advisors: Andy Schwerin, Bernie Hacket, Jeff Yemin, David Golden ->>>>>>> a1756248 (update changelog and version) :Status: Accepted :Minimum Server Version: 2.6 -<<<<<<< HEAD -======= -:Last Modified: 2022-07-27 ->>>>>>> a1756248 (update changelog and version) .. contents:: @@ -1120,18 +1108,18 @@ be used in lieu of manual caching. If using manual caching, the "Expiration" field MUST be stored and used to determine when to clear the cache. Credentials are considered -valid if they are more than five minutes away from expiring, to the reduce the +valid if they are more than five minutes away from expiring; to the reduce the chance of expiration before they are validated by the server. If there are no current valid cached credentials, the driver MUST initiate a credential request. To avoid adding a bottleneck that would override the ``maxConnecting`` setting, the driver MUST not place a lock on making a -request. he cache MUST be written atomically. +request. The cache MUST be written atomically. If AWS authentication fails for any reason, the cache MUST be cleared. .. note:: - Five minutes was chosen because based on the AWS documentation for `IAM roles for EC2 `_ : "We make new credentials available at least five minutes before the expiration of the old credentials". The intent is to have some buffer between when the driver fetches the credentials and when the server verifies them. + Five minutes was chosen based on the AWS documentation for `IAM roles for EC2 `_ : "We make new credentials available at least five minutes before the expiration of the old credentials". The intent is to have some buffer between when the driver fetches the credentials and when the server verifies them. ------------------------- Connection String Options diff --git a/source/auth/tests/mongodb-aws.rst b/source/auth/tests/mongodb-aws.rst index 4917509151..a0a54f71bd 100644 --- a/source/auth/tests/mongodb-aws.rst +++ b/source/auth/tests/mongodb-aws.rst @@ -128,12 +128,12 @@ check for the absence of the AWS_ACCESS_KEY_ID and of credentials in the URI. #. Clear the cache. #. Create a new client. -#. Ensure that a ``find`` operation adds credentials to the cache.. +#. Ensure that a ``find`` operation adds credentials to the cache. #. Override the cached credentials with an "Expiration" that is within one minute of the current UTC time. #. Create a new client. #. Ensure that a ``find`` operation updates the credentials in the cache. -#. Poison the cache with invalid auth content. +#. Poison the cache with an invalid access key id. #. Create a new client. #. Ensure that a ``find`` operation results in an error. #. Ensure that the cache has been cleared. From 331cf49d73846c5a39e18d598eea4c0b0eaaabe2 Mon Sep 17 00:00:00 2001 From: Steven Silvester Date: Mon, 17 Oct 2022 15:55:59 -0500 Subject: [PATCH 9/9] fix double space --- source/auth/auth.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/auth/auth.rst b/source/auth/auth.rst index 966c918274..d411ecc316 100644 --- a/source/auth/auth.rst +++ b/source/auth/auth.rst @@ -1114,7 +1114,7 @@ chance of expiration before they are validated by the server. If there are no current valid cached credentials, the driver MUST initiate a credential request. To avoid adding a bottleneck that would override the ``maxConnecting`` setting, the driver MUST not place a lock on making a -request. The cache MUST be written atomically. +request. The cache MUST be written atomically. If AWS authentication fails for any reason, the cache MUST be cleared.