Skip to content

feat(parameters): add feature for creating and updating Parameters and Secrets #2858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 85 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
0369d80
gotta start somewhere
stephenbawks Jul 27, 2023
5551118
small tweaks
stephenbawks Aug 1, 2023
6e0c7de
adding set param
stephenbawks Sep 2, 2023
330de35
adding exception classes
stephenbawks Sep 2, 2023
929ae7f
adding set parameter
stephenbawks Sep 2, 2023
2ae911e
fixing set
stephenbawks Sep 2, 2023
aedab55
few more updates
stephenbawks Sep 11, 2023
1548690
Merge branch 'develop' into setparams
stephenbawks Sep 11, 2023
e154a5d
missing value
stephenbawks Sep 11, 2023
805f3d8
adding documentation
stephenbawks Sep 11, 2023
12b8850
one more update
stephenbawks Sep 11, 2023
f9323ea
question about transform yet
stephenbawks Sep 14, 2023
a0b32b8
remove optional transform
stephenbawks Sep 15, 2023
ab6065e
updates
stephenbawks Sep 16, 2023
6d2012c
updating put secret value
stephenbawks Sep 16, 2023
c6a1088
fixing value on return secret
stephenbawks Sep 16, 2023
609dbc9
cleaning up naming and examples
stephenbawks Sep 16, 2023
0abb31b
fix example and return type
stephenbawks Sep 16, 2023
05b8eed
Merge branch 'develop' into setparams
stephenbawks Sep 18, 2023
2992989
update
stephenbawks Sep 18, 2023
887f3c4
fix a few new warnings
stephenbawks Sep 18, 2023
98c7d5f
simplying secret value
stephenbawks Sep 19, 2023
d837154
small tweaks
stephenbawks Sep 21, 2023
13c9cf7
cleaning up
stephenbawks Sep 21, 2023
5b91705
missed one
stephenbawks Sep 21, 2023
7bbcaa7
Merge branch 'develop' into setparams
heitorlessa Sep 21, 2023
529c059
cleaning up ruff
stephenbawks Sep 22, 2023
20dac0c
Merge branch 'develop' into setparams
leandrodamascena Sep 25, 2023
0c03c82
couple of modifications
stephenbawks Sep 25, 2023
2d98d04
forgot to remove this here
stephenbawks Sep 25, 2023
7d406de
fix
stephenbawks Sep 25, 2023
be6f7af
adding create when not existing
stephenbawks Sep 28, 2023
8fffa58
fix name
stephenbawks Sep 28, 2023
b576bb7
create flag
stephenbawks Sep 28, 2023
4396b81
Update aws_lambda_powertools/utilities/parameters/secrets.py
stephenbawks Sep 28, 2023
78e9f45
couple refinements
stephenbawks Sep 28, 2023
668ba3d
updating docstrings
stephenbawks Sep 28, 2023
7413067
💄 fix example again
stephenbawks Sep 28, 2023
2539382
📝 documentation is seriously tough
stephenbawks Sep 28, 2023
682dd39
adding examples and documentation for set_param
stephenbawks Sep 28, 2023
d264b65
trying to add some docs
stephenbawks Sep 29, 2023
7a488bc
creating "realistic" example
stephenbawks Sep 30, 2023
3b2d67c
Merge branch 'develop' into setparams
leandrodamascena Oct 13, 2023
6915419
updating example
stephenbawks Oct 14, 2023
4920274
making example more appropiate
stephenbawks Oct 14, 2023
abce28e
missed one
stephenbawks Oct 14, 2023
20925ec
Merge branch 'develop' into setparams
leandrodamascena Oct 15, 2023
d49f89f
couple tests so far
stephenbawks Oct 19, 2023
bee00a4
Merge branch 'develop' into setparams
sthulb Dec 6, 2023
973b3a9
Merge branch 'develop' into setparams
leandrodamascena Jan 10, 2024
3c05392
Merge branch 'develop' into setparams
heitorlessa Feb 15, 2024
90fa67a
Merge branch 'develop' into setparams
stephenbawks Feb 17, 2024
f9a2944
changing variable name
stephenbawks Feb 17, 2024
d55a9b4
removing the create logic for the time being
stephenbawks Feb 17, 2024
9ead47a
remove create option
stephenbawks Feb 19, 2024
8a52d21
Merge branch 'develop' into setparams
stephenbawks Feb 19, 2024
647e3b8
Merge branch 'develop' into setparams
heitorlessa Feb 26, 2024
7783aad
chore: remove set as mandatory method (temporarily)
heitorlessa Feb 26, 2024
7325bc7
fix(parameters): make cache aware of single vs multiple calls
heitorlessa Jul 25, 2023
8704337
chore: cleanup, add test for single and nested
heitorlessa Jul 25, 2023
5098ed9
Merge branch 'develop' into setparams
heitorlessa Feb 27, 2024
cd116bb
refactor: implement set() minimum contract, and set() in ssm
heitorlessa Feb 27, 2024
5ccb953
refactor: use name over path in set_parameter
heitorlessa Feb 27, 2024
e28b548
Merge remote-tracking branch 'upstream/develop' into setparams
leandrodamascena Mar 20, 2024
d9cf5fd
Adding docstring
leandrodamascena Mar 20, 2024
ae8975f
Fixing description type + adding TypeDict for set_parameter
leandrodamascena Mar 20, 2024
6502229
Refactoring tests
leandrodamascena Mar 20, 2024
3e79eac
Adding correct exception for Secrets + fixing examples
leandrodamascena Mar 20, 2024
401cdf8
Making SonarCloud happy
leandrodamascena Mar 20, 2024
44710ba
Changing return from setSecret + fix mypy issues
leandrodamascena Mar 20, 2024
31777b1
Increasing coverage
leandrodamascena Mar 20, 2024
b89c9b8
Merge branch 'develop' into setparams
leandrodamascena Mar 20, 2024
c016d24
Increasing coverage
leandrodamascena Mar 20, 2024
3bf45df
Refactoring secrets
leandrodamascena Mar 20, 2024
57eb44b
Improving docstring
leandrodamascena Mar 21, 2024
df0ffc9
Adding more tests
leandrodamascena Mar 21, 2024
0efe6d7
Making SonarCloud happy
leandrodamascena Mar 21, 2024
6fae9dd
Adding more tests and docs
leandrodamascena Mar 21, 2024
4b0af25
Merge branch 'develop' into setparams
leandrodamascena Mar 21, 2024
f65570f
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
bf5f05b
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
4cebf75
Addressing Ruben's feedback
leandrodamascena Mar 21, 2024
273c544
Addressing Ruben's feedback
leandrodamascena Mar 22, 2024
a9a3c41
Merge branch 'develop' into setparams
leandrodamascena Mar 22, 2024
ac78c0d
Merge branch 'develop' into setparams
leandrodamascena Mar 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions aws_lambda_powertools/utilities/parameters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
from .base import BaseProvider, clear_caches
from .dynamodb import DynamoDBProvider
from .exceptions import GetParameterError, TransformParameterError
from .secrets import SecretsProvider, get_secret
from .ssm import SSMProvider, get_parameter, get_parameters, get_parameters_by_name
from .secrets import SecretsProvider, get_secret, set_secret
from .ssm import SSMProvider, get_parameter, get_parameters, get_parameters_by_name, set_parameter

__all__ = [
"AppConfigProvider",
Expand All @@ -21,8 +21,10 @@
"TransformParameterError",
"get_app_config",
"get_parameter",
"set_parameter",
"get_parameters",
"get_parameters_by_name",
"get_secret",
"set_secret",
"clear_caches",
]
3 changes: 3 additions & 0 deletions aws_lambda_powertools/utilities/parameters/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,9 @@ def _get(self, name: str, **sdk_options) -> Union[str, bytes, Dict[str, Any]]:
"""
raise NotImplementedError()

def set(self, name: str, value: Any, *, overwrite: bool = False, **kwargs):
raise NotImplementedError()

def get_multiple(
self,
path: str,
Expand Down
8 changes: 8 additions & 0 deletions aws_lambda_powertools/utilities/parameters/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,11 @@ class GetParameterError(Exception):

class TransformParameterError(Exception):
"""When a provider fails to transform a parameter value"""


class SetParameterError(Exception):
"""When a provider raises an exception on setting a SSM parameter"""


class SetSecretError(Exception):
"""When a provider raises an exception on setting an AWS Secret"""
224 changes: 222 additions & 2 deletions aws_lambda_powertools/utilities/parameters/secrets.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,27 @@
AWS Secrets Manager parameter retrieval and caching utility
"""

from __future__ import annotations

import json
import logging
import os
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Union, overload

import boto3
from botocore.config import Config

from aws_lambda_powertools.utilities.parameters.types import TransformOptions
from aws_lambda_powertools.utilities.parameters.types import SetSecretResponse, TransformOptions

if TYPE_CHECKING:
from mypy_boto3_secretsmanager import SecretsManagerClient

from aws_lambda_powertools.shared import constants
from aws_lambda_powertools.shared.functions import resolve_max_age
from aws_lambda_powertools.utilities.parameters.base import DEFAULT_MAX_AGE_SECS, DEFAULT_PROVIDERS, BaseProvider
from aws_lambda_powertools.utilities.parameters.exceptions import SetSecretError

from .base import DEFAULT_MAX_AGE_SECS, DEFAULT_PROVIDERS, BaseProvider
logger = logging.getLogger(__name__)


class SecretsProvider(BaseProvider):
Expand Down Expand Up @@ -117,6 +123,135 @@ def _get_multiple(self, path: str, **sdk_options) -> Dict[str, str]:
"""
raise NotImplementedError()

def _create_secret(self, name: str, **sdk_options):
"""
Create a secret with the given name.

Parameters:
----------
name: str
The name of the secret.
**sdk_options:
Additional options to be passed to the create_secret method.

Raises:
SetSecretError: If there is an error setting the secret.
"""
try:
sdk_options["Name"] = name
return self.client.create_secret(**sdk_options)
except Exception as exc:
raise SetSecretError(f"Error setting secret - {str(exc)}") from exc

def _update_secret(self, name: str, **sdk_options):
"""
Update a secret with the given name.

Parameters:
----------
name: str
The name of the secret.
**sdk_options:
Additional options to be passed to the create_secret method.
"""
sdk_options["SecretId"] = name
return self.client.put_secret_value(**sdk_options)

def set(
self,
name: str,
value: Union[str, dict, bytes],
*, # force keyword arguments
client_request_token: Optional[str] = None,
**sdk_options,
) -> SetSecretResponse:
"""
Modify the details of a secret or create a new secret if it doesn't already exist.
It includes metadata and the secret value.

We aim to minimize API calls by assuming that the secret already exists and needs updating.
If it doesn't exist, we attempt to create a new one. Refer to the following workflow for a better understanding:


┌────────────────────────┐ ┌─────────────────┐
┌───────▶│Resource NotFound error?│────▶│Create Secret API│─────┐
│ └────────────────────────┘ └─────────────────┘ │
│ │
│ │
│ ▼
┌─────────────────┐ ┌─────────────────────┐
│Update Secret API│────────────────────────────────────────────▶│ Return or Exception │
└─────────────────┘ └─────────────────────┘

Parameters
----------
name: str
The ARN or name of the secret to add a new version to or create a new one.
value: str, dict or bytes
Specifies text data that you want to encrypt and store in this new version of the secret.
client_request_token: str, optional
This value helps ensure idempotency. Recommended that you generate
a UUID-type value to ensure uniqueness within the specified secret.
This value becomes the VersionId of the new version. This field is
auto-populated if not provided.
sdk_options: dict, optional
Dictionary of options that will be passed to the Secrets Manager update_secret API call

Raises
------
SetSecretError
When attempting to update or create a secret fails.

Returns:
-------
SetSecretResponse:
The dict returned by boto3.

Example
-------
**Sets a secret***

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> parameters.set_secret(name="llamas-are-awesome", value="supers3cr3tllam@passw0rd")

**Sets a secret and includes an client_request_token**

>>> from aws_lambda_powertools.utilities import parameters
>>> import uuid
>>>
>>> parameters.set_secret(
name="my-secret",
value='{"password": "supers3cr3tllam@passw0rd"}',
client_request_token=str(uuid.uuid4())
)

URLs:
-------
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/put_secret_value.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/create_secret.html
"""

if isinstance(value, dict):
value = json.dumps(value)

if isinstance(value, bytes):
sdk_options["SecretBinary"] = value
else:
sdk_options["SecretString"] = value

if client_request_token:
sdk_options["ClientRequestToken"] = client_request_token

try:
logger.debug(f"Attempting to update secret {name}")
return self._update_secret(name=name, **sdk_options)
except self.client.exceptions.ResourceNotFoundException:
logger.debug(f"Secret {name} doesn't exist, creating a new one")
return self._create_secret(name=name, **sdk_options)
except Exception as exc:
raise SetSecretError(f"Error setting secret - {str(exc)}") from exc


@overload
def get_secret(
Expand Down Expand Up @@ -224,3 +359,88 @@ def get_secret(
force_fetch=force_fetch,
**sdk_options,
)


def set_secret(
name: str,
value: Union[str, bytes],
*, # force keyword arguments
client_request_token: Optional[str] = None,
**sdk_options,
) -> str:
"""
Modify the details of a secret or create a new secret if it doesn't already exist.
It includes metadata and the secret value.

We aim to minimize API calls by assuming that the secret already exists and needs updating.
If it doesn't exist, we attempt to create a new one. Refer to the following workflow for a better understanding:


┌────────────────────────┐ ┌─────────────────┐
┌───────▶│Resource NotFound error?│────▶│Create Secret API│─────┐
│ └────────────────────────┘ └─────────────────┘ │
│ │
│ │
│ ▼
┌─────────────────┐ ┌─────────────────────┐
│Update Secret API│────────────────────────────────────────────▶│ Return or Exception │
└─────────────────┘ └─────────────────────┘

Parameters
----------
name: str
The ARN or name of the secret to add a new version to or create a new one.
value: str, dict or bytes
Specifies text data that you want to encrypt and store in this new version of the secret.
client_request_token: str, optional
This value helps ensure idempotency. Recommended that you generate
a UUID-type value to ensure uniqueness within the specified secret.
This value becomes the VersionId of the new version. This field is
auto-populated if not provided.
sdk_options: dict, optional
Dictionary of options that will be passed to the Secrets Manager update_secret API call

Raises
------
SetSecretError
When attempting to update or create a secret fails.

Returns:
-------
SetSecretResponse:
The dict returned by boto3.

Example
-------
**Sets a secret***

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> parameters.set_secret(name="llamas-are-awesome", value="supers3cr3tllam@passw0rd")

**Sets a secret and includes an client_request_token**

>>> from aws_lambda_powertools.utilities import parameters
>>>
>>> parameters.set_secret(
name="my-secret",
value='{"password": "supers3cr3tllam@passw0rd"}',
client_request_token="61f2af5f-5f75-44b1-a29f-0cc37af55b11"
)

URLs:
-------
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/put_secret_value.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager/client/create_secret.html
"""

# Only create the provider if this function is called at least once
if "secrets" not in DEFAULT_PROVIDERS:
DEFAULT_PROVIDERS["secrets"] = SecretsProvider()

return DEFAULT_PROVIDERS["secrets"].set(
name=name,
value=value,
client_request_token=client_request_token,
**sdk_options,
)
Loading