We've added this to our backlog, and we intend to work on this early next year.

Hey @leandrodamascena, whenever you have time this week, could you please leave a comment with some more details about what needs to be done / implemented as part of this issue?

This will help potential contributors to orient themselves.

Hey everyone! Our current implementation supports dict and lists as they can be directly converted. However, we recognize that customers may want to use more complex data structures such as Pydantic Models, DataClasses, or custom Python classes.

To accommodate these cases, we need to implement a pre-processing step to prepare the data before submitting it for erase, encrypt, or decrypt. This solution addresses the input data challenge, but it's important to note that we cannot guarantee the data will remain a valid instance of the original Pydantic model after processing, for example.

The encryption/erase process may alter the data type in ways that break the model's validation rules. For example:

from __future__ import annotations

from aws_lambda_powertools.utilities.data_masking import DataMasking

from pydantic import BaseModel

class MyModel(BaseModel):
    name: str
    age: int


data_masker = DataMasking()

data = MyModel(name="powertools", age=5)

erased = data_masker.erase(data, fields=["age"])  

print(erased)
# output: {'name': 'powertools', 'age': '*****'}

Not that age is now a string and no longer an int, breaking Pydantic's validation model.

For now, we can implement this support only when we input the data, not in the output. To do this, I suggest to create a new function called prepare_data in this file and in this function it check the data type and convert it to dict, this could be something like this:

def prepare_data(data: Any) -> Any:
    # Convert from dataclasses
    if hasattr(data, "__dataclass_fields__"):
        import dataclasses

        return dataclasses.asdict(data)

    # Convert from Pydantic model
    if callable(getattr(data, "model_dump", None)):
        return data.model_dump()
    
    # Convert from event source data class
    if callable(getattr(data, "dict", None)):
        return data.dict()

    return data

After that, call this method in the first line of erase, encrypt and decrypt methods.

Also, need to add more tests here https://github.com/aws-powertools/powertools-lambda-python/tree/develop/tests/functional/data_masking to make sure it working as expected.

@leandrodamascena, I see this a pretty old issue, I can pick this up

Sure, go ahead @VatsalGoel3.

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.