CredentialsProviderError: Could not load credentials from any providers intermittently

[danielblignaut]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Hi,

we are receiving the following error: CredentialsProviderError: Could not load credentials from any providers intermittently from our Fargate ECS task that is trying to connect to DynamoDB. We do have regular healthchecks that create new instances of the DynamoDB client and it seems reasonable to assume that this could be related to #4867

It seems reasonable to assume that the SDK should manage credentials in the background, including caching them for subsequent client instantiations, etc.

Are there plans to add support for this?

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

deploy an ECS tasks the creates multiple AWS clients in succession

Observed Behavior

we are receiving the following error: CredentialsProviderError: Could not load credentials from any providers intermittently

Expected Behavior

The AWS client should instantiate itself successfully

Possible Solution

The AWS SDK should take care to cache credentials

Additional Information/Context

No response

[kuhe]

Please add a code example of how you are initializing the client and what environment variables are being passed in.

For example:

process.env.??? = ???

new DynamoDB({
  credentials: ???
});

[danielblignaut]

We were originally following:

this._client = new DynamoDBClient({
      endpoint: opts.endpoint,
    });

after reading https://repost.aws/questions/QUXMX9z9zCSb21qsnedFZQpg/having-trouble-converting-aws-node-sdk-v2-lambda-to-v3

we now use:

import { defaultProvider } from '@aws-sdk/credential-provider-node';

this.client = new DynamoDBClient({
      endpoint: opts.endpoint,
      credentials: defaultProvider(),
    });

within our ECS task definition, in both cases, the only relevant environment variable we have set is:

AWS_REGION=us-east-1

We have the correct task & execution IAM roles set. I'm unfamiliar if AWS injects additional runtime env vars to help the credentials provider, I haven't logged and included those here for security.

On a per request basis, we do instantiate multiple AWS clients. My understanding was the the credentials config was internally a singleton like object so the credentials would be shared per client instantiation. Currently I wonder if this is not the case? In which case is it advised for us to follow a singleton pattern (one global client) to be used across requests? Are there performance drawbacks from this at all?

Finally, not sure if this is relevant but we are bunding our source code with esbuild:

npx esbuild src/server.ts --bundle --platform=node --outdir=./dist --sourcemap

I'm unsure if something during the bundling process is causing an issue?

As mentioned, the issue is intermittent so I can deploy the service and get 200 responses the majority of the time but if I start running k6 load tests we will start to see some errors.

[RanVaknin]

Hi @danielblignaut ,

Credential resolution is happening per SDK call, and not on client creation.
When the API call fires, the SDK will try to construct the request, it will check the cache that is set on that defaultProvider, if the cache is empty, it will attempt to retrieve new credentials by going through all of the credential providers on the chain until it finds a valid source of credentials (or errors out).

When you deploy your application on ECS, and you use the default provider, the SDK will look through the entire provider list until it reaches the very last one (fromContainerMetadata) which in your case should resolve, if ECS injected the correct env variables, the provider will use those to fetch info from the container metadata service and exchange that info for valid credentials.

The env variable the SDK's provider expects is ENV_CMDS_AUTH_TOKEN AND either AWS_CONTAINER_CREDENTIALS_FULL_URI OR AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

On a per request basis, we do instantiate multiple AWS clients.

Our recommendation is to construct the SDK client once per service, per region. And not per SDK call. That would explain why your credentials are never cached.

Without seeing a more complete code snippet it's hard to give advise on what went wrong.

Thanks,
Ran~

[danielblignaut]

Hi Ran, I can confirm this is the case and since changing the code and running load tests we’ve not seen this error re-occur. Thank you for the feedback

…

On Fri, 1 Mar 2024 at 19:02, Ran Vaknin ***@***.***> wrote: Hi @danielblignaut <https://github.com/danielblignaut> , Credential resolution is happening per SDK call, and not on client creation. When the API call fires, the SDK will try to construct the request, it will check the cache that is set on that defaultProvider, if the cache is empty, it will attempt to retrieve new credentials by going through all of the credential providers on the chain until it finds a valid source of credentials (or errors out). When you deploy your application on ECS, and you use the default provider, the SDK will look through the entire provider list until it reaches the very last one (fromContainerMetadata) which in your case should resolve, if ECS injected the correct env variables, the provider will use those to fetch info from the container metadata service and exchange that info for valid credentials. The env variable the SDK's provider expects is ENV_CMDS_AUTH_TOKEN AND either AWS_CONTAINER_CREDENTIALS_FULL_URI OR AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. On a per request basis, we do instantiate multiple AWS clients. Our recommendation is to construct the SDK client once per service, per region. And not per SDK call. That would explain why your credentials are never cached. Without seeing a more complete code snippet it's hard to give advise on what went wrong. Thanks, Ran~ — Reply to this email directly, view it on GitHub <#5829 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIJNSJMQ6444WYXCRBQ7TTYWDGFJAVCNFSM6AAAAABD4AXU5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTG42TSNBZGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RanVaknin]

Happy to hear that this unblocked you.

All the best,
Ran~

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.