Getting CredentialsError: Missing credentials in config starting in v2.44.0

[matsaleh13]

I use the aws-sdk from a little node.js CLI tool that I wrote for running some EC2 tasks. This has been working for several weeks with no problem. Then, I recently upgraded the version of the aws-sdk from 2.9.0 to 2.45.0. After that I started getting the error whenever my tool made a call to the aws-sdk.:

CredentialsError: CredentialsError: Missing credentials in config
    at ClientRequest.<anonymous> (C:\Dev\GitHub\vcme\cq-loadtest\node_modules\aws-sdk\lib\http\node.js:83:34)
    at ClientRequest.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:185:7)
    at Socket.emitTimeout (_http_client.js:620:10)
    at Socket.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:339:8)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)

I downgraded my aws-sdk version to 2.44.0, and the error still occurs in that version.
I downgraded again to 2.43.0, and the error no longer occurs.

So, it seems that whatever is causing this to happen is a change introduced in [email protected].

Here is an example of the code in my tool where the problem occurs:

    // excerpt
    const AWS = require('aws-sdk/global');
    const ECS = require('aws-sdk/clients/ecs');

    const ecs = new ECS();
    const params = {
        cluster: 'cluster-mycluster-XXXXXXXXXXX',
        desiredStatus = 'RUNNING'
    };

    ecs.listTasks(params, (err, taskListData) => {
        if (err) return done(err);   // callback
        // do something with taskListData
        return done(null);  // callback 
    }

I checked out he CHANGELOG for 2.44.0, and I did see a mention of a change related to the AWS_SHARED_CREDENTIALS_FILE. I'm not sure that's directly relevant though.

I can say that this tool always runs using the default credentials specified via the awscli, i.e. set via aws configure.

Not sure at this point whether this is an actual bug in aws-sdk, or in my code that was somehow revealed by the change to aws-sdk.

I'm a n00b to aws-sdk, so any suggestions or guidance is welcome.

Thanks and cheers, Matt

[chrisradek]

@matsaleh13
v2.44.0 of the SDK did introduce the ability to source credentials from the ~/.aws/config file, where previously it was only able to be sourced from ~/.aws/credentials. However, that should only happen if AWS_SDK_LOAD_CONFIG is set to a truthy value.

Can you verify in your node.js script if AWS_SDK_LOAD_CONFIG is set? You can do that by checking process.env.AWS_SDK_LOAD_CONFIG from within your script.

The error object you are receiving should also contain some special fields (I believe originalError is one). Can you JSON.stringify the error and share what that looks like as well? That may tell us what profile/file was attempted to be read.

[matsaleh13]

Thanks for the fast reply @chrisradek!

I added console output for the two bits of information. (FWIW I used util.inspect() instead of JSON.stringify() because IMO it's easier to read).

AWS_SDK_LOAD_CONFIG is not set.

Here's the top-level originalError property, with another one nested:

  originalError:
   { message: 'Could not load credentials from any providers',
     code: 'CredentialsError',
     time: 2017-04-25T21:05:42.013Z,
     retryable: true,
     originalError:
      { message: 'Connection timed out after 1000ms',
        code: 'TimeoutError',
        time: 2017-04-25T21:05:42.013Z,
        retryable: true } } }

Here's the full output for completeness:

λ node aws\loadtestctl.js status
Getting status of LoadTester ECS tasks.
AWS_SDK_LOAD_CONFIG = undefined
Error from listTasks(): { CredentialsError: Missing credentials in config
    at ClientRequest.<anonymous> (C:\Dev\GitHub\vcme\cq-loadtest\node_modules\aws-sdk\lib\http\node.js:83:34)
    at ClientRequest.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:185:7)
    at Socket.emitTimeout (_http_client.js:620:10)
    at Socket.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:339:8)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
  message: 'Missing credentials in config',
  code: 'CredentialsError',
  time: 2017-04-25T21:05:42.014Z,
  retryable: true,
  originalError:
   { message: 'Could not load credentials from any providers',
     code: 'CredentialsError',
     time: 2017-04-25T21:05:42.013Z,
     retryable: true,
     originalError:
      { message: 'Connection timed out after 1000ms',
        code: 'TimeoutError',
        time: 2017-04-25T21:05:42.013Z,
        retryable: true } } }

Hope this helps, and thanks!

[chrisradek]

@matsaleh13
Can you also verify what credentials the SDK is using?

If you log AWS.config.credentials.constructor, you should get the name of the credential provider used (e.g. SharedIniFileCredentials).
In the case that you are using SharedIniFileCredentials, can you also check the value of profile, filename, and disableAssumeRole?

[matsaleh13]

OK, tried the above, but logging AWS.config.credentials was null.

Attempted to log what appears to be related properties:

AWS.config.credentials: null
AWS.config.credentialProvider: CredentialProviderChain {
  providers: [ [Function], [Function], [Function], [Function] ] }  // via util.inspect()
AWS.config.credentialProvider (contents): {"providers":[null,null,null,null]}  // via JSON.stringify()

Seems like it can't load from any provider because it has none.

Other info:

• FYI as I mentioned in my OP, my creds are configured via aws configure, and I've verified they are valid.
• Other node.js tools that use aws-sdk do still work (e.g. cwtail. Note: my version of cwtail uses [email protected])
• I can switch back and forth between [email protected] and [email protected] repeatedly and recreate the working (2.43) and non-working (2.44) cases 100% of the time.

Hope this helps.

[chrisradek]

Can you share what your credentials look like in the version of the SDK that does work?

aws configure should add credentials to the ~/.aws/credentials file, and the node.js SDK should attempt to pull credentials from there by default.

AWS.config.credentialProvider does have 4 providers specified based on your util.inspect() output. JSON.stringify won't print functions, which is why they show up as null in your log.

What I'm trying to get at, is what credentials the SDK is loading when using v2.43.0. It sounds like it should be reading a profile from the credentials file. I'm curious if you have a matching profile in ~/.aws/config. Without sharing your actual keys, could you share how your profile (based on v2.43.0 of the SDK) is set up, and if the same profile also exists in the ~/.aws/config file?

[matsaleh13]

I just realized I neglected to mention that I'm running all this on Windows 10. Sorry for that.

Also, I have a HOME env var set that maps to C:\Users\Matthew Walker\OneDrive\Home (so that I can share config stuff between systems).

From git bash, the $HOME folder resolves to /c/Users/Matthew Walker/OneDrive/Home (as expected).

Also, from git bash, the ~ resolves to the same path as $HOME (as expected).

Now, to answer your questions above:

Logging AWS.config.credentials.constructor: when using [email protected] shows I am indeed using SharedIniFileCredentials:

AWS.config.credentials.constructor: function SharedIniFileCredentials(options) {
    AWS.Credentials.call(this);

    options = options || {};

    this.filename = options.filename;
    this.profile = options.profile || process.env.AWS_PROFILE || 'default';
    this.disableAssumeRole = !!options.disableAssumeRole;
    this.get(function() {});
  }

In my $HOME folder I do have an .aws folder with contents (via git bash):

[Matthew Walker Orig@pearl:~]
$ ls -la ~/.aws
total 10K
drwxr-xr-x 1 Matthew Walker Orig 197609   0 Nov 30 11:13 ./
drwxr-xr-x 1 Matthew Walker Orig 197609   0 Apr  9 11:44 ../
-rw-r--r-- 1 Matthew Walker Orig 197609  46 Nov 30 11:16 config
-rw-r--r-- 1 Matthew Walker Orig 197609 119 Nov 30 11:13 credentials

The contents of config are:

[default]
region = us-east-1
output = text

The contents of credentials are:

[default]
aws_access_key_id = <MY_ACCESS_KEY_ID>
aws_secret_access_key = <MY_SECRET_ACCESS_KEY>

[chrisradek]

@matsaleh13
Ah ok, are you also using the CLI through git bash? Have you also tried using bash on windows by chance?

I have a Windows 10 machine I can test with, that at least gives me something to look into! I should have asked before, which version of node.js are you using as well?

Thanks for your patience and detailed notes so far!

[matsaleh13]

No, I'm not using the aws-cli through git-bash, and I don't run my node.js tool from there either. I mainly use git-bash to get better UX for navigating the file system and to access the necessary unixy tools ;).

I have only tried Win10 bash (bash on Ubuntu on Windows) as an experiment. I don't use it for anything useful because last I heard you can't exec windows programs using it, and all my work is still Windows-based.

Node:

λ node -v
v6.9.1

aws-cli (in case...):

λ aws --version
aws-cli/1.11.80 Python/2.7.9 Windows/8 botocore/1.5.43

Thank you for your prompt responses and guidance! Happy to team with you on this.

Sorry I have such an awkward, hybridy environment! Um.. okay actually I'm not. I take pride in my ability to generate edge cases with simply a blink. :)

Cheers!

[clark-pan]

@matsaleh13 Dug through source code and figured out the issue.

Based on your configuration, you need to set AWS_SDK_LOAD_CONFIG=1, AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials and AWS_CONFIG_FILE=~/.aws/config.

Prior to 2.44.0, the ~/.aws/credentials value was hardcoded into the SDK. #1391 added support for ~/.aws/config but also removed the hardcoding of these two values.

This isn't an edge case tbh, anybody who uses 2.44.0 will be hitting this issue. You are just early to the party :)

[jeskew]

Hi @clark-pan,

The default file path for ~/.aws/credentials is still used, as is the default config file path of ~/.aws/config.

@matsaleh13 As of 2.44.0, the SDK is using os.homedir if the function is available, which will not respect the HOME environment variable when running on git bash but will instead use the USERPROFILE environment variable:

user@computer MINGW64 ~
$ HOME='/foo/bar' node
> const os = require('os')
undefined
> os.homedir()
'C:\\Users\\user'

os.homedir is still able to locate a home directory when the system's standard environment variables have not been set, but giving it higher precedence than $HOME looks like a regression for Windows users.

(@chrisradek This would not affect users running node on the Windows subsystem for Linux, as they would be running a copy of node compiled on Ubuntu that uses the *nix implementation of uv_os_homedir.)

[matsaleh13]

@clark-pan and @jeskew thanks for digging into this.

I set the environment vars mentioned above in the windows shell using:

set AWS_SDK_LOAD_CONFIG=1
set AWS_SHARED_CREDENTIALS_FILE=%HOME%/.aws/credentials
set AWS_CONFIG_FILE=%HOME%/.aws/config

(Apologies to all for the bastardized blending of OS shell conventions.)

After that, my node.js program worked; the CredentialError no longer occurs!

So, I guess the current ~/.aws/ folder and its contents were initialized via aws configure using an older version of the aws-sdk (around Nov 2016). That would have been the hard-coded implementation I guess, which would have resolved fine because of my nonstandard use of $HOME on windows.

However, because of that nonstandard setup, the change in v2.44.0 to use os.homedir broke my runtime. IMO it's very reasonable to trust os.homedir to do "the right thing" on any platform. And, it did, since I'm really running this code on windows in a standard windows shell $HOME env var notwithstanding).

I guess deciding how to support $HOME on windows is kind of a crap shoot. Although, it might be reasonable to assume that, if present, $HOME would have been added explicitly and should take precedence. I'd bet that whichever way you go will cause problems for someone though.

Might be useful to present the user a message from aws configure about the nonstandard presence of $HOME on Windows and ask what they want to do, but that doesn't help after the fact, of course.

Also might be good to add a note about it to the docs, but they're so deep already that it's likely to be missed by the reader. :(

FWIW, while troubleshooting this I did reinstall awscli to aws-cli/1.11.80 Python/2.7.9 Windows/8 botocore/1.5.43, and re-ran aws configure, and that didn't change anything. Perhaps that version of the CLI doesn't yet use [email protected]? (edit: of course not, it's not a node.js app, doh!)

At any rate, thanks for the info and guidance in this!

Cheers.

[jeskew]

@matsaleh13 The CLI uses boto (the AWS SDK for Python) under the hood, and it looks like Python's os.path.expanduser will use HOME over USERPROFILE on Windows. Since the change in 2.44.0 was meant to increase compatibility with the CLI, surfacing this runtime difference between Node and Python isn't a great user experience.

#1484 changes the precedence of how home directories are located so that HOME, USERPROFILE, HOMEPATH, and HOMEDRIVE will be checked before os.homedir is called.

[JarLowrey]

For me the issue was I put .aws/credentials in my project directory when it should have gone in ~/.aws/credentials. The error shows the full path of where the credentials are expected. Commenting here for people googling a similar question.

[mscottx88]

For those unfortunate souls that might find themselves in this same predicament, here is what I discovered.

I had my credentials in the right spot ~./aws/credentials.
I also had a profile [default] with my keys.
But I also had an environment variable AWS_PROFILE set to my actual profile.

Turns out, the SharedIniFileCredentials loader will pick the stanza from the credentials file whose name matches the AWS_PROFILE environment varialble, if present.

So, if your AWS_PROFILE is elite.programmer but your credentials file only has a [default] stanza, copy the [default] stanza and rename it to elite.programmer.

# provided by aws configure
[default]
aws_access_key_id = <secret>
aws_secret_access_key = <secret>

# will be chosen through AWS_PROFILE environment variable
[elite.programmer]
aws_access_key_id = <secret>
aws_secret_access_key = <secret>

[max77p]

@mscottx88 when i deploy to beanstalk (my react node js app) login works as intended, but after a few hours I get "missing credentials in config" any idea why that may be?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.