Problem trying to refresh assume-role credentials automatically with KCL on EC2

[jazzTheJackRabbit]

I have a Kinesis Stream Reader setup on an EC2 instance with the an instance profile with assume-role permissions to assume a reader-role to a Kinesis Stream in a separate AWS account. I can access the stream perfectly fine if I configure my credentials by using the STS service through the AWS-CLI, retrieve temporary credentials and setup the ~/.aws/credentials file accordingly. But, I'm running into problems if I try to use the STSAssumeRoleSessionCredentialsProvider in kcl.properties to automatically refresh the temporary credentials.

The credentials in kcl.properties is as follows:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account-id>:role/<role-name>|session-name

But I seem to be running into the following issue.

No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

I'm using the Amazon KCL Version : 1.4.4 with the following JARS:

• amazon-kinesis-client-1.7.5.jar
• aws-java-sdk-cloudwatch-1.11.115.jar
• aws-java-sdk-core-1.11.115.jar
• aws-java-sdk-dynamodb-1.11.115.jar
• aws-java-sdk-kinesis-1.11.115.jar
• aws-java-sdk-kms-1.11.115.jar
• aws-java-sdk-s3-1.11.115.jar
• commons-codec-1.9.jar
• commons-lang-2.6.jar
• commons-logging-1.1.3.jar
• guava-18.0.jar
• httpclient-4.5.2.jar
• httpcore-4.4.4.jar
• ion-java-1.0.2.jar
• jackson-annotations-2.6.0.jar
• jackson-core-2.6.6.jar
• jackson-databind-2.6.6.jar
• jackson-dataformat-cbor-2.6.6.jar
• jmespath-java-1.11.115.jar
• joda-time-2.8.1.jar
• protobuf-java-2.6.1.jar

Any help with respect to missing JARs or configuration specifics would be helpful.

[antgus]

Hi - I'm also trying to assume a role and so far have been unable to. I get the same message as you if I use the same format for AWSCredentialsProvider. I'm curious to know where you found the format in the first place? I haven't been able to find any kind of documentation for it.

Edit: Found the source code: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoder.java#L72

[pfifer]

The STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar, so you would need to add the aws-java-sdk-sts jar to the class path.

If you don't mind using Apache Maven this is a pom.xml that will create a launch script that will setup the class path for you.

To create the launcher script:

1. Create an empty directory
2. Create a new file in the directory called pom.xml
3. Install Apache Maven if you don't already have it
4. In the directory created in 1, run the command mvn package
5. If everything works you should have a script in target/appassembler/bin/multilang-daemon, and all the jars are in target/appassembler/repo.
6. Copy the directory target/appassembler to wherever you need it.
7. To start your application run the mutlilang-daemon script with the location of your property file.

[ykhrustalev]

@pfifer , after following your suggestion and building multilang-daemo I seem still have an issue

./bin/multilang-daemon config/kcl.local.properties
/usr/bin/java -classpath /home/worker/app/etc:/home/worker/app/repo/com/amazonaws/amazon-kinesis-client/1.7.5/amazon-kinesis-client-1.7.5.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-dynamodb/1.11.115/aws-java-sdk-dynamodb-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-s3/1.11.115/aws-java-sdk-s3-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kms/1.11.115/aws-java-sdk-kms-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kinesis/1.11.115/aws-java-sdk-kinesis-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-cloudwatch/1.11.115/aws-java-sdk-cloudwatch-1.11.115.jar:/home/worker/app/repo/com/google/guava/guava/18.0/guava-18.0.jar:/home/worker/app/repo/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar:/home/worker/app/repo/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/worker/app/repo/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/aws-java-sdk-sts-1.11.129.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-core/1.11.129/aws-java-sdk-core-1.11.129.jar:/home/worker/app/repo/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar:/home/worker/app/repo/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar:/home/worker/app/repo/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/worker/app/repo/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-databind/2.6.6/jackson-databind-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-core/2.6.6/jackson-core-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.6/jackson-dataformat-cbor-2.6.6.jar:/home/worker/app/repo/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/home/worker/app/repo/com/amazonaws/jmespath-java/1.11.129/jmespath-java-1.11.129.jar:/home/worker/app/repo/com/amazonaws/amazon-kinesis-multilang-launcher/1.0-SNAPSHOT/amazon-kinesis-multilang-launcher-1.0-SNAPSHOT.jar -Dapp.name=multilang-daemon -Dapp.pid=15086 -Dapp.repo=/home/worker/app/repo -Dapp.home=/home/worker/app -Dbasedir=/home/worker/app com.amazonaws.services.kinesis.multilang.MultiLangDaemon config/kcl.local.properties
No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

output contains extra echo call to dump classpath before calling exec.

[ykhrustalev]

@antgus here is an example https://github.com/awslabs/amazon-kinesis-client/blob/master/src/test/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoderTest.java#L63

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111:role/name|session1

but seems does not work

[ykhrustalev]

it seems works with AWSCredentialsProvider = InstanceProfileCredentialsProvider

[pfifer]

I've just tested it using the line below (with redactions), and was able to access a stream in another account:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session

I've tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I've updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

1. In you application directory create a directory called logback (you can call it something else if you want to).
2. Download this logback.xml file, and save it to logback/logback.xml
3. Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

[holmesjr]

I have a slightly different issue with this. I have the jars loaded and it's picking up the STS provider, but it can no longer find the env vars containing my long lived creds:

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.prepare(MultiLangDaemonConfig.java:116)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:107)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:77)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:64)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.main(MultiLangDaemon.java:131)

The DefaultAWSCredentialsProviderChain works in the same config.

[holmesjr]

Oh, right, I jumped the gun. That's probably the highly obscure exception raised when the AssumeRole api call gets AccessDenied, which happened when I used the CLI with the same roles.

[holmesjr]

So, once I fixed the trust relationship, all worked well.

[Belkhojayev]

Hello @pfifer!

I'm struggling with the same problem as jazzTheJackRabbit and ykhrustalev. I'm using python.
What I tried:

1. Your approach adding aws-java-sdk-sts jar to the class path but receive an error:
   Can't find any credentials provider matching com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111111111111:role/RoleName

2. Specifying
   AWSCredentialsProvider = ProfileCredentialsProvider
   with env var AWS_PROFILE=my_profile_name.
   But received an error:
   No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

Do I understand it correctly that KCL does not read configuration from ~/.aws/config and it is useless to create profiles with (assume)roles there? This approach works for me with boto3 library on python.

Any help will be highly appreciated.

[Belkhojayev]

Found solution myself. For those experiencing the same problem and using python:

1. I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from
   /target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into
   lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

2. In the sample.parameters file I wrote:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain

Previously, I didn't specify the session_name, cause boto3 didn't need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

[pierredilauro]

Hello @pfifer!

This thread has been really helpful to setup the Python KCL with STSAssumeRoleSessionCredentialsProvider. However, we enforce MFA to Assume Roles here.

I am jumping in to know how I can configure this in the properties file. Is it even supported?

Thanks,
Pierre

[jiawen-earnin]

Hello @holmesjr . I have met the same error "Unable to load AWS credentials from any provider in the chain" Can I know how did u fix the trust relationship? Thanks

[jiawen-earnin]

Hello @pfifer !
I have created an IAM role which defines all the permissions of AWS resources I will use in my application. And in my local environment, I created config file under ~/.aws path as following:
[default]
region = us-west-2
[profile my_profile_name]
role_arn = my_role_arn
region = us-west-2
source_profile = default

then I export the AWS_PROFILE variable in the following way:
export AWS_PROFILE = my_profile_name
so that the app can run according to the IAM role.

As you mentioned above, STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar. So I manually downloaded aws-java-sdk-sts-1.11.300.jar and added it into the same directory where other jars stored. Then I changed AWSCredentialsProvider property into STSAssumeRoleSessionCredentialsProvider in property file. However, I always had this error "Unable to load AWS credentials from any provider in the chain"

I have two questions:

1. Is it possible to run the consumer code locally (in my laptop environment) with IAM role defined in /.aws/config file and what causes that error?
2. There is a PR Added aws-java-sdk-sts to jar-dependencies #70 which let setup.py downloads the jar including sts.jar. Why this PR hasn't be merged? I found the corresponding STSAssumeRoleSessionCredentialsProvider class is deprecated https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html . Is this the reason why setup.py won't download sts jar for developers?