Amazon.S3.Transfer TransferUtility.UploadAsync() throwing exception under .Net 8 with latest MS and AWS SDK NuGet packages

[AlexanderDarbyVerisk]

Describe the bug

We have updated our Batch tool to run under .Net 8 and also updated all the NuGet packages to the latest, this is now causing a exception to be thrown from the class method TransferUtility.UploadAsync().

If the Batch tool NuGet packages are reverted to old versions then everything works as expected and the file uploads to S3.

The Batch tool is running on the AWS Batch Fargate compute environment.

Previous NuGet package versions when the exception did not happen, worked correctly:

"Amazon.Extensions.Configuration.SystemsManager" Version="5.0.2"
"AWSSDK.Extensions.NETCore.Setup" Version="3.7.5"
"CommandLineParser" Version="2.9.1"
"Microsoft.Extensions.Configuration.Json" Version="6.0.0"
"Microsoft.Extensions.DependencyInjection" Version="6.0.1"
"Microsoft.Extensions.Hosting" Version="6.0.1"
"Microsoft.Extensions.Options.ConfigurationExtensions" Version="6.0.0"
"Serilog.Sinks.Console" Version="4.1.0"
"AWSSDK.RedshiftDataAPIService" Version="3.7.102.9"
"AWSSDK.S3" Version="3.7.103.33"
"CsvHelper" Version="30.0.1"
"JWT" Version="10.0.2"
"Microsoft.Extensions.Options" Version="6.0.0" />
"Polly" Version="7.2.3" />
"RestSharp" Version="109.0.1" />
"Serilog" Version="2.12.0" />

Latest NuGet package versions where the exception is happening, not working correctly:

"Amazon.Extensions.Configuration.SystemsManager" Version="6.2.1"
"AWSSDK.Extensions.NETCore.Setup" Version="3.7.301"
"CommandLineParser" Version="2.9.1"
"Microsoft.Extensions.Configuration.Json" Version="8.0.0"
"Microsoft.Extensions.DependencyInjection" Version="8.0.0"
"Microsoft.Extensions.Hosting" Version="8.0.0"
"Microsoft.Extensions.Options.ConfigurationExtensions" Version="8.0.0"
"Serilog.Sinks.Console" Version="6.0.0"
"AWSSDK.RedshiftDataAPIService" Version="3.7.400.4"
"AWSSDK.S3" Version="3.7.400.4"
"CsvHelper" Version="33.0.1"
"JWT" Version="10.1.1"
"Microsoft.Extensions.Options" Version="8.0.2"
"Polly" Version="8.4.1"
"RestSharp" Version="111.4.1"
"Serilog" Version="4.0.1"

Running under the .Net 8 runtime makes no difference, the S3 method works as expected with the old NuGet package versions. The issue is a NuGet library version update that has introduced a new bug.

Expected Behavior

The method completes successfully and uploads the file to S3.

Current Behavior

The exception details are as follows:

exception={
"AmazonId2": "hidden",
"AmazonCloudFrontId": null,
"ResponseBody": null,
"Message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
"ErrorType": 0,
"ErrorCode": "SignatureDoesNotMatch",
"RequestId": "hidden",
"StatusCode": 403,
"Retryable": null,
"Data": {

},
"InnerException": {
"Response": {
"StatusCode": 403,
"IsSuccessStatusCode": false,
"ContentType": "application/xml",
"ContentLength": 0,
"ResponseBody": {

  }
},
"Message": "Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.",
"Data": {
  
},
"InnerException": null,
"HelpLink": null,
"Source": "AWSSDK.Core",
"HResult": -2146233088,
"StackTrace": " at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)\n at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)\n at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)"

},
"HelpLink": null,
"Source": "AWSSDK.Core",
"HResult": -2146233088,
"StackTrace": " at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)\n at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)\n at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.S3.Transfer.Internal.SimpleUploadCommand.ExecuteAsync(CancellationToken cancellationToken)\n at
Our code methods
}

Reproduction Steps

Code where the method is called:

public async Task WriteFileToS3Async(Stream stream, string outputFileName)
{
var s3Utility = new TransferUtility(_client);
await s3Utility.UploadAsync(stream, $"{_environmentVariables.S3BucketName}/{_environmentVariables.OutputDir}", outputFileName);
}

The class member '_client' object is a IAmazonS3 interface that is injected on app start-up i.e. 'services.AddAWSService IAmazonS3 ();'

The 'stream' object passed in is a FileStream type.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

"Amazon.Extensions.Configuration.SystemsManager" Version="6.2.1"
"AWSSDK.Extensions.NETCore.Setup" Version="3.7.301"
"CommandLineParser" Version="2.9.1"
"Microsoft.Extensions.Configuration.Json" Version="8.0.0"
"Microsoft.Extensions.DependencyInjection" Version="8.0.0"
"Microsoft.Extensions.Hosting" Version="8.0.0"
"Microsoft.Extensions.Options.ConfigurationExtensions" Version="8.0.0"
"Serilog.Sinks.Console" Version="6.0.0"
"AWSSDK.RedshiftDataAPIService" Version="3.7.400.4"
"AWSSDK.S3" Version="3.7.400.4"
"CsvHelper" Version="33.0.1"
"JWT" Version="10.1.1"
"Microsoft.Extensions.Options" Version="8.0.2"
"Polly" Version="8.4.1"
"RestSharp" Version="111.4.1"
"Serilog" Version="4.0.1"

Targeted .NET Platform

.Net 8

Operating System and version

AWS Batch Fargate, Linux docker container

[bhoradc]

Hi @AlexanderDarbyVerisk,

Thank you for reporting the issue. To confirm, the new version of AWSSDK.Core is 3.7.400.4 or above, right?

Also, it would be great if you are able to provide a self-contained code sample that can help to reproduce the error reliably.

Regards,
Chaitanya

[AlexanderDarbyVerisk]

Hello,

Yes, transitive package referenced from the parent AWS NuGet packages is AWSSDK.Core version 3.7.400.5

The AWS packages being used are:

Amazon.Extensions.Configuration.SystemsManager Version="6.2.1"
AWSSDK.Extensions.NETCore.Setup Version="3.7.301"
AWSSDK.RedshiftDataAPIService Version="3.7.400.4"
AWSSDK.S3 Version="3.7.400.4"

The parent AWS packages also use the following transitive packages:

AWSSDK.AppConfigData version 3.7.301.35
AWSSDK.SimpleSystemsManagement version 3.7.303.23

I expect it would just be a case of trying to upload a file to S3 using the TransferUtility.UploadAsync() method which will cause the exception.

The AWS credentials are automatically obtained through AWS Batch Fargate process.

[AlexanderDarbyVerisk]

It is a bit strange though that the latest stable version of the AWS NuGet package AWSSDK.Extensions.NETCore.Setup is version 3.7.301?

The previous old version was 3.7.5 that we used.

Looks like the latest version number has gone down?

[dscpinheiro]

@AlexanderDarbyVerisk Do you still see the error if you change your code to use this instead?

await s3Utility.UploadAsync(stream, _environmentVariables.S3BucketName, $"{_environmentVariables.OutputDir}/{outputFileName}");

The SDK used to support a key prefix (i.e. the / character) in the bucket name but that's no longer the case in the latest version of the AWSSDK.S3 package (see #2622 (comment) for a longer explanation).

[AlexanderDarbyVerisk]

@dscpinheiro

That has fixed our issue. Thanks for the help!

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[fgiacomelli]

I have a similar problem, in a .NET9 based project, after upgrading packages from

• AWSSDK.S3 3.7.410.7
• AWSSDK.SQS 3.7.400.65

to

• AWSSDK.S3 3.7.416.13
• AWSSDK.SQS 3.7.400.137

we started to receive the exception "XAmzContentSHA256Mismatch".

I found that the error is raised only when the request is configured with the DisableDefaultChecksumValidation set to false with no checksum specified, in this way:

  var uploadRequest = new TransferUtilityUploadRequest
  {
    BucketName = PackageInfo.GetApplicationInfo(CmdHandlerBackup.AWS_S3_BUCKET_NAME),
    FilePath = sourceFullPath,
    CannedACL = S3CannedACL.PublicReadWrite,
    Key = targetFileName,
    DisableDefaultChecksumValidation = false
  };

while setting its value to true, or setting one of ChecksumValue property to some value doesn't throw the exception.

The strange for me is that I can set any value as Checksum, because i.e. if I set the ChecksumSHA256 field to "XYZ" I don't receive the exception, so it seems that the bucket doesn't really check that the hash I passed match the one calculated when the file is uploaded.

I have to specify that the Bucket S3 that I'm using is not an Amazon Bucket but it's an Aruba Cloud Bucket.

[dscpinheiro]

@fgiacomelli Since this is a 3rd-party implementation we won't be able to help, you'll need to reach out to Aruba Cloud and ask them to figure out why their implementation doesn't match S3's.

You can enable debug logging to compare the requests before and after:

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

var s3 = new AmazonS3Client(...);
var tu = new TransferUtility(s3);

var uploadRequest = new TransferUtilityUploadRequest { ... };