"Unable to load credentials from any of the providers" for parallel calls to an S3Client with ProfileCredentialsProvider

[InsertPikachuFace]

Describe the bug

I am still encountering the behavior from this closed issue #5001.

In my case I have a list of S3-Objects that exist in a bucket and download & process them in a parallel stream.

It sometimes manages some objects, but always fails with Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain... (see stacktrace) eventually. The number of elements doesn't seem to have an impact.

In my local testing the same program worked flawlessly when either the stream was sequential instead of parallel or the S3Client has already executed another call before the parallel stream starts.

This makes the analysis from that previous issue seem likely, that there's a race condition for resolveCredentials in ProfileCredentialProvider.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The parallel stream should be processed normally.

Current Behavior

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=my profile name, profileFile=ProfileFile(sections=[profiles, sso-session, services], profiles=[Profile(name=default, properties=[region, output, ca_bundle]), Profile(name=my profile name, properties=[role_arn, region, source_profile])])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=my profile name): Cannot invoke "software.amazon.awssdk.profiles.ProfileFile.profile(String)" because "profileFile" is null, ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.lambda$trySelectAuthScheme$6(S3AuthSchemeInterceptor.java:181)
at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:81)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.trySelectAuthScheme(S3AuthSchemeInterceptor.java:181)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.selectAuthScheme(S3AuthSchemeInterceptor.java:99)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.beforeExecution(S3AuthSchemeInterceptor.java:71)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:319)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:155)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:62)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:6837)
at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:11977)
at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:12243)
at Test.lambda$main$1(Test.java:89)

Reproduction Steps

A simple version of my program could look like this:

try (S3Client s3Client = S3Client.builder().region(Region.EU_CENTRAL_1).credentialsProvider(DefaultCredentialsProvider.builder().build()).build()) {
  // works if s3Client is already called here with e.g: 
  // s3Client.listBuckets().buckets().forEach(IO::println);

  String bucketName = "...";
  List<String> s3Keys = List.of("...", "...", "...", ...);
  s3Keys.parallelStream() // works if this is just .stream()
              .map(s3Key -> {
                  try {
                      return new String(s3Client.getObject(b -> b.bucket(bucketName).key(s3Key)).readAllBytes());
                  } catch (IOException e) {
                      throw new RuntimeException(e);
                  }
              })
              .map(String::length)
              .forEach(System.out::println);
}

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.42.4

JDK version used

amazon-corretto-jdk25.0.0_36

Operating System and version

Windows 11

[bhoradc]

Hi @InsertPikachuFace

Thanks for reporting the issue. I have been attempting to reproduce this on both macOS and Windows with Corretto 25, but haven't been able to trigger the issue so far.

Would you be able to share a complete self-contained sample code that reproduces the issue, including the AWS config & credentials file structure (post redacting sensitive details)?

Regards,
Chaitanya

[InsertPikachuFace]

@bhoradc The Stacktrace already contains the AwsCredentialsProviderChain, which lists the entire config. It's basically just config- & credentials-files with a Maintainer-role profile for the AWS-Scope (it's provided by a script from our company, so I can't say much about the specifics there).