CredentialsProviderError: Could not load credentials from any providers when using VPC Endpoints

[Marcus4mation]

I've updated a project and have an ECS service, spinning up tasks in a private subnet without a Nat Gateway. I've configured a suite of VPC Endpoints and Gateways, for Secret manager, ECR, SSM, Bedrock and S3 to provide access to the resources.

Before moving the services to VPC endpoints, the service was working fine without any issues, but since, I've been getting the below error whenever trying to use an AWS Resource:

Error stack: ProviderError: Error response received from instance metadata service
    at ClientRequest.<anonymous> (/app/node_modules/.pnpm/@[email protected]/node_modules/@smithy/credential-provider-imds/dist-cjs/index.js:66:25)
    at ClientRequest.emit (node:events:518:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:716:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
    at Socket.socketOnData (node:_http_client:558:22)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23

The simplest example code I have:

// Configure client with VPC endpoint if provided
    const clientConfig: { region: string; endpoint?: string } = {
      region: process.env.AWS_REGION || 'ap-southeast-2',
    };

    // Add endpoint configuration if provided
    if (process.env.AWS_SECRETS_MANAGER_ENDPOINT) {
      logger.log(
        `Using custom Secrets Manager endpoint: ${process.env.AWS_SECRETS_MANAGER_ENDPOINT}`,
      );
      clientConfig.endpoint = process.env.AWS_SECRETS_MANAGER_ENDPOINT;
    }

    const client = new SecretsManagerClient({
      ...clientConfig,
      credentials: fromContainerMetadata({
        timeout: 5000,
        maxRetries: 3
      }),
    });

Investigation and remediation I've tried:

• When I've tried to hit http://169.254.170.2/v2/metadata I get a 200 response and details from the platform, so I'm reasonably sure I'm getting something.
• I've checked all my VPC Endpoints, relaxing their permissions to something like "secretsmanager:*" on all resources.
• VPC Endpoint policies have * for their principal
• Confirmed SG are configured correctly (they all provide access to the entire subnet
• Confirmed VPC Endpoints are assigned to the subnets
• Confirmed Task Role has necessary permissions to access services (they worked before)
• Attempted to increase timeout, and retries
• Noticed that the endpoints don't appear to be getting any traffic
• Attempted to force using fromContainerMetadata
• Reviewed Getting the error CredentialsProviderError: Could not load credentials from any providers #4956 and CredentialsProviderError: Could not load credentials from any providers intermittently #5829

I'm running out of ideas concerning how to resolve the issue, as due to restrictions I need to use the VPC endpoints, but am stuck