When referencing a module stored locally if it has a provider block, you get credential errors

[togakangaroo]

I am working on an example for a tf module we're developing at work. So the folder structure is something like

- module-files.tf
 |- examples
    |- async
       |- example.files.tf

In my example files I have

module "lambda" {
  source        = "../../"
  ...
}

This works, I can tflocal apply and everything

now the module itself (in module-files.tf) got added an aws provider block

provider "aws" { }

and suddenly this fails

│   on ../../providers.tf line 11:
│   11: provider "aws" {
│ 
│ Earlier versions of Terraform used empty provider blocks ("proxy provider configurations") for child modules to declare their need to be passed a provider configuration by their callers. That approach was
│ ambiguous and is now deprecated.
│ 
│ If you control this module, you can migrate to the new declaration syntax by removing all of the empty provider "aws" blocks and then adding or updating an entry like the following to the required_providers
│ block of module.lambda:
│     aws = {
│       source = "hashicorp/aws"
│     }
╵
╷
│ Error: No valid credential sources found
│ 
│   with module.lambda.provider["registry.terraform.io/hashicorp/aws"],
│   on ../../providers.tf line 11, in provider "aws":
│   11: provider "aws" {
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable

---

Having inspected how tflocal works, I think I understand why.

I believe that tflocal generates a temporary localstack_providers_override.tf file and then cleans it up, right? Well for the module it has no way of creating one there. If I manually create this file at the module-files.tf level then my tflocal apply works!

So it seems to me that tflocal either needs to walk to included modules, or at least provide a configuration where you can say "ALSO localstack-ify these other directories temporarily"

[togakangaroo]

Hmm, where does the localstack_providers_override.tf actually come from? I"d like to add it to my ignore and tell my users to download it, but it doesn't seem to come from any place in this repo

oh, I see

[togakangaroo]

Solution I came up with for my project in case anyone else needs it. It's not the slickest thing and I think a built-in solution would be better, but it works.

"""The way that tflocal works is by generating a temporary local override
terraform file that overrides aws provider endpoints to localstack ones. This
unfortunately doesn't work well with locally included modules like we do in our examples.

This script uses tflocal internals to generate the local override file in the
current directory. tflocal itself is a python script, so this script should be
passed the path to this script itself as installed within the python
environment. If one is not provided, the script will infer this from `which
tflocal`. It will handle resolving things out of pyenv properly as well.

More Info:
- https://github.com/localstack/terraform-local/issues/67

Usage:

   python ./examples/_tools/generate_local_overrides.py
   python ./examples/_tools/generate_local_overrides.py /home/gmauer/.pyenv/versions/3.12.6/bin/tflocal
   python ./examples/_tools/generate_local_overrides.py -h
"""
import logging
import os
import subprocess
import sys
from importlib.machinery import SourceFileLoader
from importlib.util import module_from_spec, spec_from_loader

logging.basicConfig(stream=sys.stdout, level=os.getenv('LOG_LEVEL', logging.INFO))
logger = logging.getLogger(__name__)

if len(sys.argv) > 1 and sys.argv[1] == '-h':
    print(__doc__)
    sys.exit(0)

tflocal_path = sys.argv[1] if len(sys.argv) > 1 else subprocess.check_output(['which', 'tflocal']).decode().strip()
if '.pyenv/shims/' in tflocal_path:
    tflocal_environment_path = subprocess.check_output(['pyenv', 'prefix']).decode().strip()
    tflocal_path = f"{tflocal_environment_path}/bin/tflocal"

logger.debug(f"Will attempt to load tflocal from {tflocal_path}")

spec = spec_from_loader("tflocal", SourceFileLoader("tflocal", tflocal_path))
tflocal = module_from_spec(spec)
spec.loader.exec_module(tflocal)
logger.debug(f"tflocal module loaded")

tflocal.get_tf_version(dict(os.environ))
override_file_path = tflocal.create_provider_config_file(tflocal.determine_provider_aliases())
logger.info(f"Generated override file at {os.path.abspath(override_file_path)}.\nDELETE THIS FILE IF YOU WOULD LIKE TO RUN AGAINST AN ACTUAL AWS ACCOUNT.")

[alexrashed]

Hi @togakangaroo!
Thanks for your issue, and for explaining / preserving your workaround!
Would you be up to trying to tackle this issue in tflocal directly by submitting a PR?

[togakangaroo]

You know what, yeah I think I probably could - I'll take a look in the next couple days

[togakangaroo]

PR created.

I was going to capture stderr and automatically recommend this if detecting a validation error but it seems like there's a few different ways of invoking terraform and doing so gets into some complexity so I opted to just document

[togakangaroo]

Any way to get it merged in? I'd love to remove the workaround from my project before I have to move onto other work