Ability for Terraform plugins to give status updates via stdout in CLI

[alexng-canuck]

Current Terraform Version

v0.11.7

Use-cases

Any Terraform provider may benefit from being able to log useful status updates to Terraform CLI users.

For example, the OCI provider has retry logic whenever it encounters HTTP errors from the service. This retry logic can run for at least 10 minutes and maybe hours if a user has configured a higher retry value.

Currently, we log retry attempts from the provider but they can only be seen by Terraform CLI user if they have TF_LOG=DEBUG setting.

Another example, is when creating resources that take a very long time to provision and may require some polling to wait for it to reach a final state.

Without the TF_LOG setting, the user doesn't have much visibility into why an apply command is taking so long and has the appearance that the command is hanging.

Attempted Solutions

To see these status updates, the user must have TF_LOG=DEBUG but most users don't have this enabled by default.

Proposal

There should be a way for providers to surface updates about a resource's progress w.r.t. the Create, Update, Read, Delete operations.

References

[apparentlymart]

Hi @alexng-canuck! Thanks for sharing these use-cases.

We do want to make it possible for providers to signal certain kinds of asynchronous status back to the UI, including ideas such as showing percentage completion (for situations where that can be estimated) or showing a status keyword ("creating backup", "formatting", etc). At the moment there is no backwards channel through which to write such things that isn't already used for another purpose (for example, provider stdout is used internally for plugin communication and log transit.)

The key requirements we've identified so far for a feature like this are:

• It must be a separate channel from the existing log transit channel, because existing provider logs are far too verbose. Instead, providers must explicitly indicate that they want to indicate a status to the user.
• It must be higher-level than just writing raw bytes to Terraform's stdout, because Terraform's own CLI code needs to retain full control of the terminal when running in CLI mode, and in future it is likely that there will be other frontends (such as a JSON stream output, Output Terraform log as json terraform#20390) which could be rendering the results in a tabular form rather than a streaming log form, etc. Probably it would be some sort of well-defined "status" object which providers can populate and send updates for, which could then be mapped into other UI representations.

There will be some follow-on work after v0.12.0 is released to update the SDK to support the new features v0.12 has already introduced, but after that I expect we'll start looking at designing and implementing other capabilities that require Terraform Core and the SDK to coordinate via changes to the plugin protocol, of which this is one.

[ghost]

This issue has been automatically migrated to terraform-providers/terraform-provider-oci#736 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to terraform-providers/terraform-provider-oci#736.

[alexng-canuck]

This issue should not be closed. I think the HashiBot is being a tad aggressive in migrating this issue.

[alexng-canuck]

@markanthonyfranco can you reopen this issue? I think the provider/oci label is causing HashiBot to erroneously identify it as a provider-side issue when in fact it should stay in the Terraform repo.

[apparentlymart]

Indeed, I'm not sure why this was tagged as being an issue in the OCI provider.

[markanthonyfranco]

Sorry about that. I created a different label... partner/oci .

Let's see if that one will be left alone by Hashibot

[radeksimko]

This is directly related to hashicorp/terraform#17267 - i.e. some work will need to be done in core 1st before we can expose the right interface to plugins via SDK.

[bflad]

Additional reference: hashicorp/terraform-provider-aws#10750 (in this case, we’d update that UI element to say waiting for ENI (eni-12345678) deletion or something along those lines)

[paultyng]

Nothing new here, the upstream core issue (hashicorp/terraform#17267) is required to be address first before we could add functionality to the SDK.