Proposal: Generator Plugins for Configuration Generation

[apparentlymart]

So far in my use of Terraform in my workplace I've uncovered a number of use-cases where it's useful to generate Terraform configurations based on outside data.

So far I've been handling that by having users run extra preparation steps before they run terraform plan, which write JSON-style Terraform configs into a directory. This has worked okay, but the need for extra steps means that the standard usage pattern of Terraform does not apply.

Based on what I've learned from generating Terraform configs, I'd like to offer a proposal for integrating the concept of configuration generation directly into Terraform. This is an attempt to "pave the cowpath" by taking a pattern that I've already successfully applied and building Terraform syntax around it.

Consider the following configuration as a motivating example:

// Load a directory full of files into an S3 bucket
resource "aws_s3_bucket_object" "file" {
    foreach "dir_entry" {
        path = "${path.module}/htdocs"
        recursive = true
        include_dirs = false
    }

    bucket = "${var.s3_bucket_name}"
    key = "${foreach.relative_path}"
    source = "${foreach.absolute_path}"
}

(This example is inspired by my terraform-s3-dir utility.)

The resource construct is extended with a new child block foreach, which is conceptually similar to count but rather than producing resources based on a sequence of numbers it produces resources based on executing a generator plugin, which in this example is dir_entry.

The contract for a generator plugin is to take the given input arguments and produce (essentially) a map[string]interface{}, where each map key is a hashable unique identifier for an object and the value is a structure that can be used from the ${foreach...} interpolation syntax.

Some details in the sections that follow...

Effect on terraform plan

As noted earlier, foreach is conceptually similar to count in that it causes one resource configuration block to produce multiple resource instances. In the case of count these are named like aws_s3_bucket_object.file.0. In the case offoreacha similar convention applies except that the index is replaced by the result of applying`hashcode.String`` to each item's key.

Since the resources are identified by a hash of their key, diffing can produce a sensible result as long as the keys remain consistent between runs. In the dir_entry example above the object key could be the same as the relative_path attribute value, so adding and removing files would cause the corresponding resource instances to be added and removed in the diff.

foreach configuration block

Since generation is a plan-time concept, the foreach block may only contain interpolations that are known statically, such as var, path, interpolation functions. It specifically may not reference resource or module attributes.

Otherwise the structure of the configuration block is under the control of the generator plugin, much as with provisioners.

${foreach...} interpolation syntax

The ${foreach...} interpolation syntax is again comparable to the ${count...} syntax, but the attributes within it depend on the structure returned by the generator function. Each generator is free to define its own set of attributes in an arbitrary hierarchical structure, just like resources can.

Generator plugin interface

Generator plugins have a similar interface to provisioner plugins, including the same configuration validation methods but with the Apply method replaced with Generate(*ResourceConfig) map[string]interface{} .

It is expected that a generator will produce tens of items at most, so returning the whole structure in-memory (rather than streaming it e.g. using a goroutine) should be sufficient. A generator producing hundreds or thousands of items would result in hundreds or thousands of Terraform resources, which I believe is already beyond Terraform's design assumptions.

Whereas provisioners are verbs, generators should be named as nouns describing what kind of items the generator produces, so that the declaration reads as (for example) "For each directory entry...".

Interaction with count

To avoid the combinatoric complexity that would result, using count and foreach together in the same resource block is not permitted.

Additional Example Use-cases

Some further examples of generators plugins that might be implemented, and what they could be used for...

DNS records from a standard zone file

resource "aws_route53_record" "foo" {
    foreach "dns_zone_record" {
        // A "zone file" per RFC1035
        zone = "${file('example.com.zone')}"
        // Disregard SOA records
        ignore_types = ["SOA"]
    }

    zone_id = "${var.route53_zone_id}"
    name = "${foreach.name}"
    type = "${foreach.type}"
    records = ["${foreach.records}"]
    ttl = "${foreach.ttl}"
}

Users (or indeed anything else) from a YAML file

resource "aws_iam_user" "user" {
    foreach "yaml_entry" {
        // Give a YAML document that either has a mapping or a list at its root,
        // to produce one resource instance per entry in that structure.
        source = "${file('users.yaml')}"
    }

    name = "${foreach.username}"
    path = "/staff/"
}

A custom external program for generating VPN routes

resource "aws_vpn_connection_route" "route" {
    foreach "local_exec_result" {
        // Any executable that produces a JSON object as output and
        // exits with a successful status.
        command = "${path.module}/generate-route-map"
    }

    destination_cidr_block = "${foreach.cidr_block}"
    vpn_connection_id = "${aws_vpn_connection.main.id}"
}

[apparentlymart]

I am closing this because I think the same use-cases could be met in a much simpler way in conjunction with some other proposals I've made:

• Implement Data-driven Terraform Configuration #4169 to allow Terraform to read data from various sources before planning.
• Add a foreach meta-attribute to resources that takes directly an array or map of values and "fans out" to one resource instance for each array/map item. foreach may be interpolated with var and data dependencies, just like count.
• Optionally implement Partial/Progressive Configuration Changes #4149 to allow dynamic mappings, at the expense of requiring multiple Terraform runs to successfully apply the configuration completely.

Here are two of the examples above, re-worked to fit this model:

// A new yaml_pairs interpolation function allows YAML parsing
resource "aws_iam_user" "user" {
    foreach = ["${yaml_entries(file("users.yaml"))}"]

    name = "${foreach.value.username}"
    path = "/staff/"
}

// A hypothetical "execute a local program" data-source, in conjunction with a JSON
// parsing function, supports the VPN route table example.
data "local_exec" "route_map" {
    command = "${path.module/generate-route-map}"
}
resource "aws_vpn_connection_route" "route" {
    // The generate-route-map program prints JSON to stdout,
    // which we can now parse.
    foreach = ["${json_entries(data.local_exec.route_map.stdout)}"]

    destination_cidr_block = "${foreach.value.cidr_block}"
    vpn_connection_id = "${aws_vpn_connection.main.id}"
}

However, I'm not going to make another issue for this right now since I've already created a rather substantial pyramid of aspirational architecture proposals, and so I'm going to leave this one for now and consider it again later based on the outcomes of those proposals.

[OJFord]

Any further thoughts on this @apparentlymart?

As far as I can tell, your second example in above (not original) is possible today with data.external, but the first still isn't.

It's the "bulk upload to S3 without specifying everything" use-case that I'm particularly interested in. I nearly had it with count; then ran into it not allowing interpolation of resource attributes.

[apparentlymart]

Hi @OJFord!

This is indeed not yet entirely possible. My time to work on "big stuff" in Terraform has been limited recently due to other life stuff taking priority, but I would still like to find a way to address these use-cases.

For the S3 use-case perhaps it might end up looking something like this, using similar building blocks as I described in my last comment above:

resource "aws_s3_bucket_object" "file" {
  foreach = "${deepfiles("${path.module}/htdocs")}"

  bucket = "${var.s3_bucket_name}"
  key = "${foreach.value}"
  source = "${path.module}/${foreach.value}"
}

The missing parts here are:

• The foreach meta-attribute as I described before.
• A new deepfiles interpolation function that returns the relative paths of all of the files descending from a given directory.

This case is simpler than some of the others because reading files from disk doesn't require reaching out to external services and so we can safely do it "statically" while loading the config, similarly to how file works. That means this example can get away with a limited foreach that doesn't support resource attributes, which avoids implementing #4149 before this would work.

---

If we added that deepfiles function then we could actually get this done with just count today I think, albeit with some performance drawbacks on deep trees due to repeatedly re-evaluating the function:

resource "aws_s3_bucket_object" "file" {
  count = "${length(deepfiles("${path.module}/htdocs"))}"

  bucket = "${var.s3_bucket_name}"
  key = "${element(deepfiles("${path.module}/htdocs"), count.index)}"
  source = "${path.module}/${element(deepfiles("${path.module}/htdocs"), count.index)}"
}

Could be interesting to implement that function first, since it'd probably be useful for other use-cases as well, and then if we add something like foreach later that would build on this by making the syntax more straightforward and ensuring that the function gets called only once.

[OJFord]

Thanks for the reply!

My time to work on "big stuff" in Terraform has been limited recently due to other life stuff taking priority

As it should - no problem!

If we added that deepfiles function then we could actually get this done with just count today

Oh, that would do it for me if that works. The restriction on counts interpolation is only that it can't access other resources' attributes, then; not that it can't do anything non-constant?

[pmoust]

Would you be open to re-opening this?
Related: #8573 #4410

I can see this being very valuable, albeit hard to implement safely and pre-1.0

[apparentlymart]

The Terraform team (which I am now a part of, but wasn't when I made my earlier comments) is currently leaning towards the foreach solution I described in my most recent comment above, but in order for that to be useful we first need to fix some annoyances and limitations with how Terraform deals with lists and maps in general, so this has got folded into a general bucket of configuration language improvements.

At the time of writing it's just a set of use-cases rather than a plan, but we're planning to consider all of the use-cases together and see what language features make sense to cover them as best we can with as little complexity as possible.

So I'm going to leave this closed not because there's no plan to work on something like this, but because it represents a defunct plan that will be replaced with a new plan in future. We'll open new issues when there's a more concrete plan to work on these new features.