Skip to content

Inconsistent result when creating cloudstack_firewall #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mwaag opened this issue Apr 26, 2024 · 6 comments · May be fixed by #164 or #165
Open

Inconsistent result when creating cloudstack_firewall #115

mwaag opened this issue Apr 26, 2024 · 6 comments · May be fixed by #164 or #165
Assignees
Labels
bug Something isn't working enhancement New feature or request
Milestone

Comments

@mwaag
Copy link

mwaag commented Apr 26, 2024

Hi,
this is my first try to open an issue here, please bear with me, if this is not the correct way. I will thankfully accept any hints to optimize it in the future. I will try to orientate to the issue-template from the cloudstack-project.
I may repeat this with different tf-versions / os'es. Let me know, if you need more information.

PROVIDER INFORMATION
  • Versoin: 0.5.0
TERRAFORM INFORMATION
  • Versoin: 1.8.2
HYPERVISOR INFORMATION
  • Hypervisor: VMWare ESXi
  • Version: 7.0.3
CLOUDSTACK VERSION
  • 4.17.2.0
CONFIGURATION
main.tf:
terraform {
  required_providers {
    cloudstack = {
        source = "cloudstack/cloudstack"
        version = "0.5.0"
    }
  }
}

provider "cloudstack" {
  # Configuration Options
  api_url    = "${var.cloudstack_api_url}"
  api_key    = "${var.cloudstack_api_key}"
  secret_key = "${var.cloudstack_secret_key}"
}

resource "cloudstack_template" "ubuntu2204" {
  name = "Ubuntu 22.04"
  format = "OVA"
  hypervisor = "VMware"
  os_type = "Other Linux (64-bit)"
  url = "${var.cloudstack_template_url}"
  zone = "Enterprise"
  project = "mwatest01"
  is_dynamically_scalable = true
  is_extractable = false
  is_featured = false
  is_public = true
  password_enabled = false
  is_ready_timeout = 600
  
}

resource "cloudstack_network" "snw-demo" {
    name                = "demo-network"
    display_text        = "demo-network"
    cidr                = "172.16.0.0/24"
    network_offering    = "DefaultIsolatedNetworkOfferingWithSourceNatService"
    zone                = "Enterprise"
    project             = "mwatest01"
    source_nat_ip = true
  
}

# resource "cloudstack_ipaddress" "default-ip0" {
#   network_id = cloudstack_network.snw-demo.id
#   zone = "Enterprise"
#   project = "mwatest01"
#   
# }

resource "cloudstack_instance" "instance-demo" {
    name = "vm-demo"
    display_name = "vm-demo"
    service_offering = "XS Instanz"
    template = cloudstack_template.ubuntu2204.id
    project = "mwatest01"
    zone = "Enterprise"
    start_vm = true
    expunge = true
    network_id = cloudstack_network.snw-demo.id
  
}

resource "cloudstack_firewall" "default" {
  ip_address_id = cloudstack_network.snw-demo.source_nat_ip_id

  rule {
    cidr_list = ["172.26.251.57/32"]
    protocol = "tcp"
    ports = ["22"]
  }
}

OS / ENVIRONMENT
  • OS: Ubuntu 22.04.3 LTS on Windows Subsystem for Linux (WSL 2) on Windows 11 (64-bit)
SUMMARY
Error: Provider produced inconsistent result after apply

when trying to deploy a firewall-rule to a simple isolated guest-network via cloudstack-provider.

DETAILS
  • We utilize projects for client-isolation
  • Service-offering for vm is a custom one
  • Result is reproducable with newly assigned non-snat-ip-adress (see resource cloudstack_ipadress)
STEPS TO REPRODUCE
  • create main.tf with contents mentioned above
  • run terraform apply
EXPECTED RESULTS
  • Rule will be created successfully
  • Rule will be included in terraform state
ACTUAL RESULTS
  • Rule is created successfully

  • Rule is not included in terraform state (Rerun will try to create a new rule)

  • Full Output:

terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudstack_firewall.default will be created
  + resource "cloudstack_firewall" "default" {
      + id            = (known after apply)
      + ip_address_id = (known after apply)
      + managed       = false
      + parallelism   = 2

      + rule {
          + cidr_list = [
              + "172.26.251.57/32",
            ]
          + icmp_code = (known after apply)
          + icmp_type = (known after apply)
          + ports     = [
              + "22",
            ]
          + protocol  = "tcp"
          + uuids     = (known after apply)
        }
    }

  # cloudstack_instance.instance-demo will be created
  + resource "cloudstack_instance" "instance-demo" {
      + display_name     = "vm-demo"
      + expunge          = true
      + group            = (known after apply)
      + id               = (known after apply)
      + ip_address       = (known after apply)
      + name             = "vm-demo"
      + network_id       = (known after apply)
      + project          = "mwatest01"
      + root_disk_size   = (known after apply)
      + service_offering = "XS Instanz"
      + start_vm         = true
      + tags             = (known after apply)
      + template         = (known after apply)
      + uefi             = false
      + zone             = "Enterprise"
    }

  # cloudstack_network.snw-demo will be created
  + resource "cloudstack_network" "snw-demo" {
      + acl_id                = "none"
      + cidr                  = "172.16.0.0/24"
      + display_text          = "demo-network"
      + endip                 = (known after apply)
      + gateway               = (known after apply)
      + id                    = (known after apply)
      + name                  = "demo-network"
      + network_domain        = (known after apply)
      + network_offering      = "DefaultIsolatedNetworkOfferingWithSourceNatService"
      + project               = "mwatest01"
      + source_nat_ip         = true
      + source_nat_ip_address = (known after apply)
      + source_nat_ip_id      = (known after apply)
      + startip               = (known after apply)
      + tags                  = (known after apply)
      + zone                  = "Enterprise"
    }

  # cloudstack_template.ubuntu2204 will be created
  + resource "cloudstack_template" "ubuntu2204" {
      + display_text            = (known after apply)
      + format                  = "OVA"
      + hypervisor              = "VMware"
      + id                      = (known after apply)
      + is_dynamically_scalable = true
      + is_extractable          = false
      + is_featured             = false
      + is_public               = true
      + is_ready                = (known after apply)
      + is_ready_timeout        = 600
      + name                    = "Ubuntu 22.04"
      + os_type                 = "Other Linux (64-bit)"
      + password_enabled        = false
      + project                 = "mwatest01"
      + tags                    = (known after apply)
      + url                     = "http://20.82.104.206/Ubuntu_22.04_jammy_user.ova"
      + zone                    = "Enterprise"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

cloudstack_network.snw-demo: Creating...
cloudstack_template.ubuntu2204: Creating...
cloudstack_network.snw-demo: Creation complete after 2s [id=68247c09-802a-44b6-bc59-e31a9230c7d1]
cloudstack_firewall.default: Creating...
cloudstack_template.ubuntu2204: Still creating... [10s elapsed]
cloudstack_template.ubuntu2204: Still creating... [20s elapsed]
cloudstack_template.ubuntu2204: Still creating... [30s elapsed]
cloudstack_template.ubuntu2204: Still creating... [40s elapsed]
cloudstack_template.ubuntu2204: Still creating... [50s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m0s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m10s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m20s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m30s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m40s elapsed]
cloudstack_template.ubuntu2204: Still creating... [1m50s elapsed]
cloudstack_template.ubuntu2204: Still creating... [2m0s elapsed]
cloudstack_template.ubuntu2204: Still creating... [2m10s elapsed]
cloudstack_template.ubuntu2204: Creation complete after 2m14s [id=5bd0e49f-942c-4893-853e-ab74d725d1fe]
cloudstack_instance.instance-demo: Creating...
cloudstack_instance.instance-demo: Still creating... [10s elapsed]
cloudstack_instance.instance-demo: Still creating... [20s elapsed]
cloudstack_instance.instance-demo: Still creating... [30s elapsed]
cloudstack_instance.instance-demo: Still creating... [40s elapsed]
cloudstack_instance.instance-demo: Still creating... [50s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m0s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m10s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m20s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m30s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m40s elapsed]
cloudstack_instance.instance-demo: Still creating... [1m50s elapsed]
cloudstack_instance.instance-demo: Still creating... [2m0s elapsed]
cloudstack_instance.instance-demo: Still creating... [2m10s elapsed]
cloudstack_instance.instance-demo: Still creating... [2m21s elapsed]
cloudstack_instance.instance-demo: Still creating... [2m31s elapsed]
cloudstack_instance.instance-demo: Still creating... [2m41s elapsed]
cloudstack_instance.instance-demo: Creation complete after 2m48s [id=5f1e1fbc-a259-4a3a-bd27-0c2b2ecc2c56]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to cloudstack_firewall.default, provider "provider[\"registry.terraform.io/cloudstack/cloudstack\"]" produced an unexpected new value: Root
│ object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
@mwaag
Copy link
Author

mwaag commented Apr 26, 2024

My colleague just found this: https://www.reddit.com/r/Terraform/comments/m5nv14/comment/gr29zct/?utm_source=share&utm_medium=web2x&context=3

Using the argument 'managed = true' workaround my problem, but it seems it would be cleaner when the problem is catched from the provider. What do you think?

@kiranchavala
Copy link
Collaborator

@mwaag

Thanks for reporting the issue

The issue is occurring only if project are used.

Marking it as a bug and improvement request

@kiranchavala kiranchavala added bug Something isn't working enhancement New feature or request labels Apr 29, 2024
@kiranchavala kiranchavala added this to the v0.6.0 milestone Apr 29, 2024
@baltazorbest
Copy link

Hello,
Any updates regarding the bug fix?

@baltazorbest
Copy link

Hello.

After searching for the source of the issue, I found it here, the issue is related to the condition:

After changing it to:

else if managed { d.SetId("") }

the firewall was created successfully with the parameters:

managed = false  
project = my_project_id

However, I'm not sure if this is the correct approach to solving the issue.

@DaanHoogland
Copy link
Contributor

Hey @baltazorbest , nice to meet you. As said please submit your change in a Pull Request and we can test. As you said you have already tested in your production environment, we are more than helf way merging it already ;) . Meybe a unit test to make sure it is covored would be nice.

@baltazorbest baltazorbest linked a pull request Mar 18, 2025 that will close this issue
@Longsight
Copy link

Longsight commented Mar 27, 2025

@baltazorbest I'm not convinced that swapping the logic of that conditional is the fix we're looking for.

I was hitting a similar issue with Network ACL rules, and after some hours of debugging found that it was occurring there because I wasn't supplying a project ID to cloudstack_network_acl_rule, which meant that resourceCloudStackNetworkACLRuleRead was returning early:

	_, count, err := cs.NetworkACL.GetNetworkACLListByID(
		d.Id(),
		cloudstack.WithProject(d.Get("project").(string)),
	)
	if err != nil {
		if count == 0 {
			log.Printf(
				"[DEBUG] Network ACL list %s does no longer exist", d.Id())
			d.SetId("")
			return nil
		}

		return err
	}
func (s *NetworkACLService) GetNetworkACLListByID(id string, opts ...OptionFunc) (*NetworkACLList, int, error) {
	p := &ListNetworkACLListsParams{}
	p.p = make(map[string]interface{})

	p.p["id"] = id

	for _, fn := range append(s.cs.options, opts...) {
		if err := fn(s.cs, p); err != nil {
			return nil, -1, err
		}
	}

	l, err := s.ListNetworkACLLists(p)
	if err != nil {
		if strings.Contains(err.Error(), fmt.Sprintf(
			"Invalid parameter id value=%s due to incorrect long value format, "+
				"or entity does not exist", id)) {
			return nil, 0, fmt.Errorf("No match found for %s: %+v", id, l)
		}
		return nil, -1, err
	}

        // -
        // - This is where a zero-count set results in an error, which is cascaded to the calling function
        // -

	if l.Count == 0 {
		return nil, l.Count, fmt.Errorf("No match found for %s: %+v", id, l)
	}

	if l.Count == 1 {
		return l.NetworkACLLists[0], l.Count, nil
	}
	return nil, l.Count, fmt.Errorf("There is more then one result for NetworkACLList UUID: %s!", id)
}

GetNetworkACLListByID always returned an empty set and an error if the parent NetworkACLList belonged to a project, and the project parameter was omitted from the resource. This is by design - project-scoped resources are not returned by List* API calls unless a project (called projectid in the API) parameter is supplied, even if listall is true.

When I included the project parameter in the Terraform resource, the rules were matched and created correctly.


In the case of cloudstack_firewall, the relevant read logic is:

	// Get all the rules from the running environment
	p := cs.Firewall.NewListFirewallRulesParams()
	p.SetIpaddressid(d.Id())
	p.SetListall(true)

	l, err := cs.Firewall.ListFirewallRules(p)
	if err != nil {
		return err
	}
func (s *FirewallService) ListFirewallRules(p *ListFirewallRulesParams) (*ListFirewallRulesResponse, error) {
	resp, err := s.cs.newRequest("listFirewallRules", p.toURLValues())
	if err != nil {
		return nil, err
	}

	resp, err = convertFirewallServiceResponse(resp)
	if err != nil {
		return nil, err
	}

	var r ListFirewallRulesResponse
	if err := json.Unmarshal(resp, &r); err != nil {
		return nil, err
	}

	return &r, nil
}

In this case, if the firewall is scoped to a project, this code will never find any rules, as the project is never included in the API call. However, unlike GetNetworkACLListByID , ListFirewallRules does not throw an error in the case of an empty set, so resourceCloudStackFirewallRead continues:

        // -
        // ruleMap is supposed to be a map of existing rules fetched from the API
        // BUT:
        // if the firewall belongs to a project, it will always be empty
        // This is incorrect behaviour
        // -

	ruleMap := make(map[string]*cloudstack.FirewallRule, l.Count)
	for _, r := range l.FirewallRules {
		ruleMap[r.Id] = r
	}

	// Create an empty schema.Set to hold all rules
	rules := resourceCloudStackFirewall().Schema["rule"].ZeroValue().(*schema.Set)

	// Read all rules that are configured
	if rs := d.Get("rule").(*schema.Set); rs.Len() > 0 {
		for _, rule := range rs.List() {

                        // ...rule matching logic

Under normal circumstances, ruleMap is a map of the rules fetched from the API, rs is a set of the rules passed by the Terraform resource, and rules is the resulting intersection of the two, representing rules that exist in Cloudstack and are known to Terraform. At this point in the code, when reading an existing resource, there are two valid scenarios:

  • managed is false, and rules has 1 or more elements
  • managed is true, and rules has 0 or more elements

This logic exists because a managed cloudstack_firewall with zero rules is a valid means of ensuring that no rules are added to the firewall by other means until you choose to add them with Terraform.

If the firewall is scoped to a project, and rulesMap is empty, then the rule matching logic will ensure that rules is also empty. This should not happen, and the block you've identified is the code that handles this:

        // If we've matched any of the existing Cloudstack firewall rules to the rules in the resource,
        // include them in the resulting ResourceData:
	if rules.Len() > 0 {
		d.Set("rule", rules)
        // If we haven't, AND this resource is NOT marked as `managed`, unset the resource ID
        // of the newly-created `cloudstack_firewall` resource, as this should only happen IF
        // the resource does not actually exist during Refresh, and should be removed from Terraform state.
	} else if !managed {
		d.SetId("")
	}

The logic here is correct - during a normal resource read, if rules.Len() == 0 and !managed then this suggests that a firewall resource that exists in Terraform state has been destroyed in infrastructure, and should be marked as destroyed in state.

However, since resourceCloudStackFirewallRead is also called from resourceCloudStackFirewallCreate after initial resource creation (so that unknown values like UUID can be fetched from Cloudstack), this logic causes the apply operation to fail if the firewall rule belongs to a project, as Create operations should never return a resource with an empty Id:

	// ...
	// The SetId method must be called with a non-empty value for the managed
	// resource instance to be properly saved into the Terraform state and
	// avoid a "inconsistent result after apply" error.
	// ...
	Create CreateFunc

This is why your fix appears to work: if you change else if !managed to else if managed, then the resource ID is no longer unset on apply even though the Read operation found an empty ruleset, so the actual error is masked by what looks like a successful Create that just happened to create zero rules. What you will probably find, however, is that duplicate rules are created on every apply, as existing rules will never be seen by the provider, so it will think that they're always missing. The more correct fix is to include the projectid parameter in the ListFirewallRules API call in the first place, so that the newly-created firewall resource is returned by Cloudstack and recognised by the provider.


Since cloudstack_firewall is mapped to a given ip_address_id rather than the id of the actual firewall, we cannot duplicate the project parameter logic used in other resources, as we can't call cs.Firewall.GetFirewallRuleByID without the firewall ID. What we can do, however, is ensure that the project is included in the ListFirewallRules call, by making use of cloudstack.WithProject directly:

--- a/cloudstack/resource_cloudstack_firewall.go
+++ b/cloudstack/resource_cloudstack_firewall.go
@@ -95,6 +95,12 @@ func resourceCloudStackFirewall() *schema.Resource {
                                },
                        },
 
+                       "project": {
+                               Type:     schema.TypeString,
+                               Optional: true,
+                               ForceNew: true,
+                       },
+
                        "parallelism": {
                                Type:     schema.TypeInt,
                                Optional: true,
@@ -256,6 +262,12 @@ func resourceCloudStackFirewallRead(d *schema.ResourceData, meta interface{}) er
        p.SetIpaddressid(d.Id())
        p.SetListall(true)
 
+       // Use WithProject(d.Get("project").(string)) to set the `projectid` parameter on the API call
+       // if we've passed a `project` parameter to the resource
+       if err := cloudstack.WithProject(d.Get("project").(string))(cs, p); err != nil {
+               return err
+       }
+
        l, err := cs.Firewall.ListFirewallRules(p)
        if err != nil {
                return err

I'm not in a position to test this fix myself right now as I'm not currently using firewall rules in my deployment, but I'm fairly confident it should work.

Side note: we have to supply the project resource parameter ourselves rather than look it up from the PublicIpAddress referenced in ip_address_id because, even though cs.Address.GetPublicIpAddressByID exists, we hit the same problem of scoping - unless we already know the projectid of the resource in Cloudstack, the underlying List* call will fail to return the resource we're looking for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
6 participants