No longer able to do minor version upgrades on aws_db_instance with read replicas

[w0jnar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.14.0

Affected Resource(s)

• aws_db_instance

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  region  = "us-west-1"
}

variable "postgres_engine_version" {
  type    = string
  default = "14.1"
}

variable "postgres_engine_version_replica" {
  type    = string
  default = "14.2"
}

resource "aws_db_instance" "example" {
  allocated_storage       = 10
  engine                  = "postgres"
  engine_version          = var.postgres_engine_version
  identifier              = "example"
  instance_class          = "db.t3.micro"
  username                = "postgres"
  password                = "password"
  skip_final_snapshot     = true
  backup_retention_period = 1
  apply_immediately       = true
}

resource "aws_db_instance" "example-replica" {
  #engine              = "postgres"
  #engine_version      = var.postgres_engine_version2
  identifier          = "example-replica"
  instance_class      = "db.t3.micro"
  replicate_source_db = aws_db_instance.example.arn
  apply_immediately   = true
}

Expected Behavior

Formerly, we could do minor version upgrades by upgrading the replica's engine version first, then upgrading the primary to match.

Actual Behavior

As of v4.0, engine and engine_version "Cannot be specified for a replica", so it cannot be set on the replica.
If you attempt the upgrade the primary, you receive:

Error: modifying DB Instance example: DBUpgradeDependencyFailure: One or more of the DB Instance's read replicas need to be upgraded: example-replica

The only solution we have in mind at the moment is to get rid of the replica, upgrade the primary, then recreate the replica, but that isn't ideal.

Steps to Reproduce

1. terraform apply
2. terraform apply -var postgres_engine_version=14.2

Important Factoids

Nothing in particular.

References

• RDS minor version upgrades fail when upgrading primary and replica together #22107

[csimpkins-repay]

The only solution we have in mind at the moment is to get rid of the replica, upgrade the primary, then recreate the replica, but that isn't ideal.

Is it possible that you could set apply_immediately=false on the primary and leave it as true on the replica and then after the replica has upgraded push another change with apply_immediately=true on the primary? It's a workaround, at best, but if that works the replica could stay active at all times, right?

[take-five]

Is it possible that you could set apply_immediately=false on the primary and leave it as true on the replica and then after the replica has upgraded push another change with apply_immediately=true on the primary? It's a workaround, at best, but if that works the replica could stay active at all times, right?

AWS would refuse to send even a pending upgrade if replicas have not been upgraded

[SpComb]

This is a regression in terraform-provider-aws 3.x -> 4.x that breaks our workflows, we are unable to provision new RDS read replicas with a newer minor version, nor upgrade existing RDS read replicas and thus primary instances using terraform.

The engine_version -> ConflictsWith: []string{"replicate_source_db"} schema was added in 82717f6#diff-5684e38aee8a02cd8ffe882cacb6fccda55107f3f73d76a5aec501bb07713035R190, but I'm unable to find any rationale for this change in the commit or #22178 PR context... there are separate explicit commits for the username and name attributes, but nothing for the engine_version attribute. Does specifying the engine_version for a replica actually break in some cases, or was defining this attribute as conflicting an oversight?

Will the fix for this issue be to simply remove the ConflictsWith definition, or will it require other changes to take these possible broken cases into account?

[Jahsis]

Same issue with MariaDB replicas.

[leonsmith]

Just to clarify, this makes it impossible to upgrade neither the primary or the replica via terraform

[BenPhegan]

Is there a known workaround for this issue (other than just a Console upgrade)? This seems like a really fundamental issue. @gdavison - are you able to provide any detail as to the underlying technical reason for placing the ConflictsWith on the engine_version/replicate_source_db? I can't see anything in the AWS docs that would lead me to believe that there was a fundamental issue on the RDS side, but I might be missing something? I can see the reasoning that you would not want a significant delta between the primary and replica engine_version, but it seems that it is blocking a principal RDS upgrade workflow.

[gdavison]

The ConflictsWith was set because, if I recall correctly, the engine_version cannot be set at creation time when replicating another DB. We overlooked the case of updating versions on replicas and leader.