Error enabling DynamoDB table stream

[acunabit]

Describe the bug

The table gets created successfully with streams enabled, however I'm seeing this error in the controller logs:

ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 2021-10-01T02:33:00.423Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "table", "request": "core/acktable", "error": "ValidationException: Table already has an enabled stream: TableName: acktable\n\tstatus code: 400, request id: VO65U3696SD3C7LO8NFGJ84DP3VV4KQNSO5AEMVJF66Q9ASUAAJG"}

Steps to reproduce

cat  <<EOF | kubectl apply -f -
apiVersion: dynamodb.services.k8s.aws/v1alpha1
kind: Table
metadata:
  namespace: core
  name: acktable
spec:
  tableName: acktable
  billingMode: PAY_PER_REQUEST
  attributeDefinitions:
    - attributeName: ForumName
      attributeType: S
    - attributeName: Subject
      attributeType: S
  keySchema:
    - attributeName: ForumName
      keyType: HASH
    - attributeName: Subject
      keyType: RANGE
  streamSpecification:
    streamEnabled: true
    streamViewType: NEW_AND_OLD_IMAGES
EOF

Then I see this error in the dynamodb controller logs:

ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 2021-10-01T02:42:37.001Z	DEBUG	ackrt	< r.Sync	{"kind": "Table", "namespace": "core", "name": "acktable", "generation": 1, "account": "723236915308", "role": "", "region": "ap-southeast-2", "is_adopted": false, "kind": "Table", "namespace": "core", "name": "acktable", "generation": 1}
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 2021-10-01T02:42:37.001Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "table", "request": "core/acktable", "error": "ValidationException: Table already has an enabled stream: TableName: acktable\n\tstatus code: 400, request id: DG9S2J8KQHI5PDCPL6QRCE4UB7VV4KQNSO5AEMVJF66Q9ASUAAJG"}
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller github.com/go-logr/zapr.(*zapLogger).Error
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller k8s.io/apimachinery/pkg/util/wait.BackoffUntil
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller k8s.io/apimachinery/pkg/util/wait.JitterUntil
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller k8s.io/apimachinery/pkg/util/wait.Until
ack-dynamodb-controller-5bf84dcdbc-2xnvl controller 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

Expected outcome
Table gets created successfully with streams enabled without seeing any errors in the logs.

Environment

• Kubernetes version
  1.19
• Using EKS (yes/no), if so version?
• yes : eks.6
• AWS service targeted (S3, RDS, etc.)
• Dynamodb

[jaypipes]

@a-hilaly please look into this one.

Couple things immediately stand out for me:

• There should not be stacktraces in the log output. a ValidationException is a "normal" error/fault code and should be dealt with in the normal flow of code (not make an ERROR log...). ValidationException perhaps should be listed in terminal conditions for the Table resource?
• This looks to be a situation where the Delta is erroneously picking up a need to change the streamSpecification even though it has not changed. Maybe look into whether that's happening and add an e2e test for it?

[ack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[a-hilaly]

/lifecycle frozen

[a-hilaly]

Fixed and released in dynamodb v1.1.0
/close

[ack-prow]

@a-hilaly: Closing this issue.

In response to this:

Fixed and released in dynamodb v1.1.0
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[a-hilaly]

/cc @acunabit