Skip to content

Commit 4a88f00

Browse files
Jan-MFxKu
Jan-M
and
FxKu
authored
Full AWS gp3 support for iops and througput config. (#1261)
Support new AWS EBS volume type `gp3` with `iops` and `throughput` in the manifest. Co-authored-by: Felix Kunde <[email protected]>
1 parent 4ea0b5f commit 4a88f00

File tree

15 files changed

+522
-152
lines changed

15 files changed

+522
-152
lines changed

‎charts/postgres-operator/crds/postgresqls.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -557,6 +557,8 @@ spec:
557557
required:
558558
- size
559559
properties:
560+
iops:
561+
type: integer
560562
size:
561563
type: string
562564
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
@@ -565,6 +567,8 @@ spec:
565567
type: string
566568
subPath:
567569
type: string
570+
throughput:
571+
type: integer
568572
status:
569573
type: object
570574
additionalProperties:

‎docs/developer.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,24 @@ Then you can for example check the Patroni logs:
235235
kubectl logs acid-minimal-cluster-0
236236
```
237237

238+
## Unit tests with Mocks and K8s Fake API
239+
240+
Whenever possible you should rely on leveraging proper mocks and K8s fake client that allows full fledged testing of K8s objects in your unit tests.
241+
242+
To enable mocks, a code annotation is needed:
243+
[Mock code gen annotation](https://github.com/zalando/postgres-operator/blob/master/pkg/util/volumes/volumes.go#L3)
244+
245+
To generate mocks run:
246+
```bash
247+
make mocks
248+
```
249+
250+
Examples for mocks can be found in:
251+
[Example mock usage](https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/volumes_test.go#L248)
252+
253+
Examples for fake K8s objects can be found in:
254+
[Example fake K8s client usage](https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/volumes_test.go#L166)
255+
238256
## End-to-end tests
239257

240258
The operator provides reference end-to-end (e2e) tests to

‎docs/reference/cluster_manifest.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -338,13 +338,13 @@ archive is supported.
338338
the url to S3 bucket containing the WAL archive of the remote primary.
339339
Required when the `standby` section is present.
340340

341-
## EBS volume resizing
341+
## Volume properties
342342

343343
Those parameters are grouped under the `volume` top-level key and define the
344344
properties of the persistent storage that stores Postgres data.
345345

346346
* **size**
347-
the size of the target EBS volume. Usual Kubernetes size modifiers, i.e. `Gi`
347+
the size of the target volume. Usual Kubernetes size modifiers, i.e. `Gi`
348348
or `Mi`, apply. Required.
349349

350350
* **storageClass**
@@ -356,6 +356,14 @@ properties of the persistent storage that stores Postgres data.
356356
* **subPath**
357357
Subpath to use when mounting volume into Spilo container. Optional.
358358

359+
* **iops**
360+
When running the operator on AWS the latest generation of EBS volumes (`gp3`)
361+
allows for configuring the number of IOPS. Maximum is 16000. Optional.
362+
363+
* **throughput**
364+
When running the operator on AWS the latest generation of EBS volumes (`gp3`)
365+
allows for configuring the throughput in MB/s. Maximum is 1000. Optional.
366+
359367
## Sidecar definitions
360368

361369
Those parameters are defined under the `sidecars` key. They consist of a list

‎docs/reference/operator_parameters.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -373,10 +373,13 @@ configuration they are grouped under the `kubernetes` key.
373373
possible value is `parallel`.
374374

375375
* **storage_resize_mode**
376-
defines how operator handels the difference between requested volume size and
377-
actual size. Available options are: ebs - tries to resize EBS volume, pvc -
378-
changes PVC definition, off - disables resize of the volumes. Default is "pvc".
379-
When using OpenShift please use one of the other available options.
376+
defines how operator handles the difference between the requested volume size and
377+
the actual size. Available options are:
378+
1. `ebs` : operator resizes EBS volumes directly and executes `resizefs` within a pod
379+
2. `pvc` : operator only changes PVC definition
380+
3. `off` : disables resize of the volumes.
381+
4. `mixed` :operator uses AWS API to adjust size, throughput, and IOPS, and calls pvc change for file system resize
382+
Default is "pvc".
380383

381384
## Kubernetes resource requests
382385

‎go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ module github.com/zalando/postgres-operator
33
go 1.15
44

55
require (
6-
github.com/aws/aws-sdk-go v1.36.3
6+
github.com/aws/aws-sdk-go v1.36.29
77
github.com/golang/mock v1.4.4
88
github.com/lib/pq v1.9.0
99
github.com/motomux/pretty v0.0.0-20161209205251-b2aad2c9a95d

‎go.sum

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbt
3535
github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
3636
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
3737
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
38+
github.com/Storytel/gomock-matchers v1.2.0 h1:VPsbL6c/9/eCa4rH13LOEXPsIsnA1z+INamGIx1lWQo=
39+
github.com/Storytel/gomock-matchers v1.2.0/go.mod h1:7HEuwyU/eq/W3mrSqPSYETGXiTyU2um0Rrb+dh5KmKM=
3840
github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
3941
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
4042
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@@ -45,8 +47,8 @@ github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5
4547
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
4648
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
4749
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
48-
github.com/aws/aws-sdk-go v1.36.3 h1:KYpG5OegwW3xgOsMxy01nj/Td281yxi1Ha2lJQJs4tI=
49-
github.com/aws/aws-sdk-go v1.36.3/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
50+
github.com/aws/aws-sdk-go v1.36.29 h1:lM1G3AF1+7vzFm0n7hfH8r2+750BTo+6Lo6FtPB7kzk=
51+
github.com/aws/aws-sdk-go v1.36.29/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
5052
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
5153
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
5254
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -182,6 +184,7 @@ github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF
182184
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
183185
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
184186
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
187+
github.com/golang/mock v1.2.1-0.20190311213431-837231f7bb37/go.mod h1:L3bP22mxdfCUHSUVMs+SPJMx55FrxQew7MSXT11Q86g=
185188
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
186189
github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
187190
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
@@ -525,6 +528,7 @@ golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGm
525528
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
526529
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
527530
golang.org/x/tools v0.0.0-20190125232054-d66bd3c5d5a6/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
531+
golang.org/x/tools v0.0.0-20190221204921-83362c3779f5/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
528532
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
529533
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
530534
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=

‎manifests/complete-postgres-manifest.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ spec:
4444
volume:
4545
size: 1Gi
4646
# storageClass: my-sc
47+
# iops: 1000 # for EBS gp3
48+
# throughput: 250 # in MB/s for EBS gp3
4749
additionalVolumes:
4850
- name: empty
4951
mountPath: /opt/empty

‎manifests/postgresql.crd.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -553,6 +553,8 @@ spec:
553553
required:
554554
- size
555555
properties:
556+
iops:
557+
type: integer
556558
size:
557559
type: string
558560
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
@@ -561,6 +563,8 @@ spec:
561563
type: string
562564
subPath:
563565
type: string
566+
throughput:
567+
type: integer
564568
status:
565569
type: object
566570
additionalProperties:

‎pkg/apis/acid.zalan.do/v1/crds.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -835,6 +835,9 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
835835
Type: "object",
836836
Required: []string{"size"},
837837
Properties: map[string]apiextv1.JSONSchemaProps{
838+
"iops": {
839+
Type: "integer",
840+
},
838841
"size": {
839842
Type: "string",
840843
Description: "Value must not be zero",
@@ -846,6 +849,9 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
846849
"subPath": {
847850
Type: "string",
848851
},
852+
"throughput": {
853+
Type: "integer",
854+
},
849855
},
850856
},
851857
},

‎pkg/apis/acid.zalan.do/v1/postgresql_type.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ type Volume struct {
118118
SubPath string `json:"subPath,omitempty"`
119119
Iops *int64 `json:"iops,omitempty"`
120120
Throughput *int64 `json:"throughput,omitempty"`
121+
VolumeType string `json:"type,omitempty"`
121122
}
122123

123124
// AdditionalVolume specs additional optional volumes for statefulset

‎pkg/cluster/sync.go

Lines changed: 2 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -53,41 +53,15 @@ func (c *Cluster) Sync(newSpec *acidv1.Postgresql) error {
5353
return err
5454
}
5555

56-
c.logger.Debugf("syncing volumes using %q storage resize mode", c.OpConfig.StorageResizeMode)
57-
5856
if c.OpConfig.EnableEBSGp3Migration {
5957
err = c.executeEBSMigration()
6058
if nil != err {
6159
return err
6260
}
6361
}
6462

65-
if c.OpConfig.StorageResizeMode == "mixed" {
66-
// mixed op uses AWS API to adjust size,throughput,iops and calls pvc chance for file system resize
67-
68-
// resize pvc to adjust filesystem size until better K8s support
69-
if err = c.syncVolumeClaims(); err != nil {
70-
err = fmt.Errorf("could not sync persistent volume claims: %v", err)
71-
return err
72-
}
73-
} else if c.OpConfig.StorageResizeMode == "pvc" {
74-
if err = c.syncVolumeClaims(); err != nil {
75-
err = fmt.Errorf("could not sync persistent volume claims: %v", err)
76-
return err
77-
}
78-
} else if c.OpConfig.StorageResizeMode == "ebs" {
79-
// potentially enlarge volumes before changing the statefulset. By doing that
80-
// in this order we make sure the operator is not stuck waiting for a pod that
81-
// cannot start because it ran out of disk space.
82-
// TODO: handle the case of the cluster that is downsized and enlarged again
83-
// (there will be a volume from the old pod for which we can't act before the
84-
// the statefulset modification is concluded)
85-
if err = c.syncVolumes(); err != nil {
86-
err = fmt.Errorf("could not sync persistent volumes: %v", err)
87-
return err
88-
}
89-
} else {
90-
c.logger.Infof("Storage resize is disabled (storage_resize_mode is off). Skipping volume sync.")
63+
if err = c.syncVolumes(); err != nil {
64+
return err
9165
}
9266

9367
if err = c.enforceMinResourceLimits(&c.Spec); err != nil {
@@ -590,48 +564,6 @@ func (c *Cluster) syncRoles() (err error) {
590564
return nil
591565
}
592566

593-
// syncVolumeClaims reads all persistent volume claims and checks that their size matches the one declared in the statefulset.
594-
func (c *Cluster) syncVolumeClaims() error {
595-
c.setProcessName("syncing volume claims")
596-
597-
act, err := c.volumeClaimsNeedResizing(c.Spec.Volume)
598-
if err != nil {
599-
return fmt.Errorf("could not compare size of the volume claims: %v", err)
600-
}
601-
if !act {
602-
c.logger.Infof("volume claims do not require changes")
603-
return nil
604-
}
605-
if err := c.resizeVolumeClaims(c.Spec.Volume); err != nil {
606-
return fmt.Errorf("could not sync volume claims: %v", err)
607-
}
608-
609-
c.logger.Infof("volume claims have been synced successfully")
610-
611-
return nil
612-
}
613-
614-
// syncVolumes reads all persistent volumes and checks that their size matches the one declared in the statefulset.
615-
func (c *Cluster) syncVolumes() error {
616-
c.setProcessName("syncing volumes")
617-
618-
act, err := c.volumesNeedResizing(c.Spec.Volume)
619-
if err != nil {
620-
return fmt.Errorf("could not compare size of the volumes: %v", err)
621-
}
622-
if !act {
623-
return nil
624-
}
625-
626-
if err := c.resizeVolumes(); err != nil {
627-
return fmt.Errorf("could not sync volumes: %v", err)
628-
}
629-
630-
c.logger.Infof("volumes have been synced successfully")
631-
632-
return nil
633-
}
634-
635567
func (c *Cluster) syncDatabases() error {
636568
c.setProcessName("syncing databases")
637569

0 commit comments

Comments
 (0)