Skip to content

Commit 3b7d8f7

Browse files
bonzofenixbonzofenix
authored
(feature): enable autoscaler to run with publicapi as a cf app (#3556)
* WIP: Configure publicapi-mta * Fix typo * Update devbox * Add cleanup-autoscaler-deployments target to Makefile * Remove Makefile from src/db directory * Add commons-lang3 dependency to src/db/pom.xml * Remove commons-lang3 dependency from src/db pom.xml * Add commons-lang3 dependency to src/db/pom.xml * WIP2 * WIP * WIP * Add shellcheck exception and create CERTS_DIR in apply-api-changelog.sh * Enable a previously focused test case for scaling out in the AutoScaler Public API acceptance tests. * Remove empty line from devbox.lock file * Implement support for catalog configuration via user-provided service in autoscaler API * Remove unused logs * Remove unused docs * Cleanup mta * Update todo description * remove comment * Update oha from 0.6.5 to 1.8.0 across multiple architectures * Remove WIP label check step from acceptance_tests_reusable workflow * Moved CPU threshold case handling to else block in deploy-autoscaler.sh and added CPU_LOWER_THRESHOLD to build-extension-file.sh * Add ENABLE_MTAR environment variable to deploy apps step in acceptance_tests_reusable workflow * Refactor deploy-autoscaler.sh to use cf_target function and remove redundant cf target command * Remove use-cf-services.yml from acceptance tests MTA workflow configuration * Update Python version from 3.12.5 to 3.13.2 in devbox.lock * Add step to register autoscaler in acceptance_tests_reusable workflow * Fix failing test in scheduler * Fix sonaque • Consolidate xfcc configuration into nested hashes for both eventgenerator and scalingengine specs • Ensure default xfcc valid org and space GUIDs are set correctly in properties * Debug register broker * Fix integration test
1 parent 3b894f7 commit 3b7d8f7

File tree

39 files changed

+574
-240
lines changed

39 files changed

+574
-240
lines changed

Diff for: ‎.github/workflows/acceptance_tests_mta.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,5 @@ jobs:
2323
operations/set-release-version.yml
2424
operations/enable-metricsforwarder-via-syslog-agent.yml
2525
operations/enable-scheduler-logging.yml
26-
operations/use-cf-services.yml
2726
secrets:
2827
bbl_ssh_key: "${{ secrets.BBL_SSH_KEY }}"

Diff for: ‎.github/workflows/acceptance_tests_reusable.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,18 @@ jobs:
6060
shell: bash
6161
env:
6262
OPS_FILES: ${{ inputs.ops_files }}
63+
ENABLE_MTAR: ${{ inputs.deploy_apps }}
6364
run: make --directory="${AUTOSCALER_DIR}" deploy-autoscaler
6465

6566
- name: Deploy Apps
6667
shell: bash
6768
run: make --directory="${AUTOSCALER_DIR}" deploy-apps
6869
if: ${{ inputs.deploy_apps }}
6970

71+
- name: Register autoscaler
72+
shell: bash
73+
run: make --directory="${AUTOSCALER_DIR}" deploy-register-cf
74+
7075
acceptance_tests:
7176
name: Acceptance Tests
7277
needs: [ deploy_autoscaler ]

Diff for: ‎.sonarcloud.properties

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
sonar.exclusions=spec/**

Diff for: ‎Makefile

+8-2
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,6 @@ test-app.go-mod-tidy:
298298
make --directory='${go-test-app-dir}' go-mod-tidy
299299

300300

301-
302301
.PHONY: mod-download
303302
mod-download:
304303
@for folder in $$(find . -maxdepth 3 -name "go.mod" -exec dirname {} \;);\
@@ -328,7 +327,7 @@ update-uaac-nix-package:
328327
make --directory='./nix/packages/uaac' gemset.nix
329328

330329
.PHONY: deploy-autoscaler deploy-register-cf deploy-autoscaler-bosh deploy-cleanup
331-
deploy-autoscaler: go-mod-vendor uaac db scheduler deploy-autoscaler-bosh deploy-register-cf ## Deploy autoscaler to OSS dev environment
330+
deploy-autoscaler: go-mod-vendor uaac db scheduler deploy-autoscaler-bosh
332331
deploy-register-cf:
333332
echo " - registering broker with cf"
334333
${CI_DIR}/autoscaler/scripts/register-broker.sh
@@ -403,6 +402,10 @@ acceptance-tests-config:
403402
cleanup-concourse:
404403
@${CI_DIR}/autoscaler/scripts/cleanup-concourse.sh
405404

405+
.PHONY: cleanup-autoscaler-deployments
406+
cleanup-autoscaler-deployments:
407+
@${CI_DIR}/autoscaler/scripts/cleanup-autoscaler-deployments.sh
408+
406409
.PHONY: cf-login
407410
cf-login: ## Login to OSS CF dev environment
408411
@${CI_DIR}/autoscaler/scripts/cf-login.sh
@@ -477,5 +480,8 @@ deploy-apps:
477480
echo " - deploying apps"
478481
DEBUG="${DEBUG}" ${CI_DIR}/autoscaler/scripts/deploy-apps.sh
479482

483+
undeploy-apps:
484+
echo " - undeploying apps"
485+
DEBUG="${DEBUG}" ${CI_DIR}/autoscaler/scripts/undeploy-apps.sh
480486
help: ## Show this help
481487
@grep --extended-regexp --no-filename '\s##\s' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

Diff for: ‎ci/autoscaler/pipeline.yml

+1
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ anchors:
3232
operations/enable-metricsforwarder-via-syslog-agent.yml
3333
operations/enable-scheduler-logging.yml
3434
operations/use-cf-services.yml
35+
operations/configure-cf-services.yml
3536
3637
app-autoscaler-ops-files-upgrade: &app-autoscaler-ops-files-upgrade
3738
OPS_FILES: |

Diff for: ‎ci/autoscaler/scripts/deploy-apps.sh

+6
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,12 @@ popd > /dev/null
1313
function deploy() {
1414
log "Deploying autoscaler apps for bosh deployment '${deployment_name}' "
1515
pushd "${autoscaler_dir}/src/autoscaler" > /dev/null
16+
17+
# Update the default_catalog.json with the deployment name
18+
rm -f api/default_catalog.json
19+
cp api/default_catalog.json.tpl api/default_catalog.json
20+
sed --in-place "s|DEPLOYMENT_NAME|${deployment_name}|g" api/default_catalog.json
21+
1622
make mta-deploy
1723
popd > /dev/null
1824
}

Diff for: ‎ci/autoscaler/scripts/deploy-autoscaler.sh

+10-6
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,6 @@ ${autoscaler_dir}/operations/enable-scheduler-logging.yml
2525
EOF
2626
)}
2727

28-
case "${cpu_upper_threshold}" in
29-
"100") ;;
30-
"200") ops_files+=" ${autoscaler_dir}/operations/cpu_upper_threshold_200.yml" ;;
31-
"400") ops_files+=" ${autoscaler_dir}/operations/cpu_upper_threshold_400.yml" ;;
32-
*) echo "No Ops file for cpu_upper_threshold of ${cpu_upper_threshold}"; exit 1 ;;
33-
esac
3428

3529
CURRENT_COMMIT_HASH=$(cd "${autoscaler_dir}"; git log -1 --pretty=format:"%H")
3630
bosh_release_version=${RELEASE_VERSION:-${CURRENT_COMMIT_HASH}-${deployment_name}}
@@ -166,14 +160,24 @@ function find_or_upload_release() {
166160
function pre_deploy() {
167161
if [[ "${enable_mtar}" == "true" ]]; then
168162
ops_files+=" ${autoscaler_dir}/operations/use-cf-services.yml"
163+
ops_files+=" ${autoscaler_dir}/operations/configure-cf-services.yml"
169164
cf_login
170165

166+
cf_target ${autoscaler_org} ${autoscaler_space}
171167
local autoscaler_cf_server_xfcc_valid_org_guid=$(cf org ${AUTOSCALER_ORG} --guid)
172168
local autoscaler_cf_server_xfcc_valid_space_guid=$(cf space ${AUTOSCALER_SPACE} --guid)
173169

174170
add_variable "autoscaler_cf_server_xfcc_valid_org_guid" "${autoscaler_cf_server_xfcc_valid_org_guid}"
175171
add_variable "autoscaler_cf_server_xfcc_valid_space_guid" "${autoscaler_cf_server_xfcc_valid_space_guid}"
172+
else
173+
case "${cpu_upper_threshold}" in
174+
"100") ;;
175+
"200") ops_files+=" ${autoscaler_dir}/operations/cpu_upper_threshold_200.yml" ;;
176+
"400") ops_files+=" ${autoscaler_dir}/operations/cpu_upper_threshold_400.yml" ;;
177+
*) echo "No Ops file for cpu_upper_threshold of ${cpu_upper_threshold}"; exit 1 ;;
178+
esac
176179
fi
180+
177181
}
178182

179183
log "Deploying autoscaler '${bosh_release_version}' with name '${deployment_name}'"

Diff for: ‎ci/autoscaler/scripts/undeploy-apps.sh

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#! /usr/bin/env bash
2+
# shellcheck disable=SC2086,SC2034,SC2155
3+
set -euo pipefail
4+
5+
script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
6+
source "${script_dir}/common.sh"
7+
source "${script_dir}/vars.source.sh"
8+
9+
pushd "${bbl_state_path}" > /dev/null
10+
eval "$(bbl print-env)"
11+
popd > /dev/null
12+
13+
function undeploy() {
14+
log "Undeploying autoscaler apps for bosh deployment '${deployment_name}'"
15+
pushd "${autoscaler_dir}/src/autoscaler" > /dev/null
16+
make mta-undeploy
17+
popd > /dev/null
18+
}
19+
20+
bosh_login
21+
cf_login
22+
cf_target "${autoscaler_org}" "${autoscaler_space}"
23+
undeploy

Diff for: ‎jobs/eventgenerator/spec

+3-3
Original file line numberDiff line numberDiff line change
@@ -124,15 +124,15 @@ properties:
124124
autoscaler.eventgenerator.server_key:
125125
description: "PEM-encoded server key"
126126

127-
autoscaler.cf_server.port:
127+
autoscaler.eventgenerator.cf_server.port:
128128
description: "the listening port of cf xfcc endpoint"
129129
default: 8080
130130

131-
autoscaler.cf_server.xfcc.valid_org_guid:
131+
autoscaler.eventgenerator.cf_server.xfcc.valid_org_guid:
132132
description: approve org guid for xfcc endpoint
133133
default: ''
134134

135-
autoscaler.cf_server.xfcc.valid_space_guid:
135+
autoscaler.eventgenerator.cf_server.xfcc.valid_space_guid:
136136
description: approve space guid for xfcc endpoint
137137
default: ''
138138

Diff for: ‎jobs/eventgenerator/templates/eventgenerator.yml.erb

+3-3
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@ server:
6666
node_index: <%= nodeIndex %>
6767

6868
cf_server:
69-
port: <%= p("autoscaler.cf_server.port") %>
69+
port: <%= p("autoscaler.eventgenerator.cf_server.port") %>
7070
xfcc:
71-
valid_org_guid: <%= p("autoscaler.cf_server.xfcc.valid_org_guid") %>
72-
valid_space_guid: <%= p("autoscaler.cf_server.xfcc.valid_space_guid") %>
71+
valid_org_guid: <%= p("autoscaler.eventgenerator.cf_server.xfcc.valid_org_guid") %>
72+
valid_space_guid: <%= p("autoscaler.eventgenerator.cf_server.xfcc.valid_space_guid") %>
7373

7474
logging:
7575
level: <%= p("autoscaler.eventgenerator.logging.level") %>

Diff for: ‎jobs/scalingengine/spec

+3-3
Original file line numberDiff line numberDiff line change
@@ -169,15 +169,15 @@ properties:
169169
autoscaler.scalingengine.server_key:
170170
description: "PEM-encoded server key"
171171

172-
autoscaler.cf_server.port:
172+
autoscaler.scalingengine.cf_server.port:
173173
description: "the listening port of cf xfcc endpoint"
174174
default: 8080
175175

176-
autoscaler.cf_server.xfcc.valid_org_guid:
176+
autoscaler.scalingengine.cf_server.xfcc.valid_org_guid:
177177
description: allowed org guid for xfcc endpoint
178178
default: ''
179179

180-
autoscaler.cf_server.xfcc.valid_space_guid:
180+
autoscaler.scalingengine.cf_server.xfcc.valid_space_guid:
181181
description: allowed space guid for xfcc endpoint
182182
default: ''
183183

Diff for: ‎jobs/scalingengine/templates/scalingengine.yml.erb

+3-3
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,10 @@ cf:
5252
max_idle_conns_per_host_ms: <%= p("autoscaler.cf.max_idle_conns_per_host_ms") %>
5353

5454
cf_server:
55-
port: <%= p("autoscaler.cf_server.port") %>
55+
port: <%= p("autoscaler.scalingengine.cf_server.port") %>
5656
xfcc:
57-
valid_org_guid: <%= p("autoscaler.cf_server.xfcc.valid_org_guid") %>
58-
valid_space_guid: <%= p("autoscaler.cf_server.xfcc.valid_space_guid") %>
57+
valid_org_guid: <%= p("autoscaler.scalingengine.cf_server.xfcc.valid_org_guid") %>
58+
valid_space_guid: <%= p("autoscaler.scalingengine.cf_server.xfcc.valid_space_guid") %>
5959

6060
server:
6161
port: <%= p("autoscaler.scalingengine.server.port") %>

Diff for: ‎operations/configure-cf-services.yml

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## SCHEDULER - Enable cf Server to receive calls from api running on cf --
2+
- type: replace
3+
path: /instance_groups/name=scheduler/jobs/name=scheduler/properties/autoscaler/scheduler/cf_server?/xfcc?/valid_org_guid?
4+
value: ((!autoscaler_cf_server_xfcc_valid_org_guid))
5+
6+
- type: replace
7+
path: /instance_groups/name=scheduler/jobs/name=scheduler/properties/autoscaler/scheduler/cf_server?/xfcc?/valid_space_guid?
8+
value: ((!autoscaler_cf_server_xfcc_valid_space_guid))
9+
10+
- type: replace
11+
path: /instance_groups/name=scheduler/jobs/name=scheduler/properties/autoscaler/scheduler/http?/port?
12+
value: 6105
13+
14+
- type: replace
15+
path: /instance_groups/name=scheduler/jobs/name=route_registrar/properties/route_registrar/routes/-
16+
value:
17+
name: ((deployment_name))-cf-scheduler
18+
registration_interval: 20s
19+
port: 6105
20+
tags:
21+
component: autoscaler_cf_scheduler
22+
uris:
23+
- ((deployment_name))-cf-scheduler.((system_domain))
24+
25+
## SCALINGENGINE - Enable cf Server to receive calls from api running on cf --
26+
- type: replace
27+
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/xfcc?/valid_org_guid?
28+
value: ((!autoscaler_cf_server_xfcc_valid_org_guid))
29+
30+
- type: replace
31+
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/xfcc?/valid_space_guid?
32+
value: ((!autoscaler_cf_server_xfcc_valid_space_guid)) # variable in bosh
33+
34+
- type: replace
35+
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/port?
36+
value: 6205
37+
38+
- type: replace
39+
path: /instance_groups/name=scalingengine/jobs/name=route_registrar/properties/route_registrar/routes/-
40+
value:
41+
name: ((deployment_name))-cf-scalingengine
42+
registration_interval: 20s
43+
port: 6205
44+
tags:
45+
component: autoscaler_cf_scalingengine
46+
uris:
47+
- ((deployment_name))-cf-scalingengine.((system_domain))
48+
49+
50+
## EVENTGENERATOR - Enable cf Server to receive calls from api running on cf --
51+
52+
- type: replace
53+
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_org_guid?
54+
value: ((!autoscaler_cf_server_xfcc_valid_org_guid))
55+
56+
- type: replace
57+
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_space_guid?
58+
value: ((!autoscaler_cf_server_xfcc_valid_space_guid))
59+
60+
61+
- type: replace
62+
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/port?
63+
value: 8080
64+
65+
- type: replace
66+
path: /instance_groups/name=eventgenerator/jobs/name=route_registrar/properties/route_registrar/routes/-
67+
value:
68+
name: ((deployment_name))-cf-eventgenerator
69+
registration_interval: 20s
70+
port: 8080
71+
tags:
72+
component: autoscaler_cf_eventgenerator
73+
uris:
74+
- ((deployment_name))-cf-eventgenerator.((system_domain))

Diff for: ‎operations/use-cf-services.yml

+6-55
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
## add router tcp route for postgres
1+
## POSTGRES- Add router tcp route for postgres
22
- type: replace
33
path: /instance_groups/name=postgres/jobs/-
44
value:
@@ -35,11 +35,6 @@
3535
uris:
3636
- ((deployment_name))-postgres.tcp.((system_domain))
3737

38-
- type: replace
39-
path: /instance_groups/name=apiserver/jobs/name=golangapiserver/properties/autoscaler/apiserver/metrics_forwarder/host
40-
value: ((deployment_name))-metricsforwarder.((system_domain))
41-
42-
4338
- type: replace
4439
path: /variables/name=postgres_server/options/alternative_names/-
4540
value: ((deployment_name))-postgres.tcp.((system_domain))
@@ -48,57 +43,13 @@
4843
path: /variables/name=postgres_client/options/alternative_names/-
4944
value: ((deployment_name))-postgres.tcp.((system_domain))
5045

46+
## APISERVER - remove publicapiserver
5147
- type: remove
52-
path: /instance_groups/name=metricsforwarder
53-
54-
55-
## SCALINGENGINE - Enable cf Server to receive calls from api running on cf --
56-
- type: replace
57-
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/xfcc?/valid_org_guid?
58-
value: ((autoscaler_cf_server_xfcc_valid_org_guid))
59-
60-
- type: replace
61-
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/xfcc?/valid_space_guid?
62-
value: ((autoscaler_cf_server_xfcc_valid_space_guid))
63-
64-
65-
- type: replace
66-
path: /instance_groups/name=scalingengine/jobs/name=scalingengine/properties/autoscaler/scalingengine/cf_server?/port?
67-
value: &scalingEngineCfPort 6205
48+
path: /instance_groups/name=apiserver
6849

69-
- type: replace
70-
path: /instance_groups/name=postgres/jobs/name=route_registrar/properties/route_registrar/routes/-
71-
value:
72-
name: ((deployment_name))-cf-scalingengine
73-
registration_interval: 20s
74-
port: *scalingEngineCfPort
75-
tags:
76-
component: autoscaler_cf_scalingengine
77-
uris:
78-
- ((deployment_name))-cf-scalingengine.((system_domain))
79-
80-
## EVENTGENERATOR - Enable cf Server to receive calls from api running on cf --
81-
82-
- type: replace
83-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_org_guid?
84-
value: ((autoscaler_cf_server_xfcc_valid_org_guid))
85-
86-
- type: replace
87-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_space_guid?
88-
value: ((autoscaler_cf_server_xfcc_valid_space_guid))
8950

51+
## METRICSFORWARDER - remove metricsforwarder
52+
- type: remove
53+
path: /instance_groups/name=metricsforwarder
9054

91-
- type: replace
92-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/port?
93-
value: &EventGeneratorCfPort 6205
9455

95-
- type: replace
96-
path: /instance_groups/name=postgres/jobs/name=route_registrar/properties/route_registrar/routes/-
97-
value:
98-
name: ((deployment_name))-cf-eventgenerator
99-
registration_interval: 20s
100-
port: *EventGeneratorCfPort
101-
tags:
102-
component: autoscaler_cf_eventgenerator
103-
uris:
104-
- ((deployment_name))-cf-eventgenerator.((system_domain))

Diff for: ‎spec/jobs/eventgenerator/eventgenerator_spec.rb

+11-6
Original file line numberDiff line numberDiff line change
@@ -85,14 +85,19 @@
8585
end
8686

8787
it "defaults xfcc valid org and space " do
88-
properties["autoscaler"]["cf_server"] = {}
89-
properties["autoscaler"]["cf_server"]["xfcc"] = {
90-
"valid_org_guid" => "some-valid-org-guid",
91-
"valid_space_guid" => "some-valid-space-guid"
88+
properties["autoscaler"]["eventgenerator"] = {
89+
"cf_server" => {
90+
"xfcc" => {
91+
"valid_org_guid" => "some-valid-org-guid",
92+
"valid_space_guid" => "some-valid-space-guid"
93+
}
94+
}
9295
}
9396

94-
expect(rendered_template["cf_server"]["xfcc"]["valid_org_guid"]).to eq(properties["autoscaler"]["cf_server"]["xfcc"]["valid_org_guid"])
95-
expect(rendered_template["cf_server"]["xfcc"]["valid_space_guid"]).to eq(properties["autoscaler"]["cf_server"]["xfcc"]["valid_space_guid"])
97+
expect(rendered_template["cf_server"]["xfcc"]).to include({
98+
"valid_org_guid" => "some-valid-org-guid",
99+
"valid_space_guid" => "some-valid-space-guid"
100+
})
96101
end
97102
end
98103

0 commit comments

Comments
 (0)