Add longer delay after kill command

[johscheuer]

Description

Adding back the status request when killing fdbserver processes to ensure the reboot request was sent to all processes.

Type of change

Please select one of the options below.

• Bug fix (non-breaking change which fixes an issue)

Discussion

We had this check in an earlier version of the fdbclient but then removed it. This change adds it back again to ensure the reboot command is sent to all processes.

Testing

e2e test.

Documentation

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 9d2ab39379408448353abdbf1339e2ae6a9fb6ec
• Duration 1:44:20
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 9d2ab39379408448353abdbf1339e2ae6a9fb6ec
• Duration 1:49:45
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

[Container] 2023/03/07 15:20:44 Running command if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; export fail_test=true; fi
TESTS FAILED SEE THESE LOGS:

logs/test_operator_ha_upgrades.log

Failure:

• [FAILED] [881.392 seconds]
Operator HA Upgrades Upgrading a multi-DC cluster, with a random pod deleted during the staging phase [It] Upgrade, with a random pod deleted during the staging phase, from 6.3.25 to 7.1.27
/codebuild/output/src356759027/src/github.com/FoundationDB/fdb-kubernetes-operator/fdb-kubernetes-tests/tests/fixtures/command_line_flags.go:298

  [FAILED] Timed out after 600.001s.
  Expected
      <bool>: false
  to be true
  In [It] at: /codebuild/output/src356759027/src/github.com/FoundationDB/fdb-kubernetes-operator/fdb-kubernetes-tests/tests/test_operator_ha_upgrades/operator_ha_upgrade_test.go:317 @ 03/07/23 14:37:51.792
------------------------------

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 48548fd281a40bec63187873229de7799291c62c
• Duration 1:58:37
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c7f4bfa
• Duration 2:17:56
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c7f4bfa
• Duration 2:17:56
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

Had one failure in the operator test for converting the public IP source. I noticed that the exclude was blocked by ERROR: This exclude may cause the total free space in the cluster to drop below 10%. which doesn't make any sense since the excluded processes are all log + stateless.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c7f4bfa
• Duration 1:52:43
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)