Updates for OCP 3.4 cluster upgrades (automated + manual) #3352

adellape · 2016-12-08T23:25:46Z

Preview build:

http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html

Includes:

Updated "In-place or Blue-Green Upgrades" in the Overview topic with some more details
- http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/index.html#install-config-upgrading-type
New control plane vs node upgrade options
- Description: http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html#upgrading-control-plane-nodes-separate-phases
- Steps inline: http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html#upgrading-to-ocp-3-4
Mentions --tag pre_upgrade option in a Note box.
Customized node upgrades (serialization + label grouping)
- http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html#customizing-node-upgrades
etcd3 upgrade
- Inline for quick installer: http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html#upgrading-using-the-installation-utility-to-upgrade
- Inline for playbooks: http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/automated_upgrades.html#upgrading-to-ocp-3-4
Removes the "Upgrading to <version> Asynchronous Releases" section and consolidates it into the re-titled "Upgrading to the Latest OpenShift Container Platform 3.4 Release", as the former was going to become really redundant w/ all the control plane stuff getting added.
Starts a stub of the 3.4 release notes because I needed to link to it.

dgoodwin · 2016-12-09T12:30:42Z

install_config/upgrading/automated_upgrades.adoc

+$ ansible-playbook -i <path/to/inventory/file> \
+    </path/to/upgrade/playbook> \
+    -e openshift_upgrade_nodes_serial="2" \
+    -e openshift_upgrade_nodes_label="region=group1"


All looks spot on. 👍

dgoodwin · 2016-12-09T12:32:08Z

install_config/upgrading/automated_upgrades.adoc

+*Option A) Upgrade masters and nodes in a single phase.*
+
+Run the *_upgrade.yml_* playbook to upgrade the cluster in one phase: master
+components first, then nodes in-place:


May want to consider referring to this as the control plane, rather than master components. Not sure if now is a good time or sometime in future. Control plane matches the terminology used upstream.

@dgoodwin OK, "control plane" was defined previously in the earlier section, but now I've made sure all sections prefer the "control plane" term.

dgoodwin · 2016-12-09T12:32:51Z

install_config/upgrading/automated_upgrades.adoc

+When upgrading in multiple phases, the control plane upgrade phase includes:
+
+- master components
+- Docker only on any stand-alone etcd hosts


Etcd is upgraded during control plane upgrade as well right @sdodson ?

Not currently. I'd be ok with including it there too.

dgoodwin · 2016-12-09T12:35:47Z

install_config/upgrading/automated_upgrades.adoc

+
+- node services running on masters
+- Docker running on masters
+- node services running on stand-alone nodes


I have a pending TODO to bring this doc to your attention once it's published: https://docs.google.com/a/redhat.com/document/d/1Jv5ROqosiG2-WhdWEkwpXPmvKbRaUC5yiLEAnZnzkeg/edit?usp=sharing

The point was raised that we don't clearly state if it's possible to do a zero downtime upgrade or not, and if so what would be required in terms of infrastructure. I don't know if there's time now to look into this but hopefully we can get something into official docs (at some point) stating that you can do zero downtime upgrades if your app is capable, and you have sufficient replication for your app, extra nodes, and at least 3 infra nodes.

Also the description of the steps of upgrade might be useful.

sdodson · 2016-12-09T13:40:39Z

FYI, @mwoodson docs changes for 3.4 upgrade changes.

mwoodson · 2016-12-09T17:38:24Z

thanks

adellape · 2016-12-09T23:00:23Z

install_config/upgrading/automated_upgrades.adoc

+preforms all pre-upgrade checks without actually upgrading any hosts, and
+reports any problems found.
+====
+


@dgoodwin I moved the bit here about <customized_node_upgrade_variables> inline with the commands, and replaced it with a note about --tag pre_upgrade.

adellape · 2016-12-09T23:02:04Z

Setting this to ON_QA in https://bugzilla.redhat.com/show_bug.cgi?id=1383278.

@ahardin-rh PTAL for peer review?

ahardin-rh · 2016-12-09T23:40:40Z

release_notes/ocp_3_4_release_notes.adoc

+== Technology Preview Features
+
+Some features in this release are currently in Technology Preview. These
+experimental features are not intended for production use. Please note the


s/Please note/Note

ahardin-rh · 2016-12-09T23:41:25Z

release_notes/ocp_3_4_release_notes.adoc

+[[ocp-34-known-issues]]
+== Known Issues
+
+* Setting the `*forks*` parameter in the *_/etc/ansible/ansible.cfg_* file to 11


ahardin-rh · 2016-12-09T23:49:07Z

@adellape LGTM! Made some comments in the release notes, but realized that's a WIP. Everything else looks great.

adellape · 2016-12-12T23:18:00Z

Went ahead and updated manual cluster upgrade steps too here, per notes from https://bugzilla.redhat.com/show_bug.cgi?id=1383278#c5.

http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html

adellape · 2016-12-13T17:09:32Z

install_config/upgrading/manual_upgrades.adoc

 ----
-$ oc new-app -f metrics-deployer.yaml \
+$ oc new-app --as=system:serviceaccount:openshift-infra:metrics-deployer \
+    -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hm.example.com \
    -p MODE=refresh <1>
 ----
 <1> In the original deployment command, there was no `MODE=refresh`.


@mwringe Does --as=system:serviceaccount:openshift-infra:metrics-deployer belong here in the metrics upgrade command, like it was added to the metrics install doc per #3018? I've gone ahead and added it for now.

Please see also the first step added above (about the view role for the hawkular SA), per QE feedback in https://bugzilla.redhat.com/show_bug.cgi?id=1383278#c5.

Preview build:

http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html#manual-upgrading-cluster-metrics

Yes, the as serviceaccount option should be in there. Thanks for the catch.

adellape · 2016-12-13T18:50:14Z

release_notes/ocp_3_4_release_notes.adoc

+*Updated Elasticsearch in EFK Stack*
+
+The latest EFK stack now uses Elasticsearch 2.4 with a common data model. This
+means Fluentd sends logs to Elasticsearch with a new indexing pattern for


Confirmed with @ewolinetz that this should now be Elasticsearch 2.4 instead of 2.3 (which we originally had in this note per #3211, but has changed for OCP 3.4 since then).

Also FYI @danmacpherson @vikram-redhat I moved this note to here in the 3.4 release notes, instead of at the tail end of the logging upgrade section because it felt buried and wasn't specific to the upgrade task. Also fixed a rendering issue in this note due to some superfluous +s.

Thanks @adellape .

adellape · 2016-12-15T15:39:29Z

@dgoodwin I've added manual docker upgrade steps since you last looked. Can you review?

For masters (near the end):
http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html#upgrading-masters

For nodes (step 5):
http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html#upgrading-nodes

dgoodwin · 2016-12-15T16:00:26Z

Instructions for Docker upgrade seem right, however it's worth noting this would technically be doing it twice as all masters are implicitly nodes as well. In automated upgrade I only hit Docker during node upgrade phase, should we do the same here?

adellape · 2016-12-15T16:22:26Z

@dgoodwin There's a warning right before the Docker upgrade in master sorta to that effect:

The node component on masters is set by default to unschedulable status during initial installation, so that pods are not deployed to them. However, it is possible to set them schedulable during the initial installation or manually thereafter. If any of your masters are also configured as a schedulable node, skip the following Docker upgrade steps for those masters and instead run all steps described in Upgrading Nodes when you get to that section for those hosts as well.

So basically if it's a master that is unschedulable, the "Upgrading Masters" section handles everything (step 3 also has you upgrade node/openvswitch packages) and you don't have to also go through the "Upgrading Nodes" steps for that host. But if it is a master that is schedulable for some reason, then don't do the docker upgrade there and go through all of "Upgrading Nodes" as well for that host, where you'll do the Docker upgrade.

If that's too convoluted and you think all master hosts should just go through the "Upgrading Masters" and "Upgrading Nodes" sections, then the latter section will need to get re-written a bit cuz it currently assumes a schedulable node and talks about some stuff that would be superfluous for unschedulable masters (like setting --scheduable=false and evac'ing pods).

dgoodwin · 2016-12-15T16:23:26Z

Ah I understand, no that sounds fine to me.

Also a sign I might need to re-think my structure here with this yet again.

adellape · 2016-12-15T20:05:11Z

@sdodson Now that upgrade_etcd.yml is called with the normal upgrade playbooks, I've made some changes (currently a separate commit 015f600). PTAL:

Remove steps for manually running upgrade_etcd.yml (from both quick installer + ansible-playbook sections)
Add note about etcd 3 upgrade + v2 API backcompat to Release Notes: http://file.rdu.redhat.com/~adellape/120216/upgrade34/release_notes/ocp_3_4_release_notes.html#ocp-34-notable-technical-changes
Add manual etcd upgrade steps (RPM+containerized): http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html#upgrading-masters

Also @ahardin-rh for peer review of ^ and http://file.rdu.redhat.com/~adellape/120216/upgrade34/install_config/upgrading/manual_upgrades.html in general (I don't think I had modified that at all when you last looked in this PR).

adellape · 2016-12-15T21:26:03Z

@tdawson @sdodson I've also now added steps in the upgrade docs about the *-excluder packages. Currently in the following separate commit: 65b4e1a.

I'll add excluder steps to the normal install docs in a separate PR about 3.4 installs.

sdodson · 2016-12-15T21:28:20Z

release_notes/ocp_3_4_release_notes.adoc

+- etcd has been updated to 3.1.0-rc.0.
+
+While etcd has been updated from etcd 2 to 3, {product-title} 3.4 continues to
+use the etcd v2 API, which is backwards compatible with the etcd 3, for both new


I'd get rid 'which is backwards compatible with the etcd 3'

lol @ "the"

ahardin-rh · 2016-12-15T21:50:53Z

install_config/upgrading/automated_upgrades.adoc

+- Docker running on masters
+- node services running on stand-alone nodes
+
+When upgrading only the nodes, it is required that the control plane has already


This reads a little awkward. Maybe something like:
When upgrading only the nodes, the control plane must already be upgraded.

ahardin-rh · 2016-12-15T21:54:55Z

install_config/upgrading/blue_green_deployments.adoc

+The
+xref:../../install_config/upgrading/blue_green_deployments.adoc#upgrading-blue-green-deployments[blue-green deployment] upgrade method follows a similar flow to the in-place method:
+masters and etcd servers are still upgraded first, however a parallel
+environment is created for new nodes instead of upgrading them in-place.


This last instance of in-place should be "in place" (not "in-place") since it's not modifying a noun, like "in-place method" earlier in the paragraph

ahardin-rh · 2016-12-15T21:56:25Z

@adellape just 2 minor nits from me ⭐

sdodson · 2016-12-15T22:01:56Z

@adellape excluder bits look good to me

Also logging + cluster upgrade tweaks per QE feedback Add etcd v2 API note to release notes Add manual etcd upgrade steps (RPM+containerized) Add upgrade steps for excluder pkgs

adellape · 2016-12-15T22:21:01Z

Thanks @ahardin-rh @sdodson!

Squashed and merging.

adellape added the branch/enterprise-3.4 label Dec 8, 2016

adellape added this to the Future Release milestone Dec 8, 2016

dgoodwin reviewed Dec 9, 2016

View reviewed changes

adellape changed the title ~~[WIP] Updates for OCP 3.4 upgrade process~~ [WIP] Updates for OCP 3.4 automated upgrades Dec 9, 2016

adellape force-pushed the upgrade34 branch 2 times, most recently from a59a344 to 0a693d9 Compare December 9, 2016 19:57

adellape changed the title ~~[WIP] Updates for OCP 3.4 automated upgrades~~ Updates for OCP 3.4 automated upgrades Dec 9, 2016

adellape force-pushed the upgrade34 branch from 0a693d9 to dcf24c6 Compare December 9, 2016 22:56

adellape commented Dec 9, 2016

View reviewed changes

ahardin-rh reviewed Dec 9, 2016

View reviewed changes

adellape force-pushed the upgrade34 branch 2 times, most recently from b7fe3f9 to 7101d49 Compare December 12, 2016 23:17

adellape force-pushed the upgrade34 branch from 7101d49 to fe2fa19 Compare December 13, 2016 17:04

adellape commented Dec 13, 2016

View reviewed changes

adellape force-pushed the upgrade34 branch from fe2fa19 to 6258161 Compare December 13, 2016 17:59

adellape commented Dec 13, 2016

View reviewed changes

adellape changed the title ~~Updates for OCP 3.4 automated upgrades~~ Updates for OCP 3.4 cluster upgrades (automated + manual) Dec 13, 2016

adellape force-pushed the upgrade34 branch 3 times, most recently from 57ff90b to 4576c69 Compare December 15, 2016 15:16

adellape force-pushed the upgrade34 branch 2 times, most recently from 3d0354a to 015f600 Compare December 15, 2016 20:03

sdodson reviewed Dec 15, 2016

View reviewed changes

ahardin-rh reviewed Dec 15, 2016

View reviewed changes

Updates for OCP 3.4 cluster upgrade process (automated+manual in-place)

0888b96

Also logging + cluster upgrade tweaks per QE feedback Add etcd v2 API note to release notes Add manual etcd upgrade steps (RPM+containerized) Add upgrade steps for excluder pkgs

adellape force-pushed the upgrade34 branch from 65b4e1a to 0888b96 Compare December 15, 2016 22:19

adellape merged commit 75058c7 into openshift:master Dec 15, 2016

vikram-redhat modified the milestones: Future Release, Staging, OCP 3.4 GA Jan 16, 2017

adellape deleted the upgrade34 branch November 9, 2017 19:13

Updates for OCP 3.4 cluster upgrades (automated + manual) #3352

Updates for OCP 3.4 cluster upgrades (automated + manual) #3352

Uh oh!

Conversation

adellape commented Dec 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgoodwin Dec 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdodson commented Dec 9, 2016

Uh oh!

mwoodson commented Dec 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adellape commented Dec 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardin-rh commented Dec 9, 2016

Uh oh!

adellape commented Dec 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adellape commented Dec 15, 2016

Uh oh!

dgoodwin commented Dec 15, 2016

Uh oh!

adellape commented Dec 15, 2016

Uh oh!

dgoodwin commented Dec 15, 2016

Uh oh!

adellape commented Dec 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adellape commented Dec 15, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardin-rh commented Dec 15, 2016

Uh oh!

sdodson commented Dec 15, 2016

Uh oh!

adellape commented Dec 15, 2016

Uh oh!

Uh oh!

adellape commented Dec 8, 2016 •

edited

Loading

dgoodwin Dec 9, 2016 •

edited

Loading

adellape commented Dec 12, 2016 •

edited

Loading

adellape commented Dec 15, 2016 •

edited

Loading