HCIDOCS-188.1: Explains host networking when scaling a UPI cluster with the BMO..

johnwilkins · johnwilkins · commit 6e61cd870d0f · 2025-01-08T08:17:42.000-08:00
diff --git a/installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc b/installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
@@ -6,14 +6,25 @@ include::_attributes/common-attributes.adoc[]
 
 toc::[]
 
-After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal3 components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.
+After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal^3^ components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.
 
 include::modules/about-scaling-a-user-provisioned-installation-with-the-bare-metal-operator.adoc[leveloffset=+1]
 include::modules/upi-prerequisites-for-scaling-a-upi-cluster.adoc[leveloffset=+2]
 include::modules/upi-limitations-for-scaling-a-upi-cluster.adoc[leveloffset=+2]
 include::modules/configuring-a-provisioning-resource-to-scale-user-provisioned-clusters.adoc[leveloffset=+1]
 include::modules/upi-provisioning-new-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
-include::modules/upi-managing-existing-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
-include::modules/upi-removing-hosts-from-a-upi-cluster.adoc[leveloffset=+1]
 
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../../installing/installing_bare_metal/ipi/ipi-install-expanding-the-cluster.adoc#preparing-the-bare-metal-node_ipi-install-expanding[Preparing the bare-metal node]
+
+* xref:../../../installing/installing_bare_metal/ipi/ipi-install-installation-workflow.adoc#root-device-hints_ipi-install-installation-workflow[Root device hints]
+
+* xref:../../../installing/installing_bare_metal/ipi/ipi-install-expanding-the-cluster.adoc#ipi-install-diagnosing-duplicate-mac-address_ipi-install-expanding[Diagnosing a duplicate MAC address when provisioning a new host in the cluster]
+
+// Managing existing hosts in a user-provisioned cluster by using the BMO
+include::modules/upi-managing-existing-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
 
+// Removing hosts from a user-provisioned cluster by using the BMO
+include::modules/upi-removing-hosts-from-a-upi-cluster.adoc[leveloffset=+1]
diff --git a/modules/about-scaling-a-user-provisioned-installation-with-the-bare-metal-operator.adoc b/modules/about-scaling-a-user-provisioned-installation-with-the-bare-metal-operator.adoc
@@ -7,4 +7,4 @@
 [id="about-scaling-a-user-provisioned-cluster-with-the-bare-metal-operator_{context}"]
 = About scaling a user-provisioned cluster with the Bare Metal Operator
 
-You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal hosts in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.
+You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal^3^ components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal nodes in a cluster. However, it is possible to use the BMO and other metal^3^ components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.
diff --git a/modules/upi-provisioning-new-hosts-in-a-upi-cluster.adoc b/modules/upi-provisioning-new-hosts-in-a-upi-cluster.adoc
@@ -1,6 +1,6 @@
 // Module included in the following assemblies:
 //
-// * installing/installing_bare_metal/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
+// * installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
 :_mod-docs-content-type: PROCEDURE
 [id="upi-provisioning-new-hosts-in-a-upi-cluster_{context}"]
 = Provisioning new hosts in a user-provisioned cluster by using the BMO
@@ -9,7 +9,7 @@ You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a use
 
 [NOTE]
 ====
-To provision bare-metal hosts to the cluster by using the BMO, you must set the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false`.
+Provisioning bare-metal hosts to the cluster by using the BMO sets the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false` by default. Do not set the `spec.externallyProvisioned` specification to `true`, because this setting results in unexpected behavior.
 ====
 
 .Prerequisites
@@ -20,45 +20,153 @@ To provision bare-metal hosts to the cluster by using the BMO, you must set the
 
 .Procedure
 
-. Create the `Secret` CR and the `BareMetalHost` CR.
+. Create a configuration file for the bare-metal node. Depending if you use either a static configuration or a DHCP server, choose one of the following example `bmh.yaml` files and configure it to your needs by replacing values in the YAML to match your environment:
 
-.. Save the following YAML in the `bmh.yaml` file:
+  * To deploy with a static configuration, create the following `bmh.yaml` file:
 +
 [source,yaml]
 ----
 ---
 apiVersion: v1
 kind: Secret
 metadata:
-  name: worker1-bmc
+  name: openshift-worker-<num>-network-config-secret <1>
+  namespace: openshift-machine-api
+type: Opaque
+stringData:
+  nmstate: | <2>
+    interfaces: <3>
+    - name: <nic1_name> <4>
+      type: ethernet
+      state: up
+      ipv4:
+        address:
+        - ip: <ip_address> <5>
+          prefix-length: 24
+        enabled: true
+    dns-resolver:
+      config:
+        server:
+        - <dns_ip_address> <6>
+    routes:
+      config:
+      - destination: 0.0.0.0/0
+        next-hop-address: <next_hop_ip_address> <7>
+        next-hop-interface: <next_hop_nic1_name> <8>
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: openshift-worker-<num>-bmc-secret
   namespace: openshift-machine-api
 type: Opaque
 data:
-  username: <base64_of_uid>
+  username: <base64_of_uid> <9>
   password: <base64_of_pwd>
 ---
 apiVersion: metal3.io/v1alpha1
 kind: BareMetalHost
 metadata:
-  name: worker1
+  name: openshift-worker-<num>
   namespace: openshift-machine-api
 spec:
+  online: true
+  bootMACAddress: <nic1_mac_address> <10>
   bmc:
-    address: <protocol>://<bmc_url> <1>
-    credentialsName: "worker1-bmc"
-  bootMACAddress: <nic1_mac_address>
-  externallyProvisioned: false <2>
+    address: <protocol>://<bmc_url> <11>
+    credentialsName: openshift-worker-<num>-bmc-secret
+    disableCertificateVerification: false
   customDeploy:
     method: install_coreos
-  online: true
   userData:
     name: worker-user-data-managed
     namespace: openshift-machine-api
+  rootDeviceHints:
+    deviceName: <root_device_hint> <12>
+  preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret
+----
+<1> Replace all instances of `<num>` with a unique compute node number for the bare-metal nodes in the `name`, `credentialsName`, and `preprovisioningNetworkDataName` fields.
+<2> Add the NMState YAML syntax to configure the host interfaces. To configure the network interface for a newly created node, specify the name of the secret that has the network configuration. Follow the `nmstate` syntax to define the network configuration for your node. See "Preparing the bare-metal node" for details on configuring NMState syntax.
+<3> Optional: If you have configured the network interface with `nmstate`, and you want to disable an interface, set `state: up` with the IP addresses set to `enabled: false`.
+<4> Replace `<nic1_name>` with the name of the bare-metal node's first network interface controller (NIC).
+<5> Replace `<ip_address>` with the IP address of the bare-metal node's NIC.
+<6> Replace `<dns_ip_address>` with the IP address of the bare-metal node's DNS resolver.
+<7> Replace `<next_hop_ip_address>` with the IP address of the bare-metal node's external gateway.
+<8> Replace `<next_hop_nic1_name>` with the name of the bare-metal node's external gateway.
+<9> Replace `<base64_of_uid>` and `<base64_of_pwd>` with the base64 string of the user name and password.
+<10> Replace `<nic1_mac_address>` with the MAC address of the bare-metal node's first NIC. See the "BMC addressing" section for additional BMC configuration options.
+<11> Replace `<protocol>` with the BMC protocol, such as IPMI, Redfish, or others. Replace `<bmc_url>` with the URL of the bare-metal node's baseboard management controller.
+<12> Optional: Replace `<root_device_hint>` with a device path when specifying a root device hint. See "Root device hints" for additional details.
+
+  * When configuring the network interface with a static configuration by using `nmstate`, set `state: up` with the IP addresses set to `enabled: false`:
++
+[source,yaml]
+----
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: openshift-worker-<num>-network-config-secret
+  namespace: openshift-machine-api
+ # ...
+interfaces:
+  - name: <nic_name>
+    type: ethernet
+    state: up
+    ipv4:
+      enabled: false
+    ipv6:
+      enabled: false
+# ...
 ----
-<1> You can only use bare-metal host drivers that support virtual media networking booting, for example `redfish-virtualmedia` and `idrac-virtualmedia`.
-<2> You must set the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false`. The default value is `false`.
 
-. Create the bare-metal host object by running the following command:
+  * To deploy with a DHCP configuration, create the following `bmh.yaml` file:
++
+[source,yaml]
+----
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: openshift-worker-<num>-bmc-secret <1>
+  namespace: openshift-machine-api
+type: Opaque
+data:
+  username: <base64_of_uid> <2>
+  password: <base64_of_pwd>
+---
+apiVersion: metal3.io/v1alpha1
+kind: BareMetalHost
+metadata:
+  name: openshift-worker-<num>
+  namespace: openshift-machine-api
+spec:
+  online: true
+  bootMACAddress: <nic1_mac_address> <3>
+  bmc:
+    address: <protocol>://<bmc_url> <4>
+    credentialsName: openshift-worker-<num>-bmc
+    disableCertificateVerification: false
+  customDeploy:
+    method: install_coreos
+  userData:
+    name: worker-user-data-managed
+    namespace: openshift-machine-api
+  rootDeviceHints:
+    deviceName: <root_device_hint> <5>
+----
+<1> Replace `<num>` with a unique compute node number for the bare-metal nodes in the `name` and `credentialsName` fields.
+<2> Replace `<base64_of_uid>` and `<base64_of_pwd>` with the base64 string of the user name and password.
+<3> Replace `<nic1_mac_address>` with the MAC address of the bare-metal node's first NIC. See the "BMC addressing" section for additional BMC configuration options.
+<4> Replace `<protocol>` with the BMC protocol, such as IPMI, Redfish, or others. Replace `<bmc_url>` with the URL of the bare-metal node's baseboard management controller.
+<5> Optional: Replace `<root_device_hint>` with a device path when specifying a root device hint. See "Root device hints" for additional details.
++
+[IMPORTANT]
+====
+If the MAC address of an existing bare-metal node matches the MAC address of the bare-metal host that you are attempting to provision, then the installation will fail. If the host enrollment, inspection, cleaning, or other steps fail, the Bare Metal Operator retries the installation continuously. See "Diagnosing a duplicate MAC address when provisioning a new host in the cluster" for additional details.
+====
+
+. Create the bare-metal node by running the following command:
 +
 [source,terminal]
 ----
@@ -68,27 +176,31 @@ $ oc create -f bmh.yaml
 .Example output
 [source,terminal]
 ----
-secret/worker1-bmc created
-baremetalhost.metal3.io/worker1 created
+secret/openshift-worker-<num>-network-config-secret created
+secret/openshift-worker-<num>-bmc-secret created
+baremetalhost.metal3.io/openshift-worker-<num> created
 ----
 
-. Approve all certificate signing requests (CSRs).
-
-.. Verify that the provisioning state of the host is `provisioned` by running the following command:
+. Inspect the bare-metal node by running the following command:
 +
 [source,terminal]
 ----
-$ oc get bmh -A
+$ oc -n openshift-machine-api get bmh openshift-worker-<num>
 ----
 +
+where:
++
+<num>:: Specifies the compute node number.
++
 .Example output
 [source,terminal]
 ----
-NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
-openshift-machine-api   controller1   externally provisioned              true             5m25s
-openshift-machine-api   worker1       provisioned                         true             4m45s
+NAME                    STATE       CONSUMER   ONLINE   ERROR
+openshift-worker-<num>  provisioned true
 ----
 
+. Approve all certificate signing requests (CSRs).
+
 .. Get the list of pending CSRs by running the following command:
 +
 [source,terminal]
@@ -132,5 +244,4 @@ $ oc get nodes
 NAME        STATUS   ROLES           AGE     VERSION
 app1        Ready    worker          47s     v1.24.0+dc5a2fd
 controller1 Ready    master,worker   2d22h   v1.24.0+dc5a2fd
-----
-
+----
diff --git a/modules/upi-removing-hosts-from-a-upi-cluster.adoc b/modules/upi-removing-hosts-from-a-upi-cluster.adoc
@@ -1,6 +1,6 @@
 // Module included in the following assemblies:
 //
-// * installing/installing_bare_metal/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
+// * installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
 :_mod-docs-content-type: PROCEDURE
 [id="upi-removing-hosts-from-a-upi-cluster_{context}"]
 = Removing hosts from a user-provisioned cluster by using the BMO
@@ -15,7 +15,7 @@ You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user
 
 .Procedure
 
-. Cordon and drain the host by running the following command:
+. Cordon and drain the node by running the following command:
 +
 [source,terminal]
 ----
@@ -74,6 +74,18 @@ openshift-machine-api   controller1   externally provisioned              true
 openshift-machine-api   worker1       deprovisioning                      true             57m
 ----
 
+. Delete the host by running the following command when the `BareMetalHost` state changes to `available`:
++
+[source,terminal]
+----
+$ oc delete bmh -n openshift-machine-api <bmh_name>
+----
++
+[NOTE]
+====
+You can run this step without having to edit the `BareMetalHost` CR. It might take some time for the `BareMetalHost` state to change from `deprovisioning` to `available`. 
+====
+
 . Delete the node by running the following command:
 +
 [source,terminal]
@@ -83,7 +95,7 @@ $ oc delete node <node_name>
 
 .Verification
 
-* Verify the node is deleted by running the following command:
+* Verify that you deleted the node by running the following command:
 +
 [source,terminal]
 ----
@@ -95,5 +107,4 @@ $ oc get nodes
 ----
 NAME          STATUS   ROLES           AGE     VERSION
 controller1   Ready    master,worker   2d23h   v1.24.0+dc5a2fd
-----
-
+----
