Immutable Infrastructure

Upgrading the global Cluster

This document describes how to upgrade a global cluster that runs on Immutable Infrastructure. Upgrades replace nodes with new Alauda OS images managed by the Cluster API provider; in-place node upgrades are not used.

For a Huawei DCS global cluster, review Alauda OS and Provider Compatibility before selecting the target DCS Provider package and Alauda OS image.

TOC

When to Use This PathTwo-Phase Upgrade OverviewCommon PrerequisitesProcedureStep 1 — Upgrade Kube-OVNStep 2 — Update the global Cluster ManifestStep 3 — Apply the Updated ManifestStep 4 — Monitor the Rolling ReplacementVerificationRecovery ConsiderationsNext Steps

When to Use This Path

Choose this upgrade path when:

  • The global cluster was originally installed on Immutable Infrastructure. See Installing the global Cluster.
  • Your infrastructure is one of the documented providers: Huawei DCS, VMware vSphere, Huawei Cloud Stack, or Bare Metal when provider v0.0.1 or later is included in the installed release.

For traditional-OS global clusters, use the standard upgrade path instead.

Two-Phase Upgrade Overview

Like workload clusters, the global cluster on Immutable Infrastructure follows a two-phase upgrade.

  1. Phase 1 — ACP Core and Distribution Version: prepare artifacts and plugin packages, then use CVO to move the global tier to the target Distribution Version. The ACP product documentation owns this procedure; see Pre-Upgrade Preparation and Upgrade the global cluster.
  2. Phase 2 — Kubernetes and OS Image: nodes are replaced with new Alauda OS images that contain the target Kubernetes version. This document focuses on Phase 2 for the global cluster.
Phase 1 Compatibility

Before starting Phase 2, verify that every workload cluster falls within the Compatible Versions matrix of the target Distribution Version. Workload clusters that are out of range must be upgraded first.

Common Prerequisites

  • The global cluster has completed Phase 1 (Distribution Version upgrade).
  • An etcd backup of the global cluster has been taken and verified.
  • The target row in OS Support Matrix is available, and its Alauda OS image, Kubernetes version, CoreDNS tag, etcd tag, and Kube-OVN chart are staged.
  • The previous machine templates and bootstrap templates are retained until Phase 2 is verified.
  • A maintenance window plan that accounts for rolling control plane replacement.
  • For cross-version upgrades that span more than one Kubernetes minor, the intermediate-version Core images and OS images are pre-staged. See Cross-Version Upgrade Preparation.

Procedure

After installation, the Cluster API controllers that manage the global cluster run on the global cluster itself. Use the global kubeconfig for the kubectl commands in this procedure.

Required when the target is Kubernetes 1.35

In Kubernetes 1.35, kubelet credential verification can apply to images that are already present on a node. On the hop to Kubernetes 1.35 or later, add imagePullCredentialsVerificationPolicy: NeverVerify to the kubelet patch used by every replacement node. Update the control-plane file in the same KubeadmControlPlane edit as spec.version. For workers, create a new KubeadmConfigTemplate with the updated patch and switch MachineDeployment.spec.template.spec.bootstrap.configRef.name in the same edit as the version. Do not add this parameter for Kubernetes 1.34 or earlier. See Required kubelet patch for Kubernetes 1.35.

Step 1 — Upgrade Kube-OVN

Read the kube-ovn (chart) value from the target OS Support Matrix row, then follow the shared provider-version-specific Kube-OVN procedure with <cluster-name> set to global.

That procedure first checks the installed provider revision, then selects the supported new or legacy flow, handles the Kube-OVN chart-name boundary at v4.4, and verifies the chart name, target revision, installed revision, phase, and conditions. Use the global kubeconfig for both the provider and cni-kube-ovn AppRelease checks. Do not begin control-plane replacement until all shared verification criteria pass.

Step 2 — Update the global Cluster Manifest

Update the Cluster API manifests of the global cluster to reference the new Alauda OS image and Kubernetes version. The manifest fields to update are provider-specific.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

For DCS, create new immutable infrastructure templates instead of editing templates that are already referenced by running machines.

Update the control plane resources:

  • Create a new DCSMachineTemplate for the target image and set spec.template.spec.vmTemplateName to the Alauda OS template that matches the target Kubernetes version.
  • Keep preserved node-local data, including /var/cpaas, in DCSIpHostnamePool.spec.pool[].persistentDisk. Do not move preserved disks back into DCSMachineTemplate.
  • Set KubeadmControlPlane.spec.version to the target Kubernetes version.
  • Set the CoreDNS and etcd image tags in KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration from the same OS Support Matrix row.
  • When the target is Kubernetes 1.35 or later, update the control-plane kubelet patch in KubeadmControlPlane.spec.kubeadmConfigSpec.files in the same manifest edit.
  • Point KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name to the new DCSMachineTemplate.
  • Keep KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge: 0 when the cluster uses pool-managed persistent disks.

Update worker node resources:

  • Create a new worker DCSMachineTemplate with the target vmTemplateName.
  • Set each MachineDeployment.spec.template.spec.version to the target Kubernetes version.
  • When the target is Kubernetes 1.35 or later, create a new worker KubeadmConfigTemplate with the required kubelet patch and point MachineDeployment.spec.template.spec.bootstrap.configRef.name to it.
  • Point each MachineDeployment.spec.template.spec.infrastructureRef.name to the new worker DCSMachineTemplate.
  • Keep each MachineDeployment.spec.strategy.rollingUpdate.maxSurge: 0 when the worker pool uses pool-managed persistent disks.

Pool-managed persistent disks are declared on the IP pool, not on the machine template:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: <global-pool-name>
  namespace: cpaas-system
spec:
  pool:
    - ip: <node-ip>
      hostname: <node-hostname>
      persistentDisk:
        - slot: 0
          quantityGB: 40
          datastoreName: <datastore-name>
          path: /var/cpaas
          format: xfs
          mountOptions:
            - defaults

Use isThin only when the DCS environment requires an explicit thin-provisioning value. If omitted, the provider does not send isThin and DCS uses the platform default. New persistent volumes are created as independent persistent normal volumes.

Use the IP pool status to confirm that preserved disks are detached from old VMs and attached to replacement VMs during the rolling replacement.

Step 3 — Apply the Updated Manifest

Apply the updated manifest against the global cluster.

kubectl --kubeconfig <global-kubeconfig> apply -f <updated-manifest>

The Cluster API provider begins replacing control plane and worker nodes by using the new image. When maxSurge: 0 is set, each old node is drained and deleted before its replacement can reuse the same fixed identity, IP address, or preserved disk.

Step 4 — Monitor the Rolling Replacement

Watch the rolling replacement until all control plane and worker nodes have been replaced.

kubectl --kubeconfig <global-kubeconfig> get machines -A -o wide
kubectl --kubeconfig <global-kubeconfig> get kubeadmcontrolplane -A

The upgrade is complete when every Machine reports the new Kubernetes version and Phase: Running, and the KubeadmControlPlane reports Ready: True against the new version.

Verification

After the rolling replacement finishes, verify that the upgraded global cluster is healthy.

kubectl --kubeconfig <global-kubeconfig> get nodes -o wide
kubectl --kubeconfig <global-kubeconfig> get clusterversionshadow -o yaml
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system

All nodes must report the new Kubernetes version, the ClusterVersionShadow must reflect the target Distribution Version, and core platform pods must be Running.

Recovery Considerations

Do not treat a Kubernetes minor downgrade as an ordinary rollback. Choose the recovery path from the rollout stage:

  1. No target-version control-plane Machine has been created: restore the previous Kube-OVN state and the previous manifest values. This cancels the target rollout before a new control-plane data format is introduced.
  2. Only the machine template or OS image changed, and the Kubernetes minor did not change: point the controlling resource back to the previous template and let Cluster API perform another replacement rollout. Keep the Kubernetes minor unchanged.
  3. A control-plane Machine on the target Kubernetes minor has joined the cluster: do not patch Kubernetes, CoreDNS, or etcd back to the previous minor. Stop further rollout and repair forward on the target minor, or recover the global cluster from the verified pre-upgrade etcd backup or Global DR procedure.

If a target-minor control-plane Machine was created but never joined, restore healthy etcd quorum and determine whether the failed replacement can be removed safely before changing manifests.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

For DCS clusters that use pool-managed persistent disks, confirm disk state before rollback:

First, check DCSIpHostnamePool.status.persistentDiskStatus before deleting or recreating machines. Do not delete retained DCS volumes that are listed in DCSIpHostnamePool.spec.pool[].persistentDisk.

For stage 1, use the DCS rule in Restore Kube-OVN During Stage-1 Recovery; the action depends on the installed provider version. Keep maxSurge: 0 while returning to previous machine templates in a same-minor replacement recovery. If a target-minor control-plane machine has joined, retain the target baseline and use forward recovery or backup restore.

Next Steps