#Upgrading Clusters

This section provides instructions for upgrading Kubernetes clusters managed by Cluster API.

#Upgrade Overview

Cluster upgrades on Immutable Infrastructure follow a two-phase approach:

1. Phase 1: Upgrade ACP Core and the cluster Distribution Version through CVO.
2. Phase 2: Upgrade Kubernetes and replace nodes with the matching Alauda OS image.

#Distribution Version vs Kubernetes Version

An ACP minor release line can offer more than one validated Kubernetes target over its patch lifecycle. Use OS Support Matrix for the exact patch-specific Kubernetes, component, and Alauda OS image values offered for Phase 2.

#Prerequisites

• Complete Pre-Upgrade Preparation, including validation that the source ACP release can upgrade to the target Distribution Version.
• Complete the ACP Core and Distribution Version upgrade before starting Phase 2.
• Verify that the control plane is reachable and healthy.
• Verify that all nodes are in the Ready state.
• Take and verify the backups required by the provider-specific procedure.
• Confirm sufficient IP or machine-slot capacity for rolling replacement.

#Phase 1: Upgrade ACP Core and Distribution Version

Use the ACP product documentation as the single procedure owner for this phase:

• Upgrade Overview
• Pre-Upgrade Preparation
• Upgrade the global cluster
• Upgrade workload clusters

Follow that sequence to prepare artifacts and plugin packages, upgrade the global tier, and then move each workload cluster to the target Distribution Version through CVO. Do not use intermediate Kubernetes hops in Phase 2 to bypass an unsupported ACP Core upgrade path.

#Post-Upgrade Verification

After the Distribution Version upgrade completes:

• Verify that the cluster reports the target Distribution Version and CVO is no longer progressing or blocked.
• Confirm that the target row and all required intermediate rows exist in the OS Support Matrix.
• Proceed to Phase 2 only after the management-side Cluster API providers are healthy.

#Phase 2: Upgrading Kubernetes Version

After upgrading the Distribution Version, upgrade Kubernetes and the Alauda OS image through the provider-specific Cluster API procedure.

For Huawei DCS, Huawei Cloud Stack, VMware vSphere, and bare-metal clusters, upgrade the Kube-OVN chart for the current upgrade hop before changing KubeadmControlPlane. Wait until the workload cluster's cni-kube-ovn AppRelease reports the target installed revision and phase=Success, then upgrade the control plane, followed by worker nodes. Repeat this Kube-OVN-first ordering for every intermediate Kubernetes minor in a cross-version upgrade.

#Upgrade Kube-OVN Before the Control Plane

Use this section as the authoritative Kube-OVN procedure for workload and global clusters. For every Kubernetes and OS hop, read the kube-ovn (chart) version from the matching OS Support Matrix row, complete the provider-version-specific procedure below, and verify the resulting AppRelease before changing KubeadmControlPlane.

Run provider-version checks and Cluster or infrastructure-cluster commands against the management cluster. Run cni-kube-ovn AppRelease commands against the cluster being upgraded. When upgrading the global cluster, both resources are on the global cluster, so use the global kubeconfig for all commands.

#Identify the Installed Provider Version

Do not infer the provider version from the ACP Distribution Version. Inspect the provider AppRelease on the management cluster. The resource name can vary, so the following command finds the provider by chart name and prints the requested and installed revisions:

kubectl -n cpaas-system get apprelease \
  -o custom-columns='APPRELEASE:.metadata.name,CHART:.spec.source.charts[*].name,REQUESTED:.spec.source.charts[*].targetRevision,INSTALLED:.status.charts[*].installedRevision,PHASE:.status.charts[*].phase'

Find the row whose chart name ends with chart-cluster-api-provider-dcs, chart-cluster-api-provider-hcs, chart-cluster-api-provider-vsphere, or chart-cluster-api-provider-baremetal, as appropriate. Use the installed revision to select a tab. If the requested and installed revisions differ, or the provider chart is not in Success, finish or repair the provider upgrade before starting the cluster upgrade.

The Kube-OVN chart source also changes at chart version v4.4:

If the target OS Support Matrix row uses Kube-OVN chart v4.4 or later, upgrade the infrastructure provider to the full-source reconciliation version in the table before continuing. A manual Kube-OVN source patch is not a substitute for that provider upgrade because the same provider releases also reconcile the required CoreDNS and kube-proxy repository changes.

#Apply the Provider-Version-Specific Procedure

#Huawei DCS

Verify that the DCS infrastructure cluster uses Kube-OVN:

kubectl get dcscluster <cluster-name> -n cpaas-system \
  -o jsonpath='{.spec.networkType}{"\n"}'
# Expected output: kube-ovn

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

kubectl patch apprelease cni-kube-ovn -n cpaas-system --type='json' \
  -p='[{"op":"replace","path":"/spec/source/charts/0/targetRevision","value":"<kube-ovn-version-from-matrix>"}]'

#Huawei Cloud Stack

Verify that the HCS infrastructure cluster uses Kube-OVN:

kubectl get hcscluster <cluster-name> -n cpaas-system \
  -o jsonpath='{.spec.networkType}{"\n"}'
# Expected output: kube-ovn

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

kubectl patch apprelease cni-kube-ovn -n cpaas-system --type='json' \
  -p='[{"op":"replace","path":"/spec/source/charts/0/targetRevision","value":"<kube-ovn-version-from-matrix>"}]'

#VMware vSphere

Verify that the CAPI Cluster enables Kube-OVN reconciliation:

kubectl get cluster <cluster-name> -n cpaas-system \
  -o jsonpath='{.metadata.annotations.cpaas\.io/network-type}{"\n"}'
# Expected output: kube-ovn

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

#Bare Metal

This procedure applies to Bare Metal provider v0.0.1 and later. There is no earlier supported release flow to select.

Verify that the infrastructure cluster uses Kube-OVN, then update the target chart version on the CAPI Cluster:

kubectl get baremetalcluster <cluster-name> -n cpaas-system \
  -o jsonpath='{.spec.networkType}{"\n"}'
# Expected output: kube-ovn

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

The provider selects the chart name from the target version and reconciles the complete AppRelease source and specification. Do not patch the AppRelease source manually.

#Verify the Kube-OVN AppRelease

Run these commands against the cluster being upgraded:

kubectl get apprelease cni-kube-ovn -n cpaas-system \
  -o jsonpath='Chart: {.spec.source.charts[0].name}{"\n"}Target: {.spec.source.charts[0].targetRevision}{"\n"}Installed: {.status.charts.*.installedRevision}{"\n"}Phase: {.status.charts.*.phase}{"\n"}'

kubectl get apprelease cni-kube-ovn -n cpaas-system \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'

Before changing the control plane, verify all of the following:

• spec.source.charts[0].name matches the chart-name boundary above.
• targetRevision and installedRevision both match the OS Support Matrix row.
• phase is Success.
• The Sync and Health conditions are True.

The normal phase sequence is Upgrading → HealthChecking → Success. Do not continue based on installedRevision alone: it can change during HealthChecking, before Kube-OVN pods have been verified Ready. If the AppRelease reaches DownloadFailed, DeployFailed, or NotReady, stop and inspect its condition messages:

kubectl describe apprelease cni-kube-ovn -n cpaas-system

#Restore Kube-OVN During Stage-1 Recovery

Use this recovery only when no control-plane Machine on the target Kubernetes minor has been created. If a target-minor control-plane machine has joined, keep the target Kube-OVN baseline and use forward recovery or the supported backup or DR restore procedure.

Restore the previous chart version on the CAPI Cluster in the management cluster:

kubectl annotate cluster <cluster-name> -n cpaas-system \
  cpaas.io/kube-ovn-version=<previous-kube-ovn-version> --overwrite

• DCS v1.0.22+, HCS v1.0.4+, vSphere v1.0.16+, and Bare Metal v0.0.1+: restore the previous cpaas.io/kube-ovn-version annotation. The provider restores the matching chart name, revision, and complete specification. Do not patch the source manually.

• DCS v1.0.21 and earlier or HCS v1.0.3 and earlier, with a target earlier than chart v4.4: after restoring the annotation, patch the existing targetRevision back to the previous chart revision on the cluster being recovered:

  kubectl patch apprelease cni-kube-ovn -n cpaas-system --type='json' \
    -p='[{"op":"replace","path":"/spec/source/charts/0/targetRevision","value":"<previous-kube-ovn-version>"}]'

• vSphere v1.0.15 and earlier, with a target earlier than chart v4.4: restore the previous annotation only; the provider reconciles the complete legacy specification.

After restoring the previous target, repeat Verify the Kube-OVN AppRelease before changing any control-plane manifests.

#Required kubelet patch for Kubernetes 1.35

When an upgrade hop targets Kubernetes 1.35 or later, the kubelet patch file at /etc/kubernetes/patches/kubeletconfiguration0+strategic.json must include the following setting:

{
  "imagePullCredentialsVerificationPolicy": "NeverVerify"
}

In Kubernetes 1.35, kubelet credential verification can apply even when an image is already present on the node. NeverVerify tells the kubelet not to verify pull credentials for these pre-pulled images. Add this field only to kubelet configurations that will run on Kubernetes 1.35 or later; do not add it for Kubernetes 1.34 or earlier.

Apply this change only on the hop to Kubernetes 1.35 or later:

• For control-plane nodes, update the matching file in KubeadmControlPlane.spec.kubeadmConfigSpec.files in the same edit that changes spec.version to 1.35. Do not roll out the policy separately while the target version is still 1.34.
• For worker nodes, create a new immutable KubeadmConfigTemplate containing the updated file, then update MachineDeployment.spec.template.spec.bootstrap.configRef.name in the same edit as the target version and infrastructure template.
• If the file uses contentFrom.secret, verify that the referenced Secret key contains this parameter. An older version-specific Secret is not automatically suitable for the 1.35 hop.

For Kubernetes 1.34 or earlier, omit imagePullCredentialsVerificationPolicy. Add it only on the upgrade hop whose target is Kubernetes 1.35 or later.

See platform-specific guides:

• Huawei DCS
• Huawei Cloud Stack
• VMware vSphere
• Bare Metal

#Preparing for Cross-Version Upgrades

When the target ACP Distribution Version is more than one Kubernetes minor above the cluster's current version, additional preparation is required before starting Phase 2. Single-minor upgrades skip this section.

On Immutable Infrastructure, each Kubernetes minor is baked into a matching OS image, and each Kubernetes minor maps to a specific ACP version row in OS Support Matrix. The Kubernetes version therefore moves forward one minor at a time through the required intermediate OS images. Keep these Kubernetes and OS hops separate from the ACP Core upgrade path: the Distribution Version must follow every source-to-target hop offered by CVO, while Phase 2 can require additional intermediate rows only to advance Kubernetes and the OS image.

Before starting Phase 2, identify every intermediate Kubernetes minor and its matching OS Support Matrix row. Confirm separately that Phase 1 completed every required Distribution Version hop according to Pre-Upgrade Preparation.

#Step 1 — Confirm Core hops and pre-sync intermediate images

If CVO offers an intermediate Distribution Version as part of the supported Core path, perform the full ACP artifact preparation and Core/CVO upgrade for that hop during Phase 1. This is required for paths such as ACP 4.0.x to 4.4, and --only-sync-image does not replace it. Verify availableUpdates and VersionUpgradePath at every Core hop.

After all required Core hops are complete, some intermediate OS Support Matrix rows may still be needed only for the sequential Kubernetes and OS rollout in Phase 2. For each of these Kubernetes/OS-only rows, download the corresponding Core package and synchronize only its images into the registry. Do not request an additional Core upgrade solely because a row is used as a Kubernetes or OS stepping stone.

bash upgrade.sh --only-sync-image

See Sync upgrade artifacts for command options.

This places the intermediate-version CoreDNS, etcd, and Kube-OVN chart images in the registry so the staged Kubernetes rollout can pull them when KubeadmControlPlane is patched to the matching OS Support Matrix row. In an air-gapped environment, missing intermediate images prevent the new control plane components from starting.

For a required Core hop, prepare and upgrade Aligned plugins as directed by the ACP Phase 1 procedure. For a Kubernetes/OS-only intermediate row, Aligned plugins do not need an extra violet push; the Kube-OVN chart used by that intermediate Kubernetes hop is covered by the Core image sync above.

#Step 2 — Stage intermediate OS images for the provisioning mechanism

For each intermediate Kubernetes/OS row, make the matching OS image (the Alauda OS Image Version value from the OS Support Matrix row) available to the node provisioning mechanism so the Kubernetes step can replace nodes from each image in turn:

• IaaS providers (Huawei DCS, VMware vSphere, Huawei Cloud Stack): upload the OS image and register it as a VM template (or VM image, depending on the platform) under the exact name listed in the OS Support Matrix row.
• Bare metal: make the OS image available to the node re-provisioning flow used by your cluster.

The control plane and worker rollouts then step through the intermediate OS images one at a time, matched to the staged Kubernetes minor, until the cluster reaches the target ACP version row.

#Step 3 — Apply the platform-specific procedure for each intermediate Kubernetes/OS row

Repeat the platform-specific Phase 2 procedure for each intermediate Kubernetes/OS row in turn, using that row in OS Support Matrix for Kube-OVN, KubeadmControlPlane, and MachineDeployment values. For each hop, upgrade Kube-OVN first and wait for phase=Success, then upgrade the control plane, followed by workers. On the hop to Kubernetes 1.35, also apply the required kubelet patch to the control-plane and worker bootstrap configurations. Complete the whole hop before starting the next one. After all intermediate hops succeed, apply the same procedure once more for the target row.

#Platform-Specific Instructions

Select your platform for detailed upgrade instructions:

#Related Topics

• Creating Clusters
• Managing Nodes
• Infrastructure Resources