Migrating workloads before resource migration
This guide will help you prepare in advance for the migration of cluster resources between availability zones, which will be performed by VK Cloud staff. You will perform the following steps:
- Determine whether your workload depends on the current availability zone.
- Create a new worker node group in the availability zone to which the migration will take place, and move your workload there.
- Move stateful workloads to new PVCs.
- Wait for the resource migration work on the VK Cloud side to be completed.
- If zone affinity was set in Helm charts or when configuring add-ons, update these values to the new availability zone.
-
Determine which Kubernetes cluster needs to be moved to the new availability zone.
-
Install and configure
kubectl, if you have not already done so. -
Connect to the cluster using
kubectl. -
(Optional) Install the
rename-pvcutility for renaming PVCs.
Determine whether the migration will affect the workload of your clusters. To do this, find out whether an availability zone is specified:
-
in the pod's scheduling rules:
- in the
nodeAffinity,topologySpreadConstraints, ornodeSelectorparameters; - in the labels
topology.kubernetes.io/zone,failure-domain.beta.kubernetes.io/zone, andtopology.cinder.csi.openstack.org/zone.
- in the
-
in the storage class (
StorageClass) for StatefulSet pods.
If the availability zone is not specified, migrating to a different availability zone will not affect your workload in any way, and no additional migration actions will be required from you.
- Add a worker node group and specify the availability zone to which the migration is being performed for it.
- Gradually move your workload to this node group. For more details, see the official Kubernetes documentation.
- Delete the node group with the old availability zone or reduce the number of nodes in it to zero so that you can use it later.
Since the cluster resources have not been moved yet, the PVC used by the stateful workload is physically and metadata-wise located in the old availability zone.
To move a stateful workload to a PVC in the new availability zone:
To prevent accidental deletion of data from the PVC when migrating the workload:
-
Change the PV's reclaim policy to
Retain:pv_name=$(kubectl get pvc <PVC_NAME> -n <PVC_NAMESPACE> -o jsonpath='{.spec.volumeName}')kubectl patch pv "$pv_name" -n <PVC_NAMESPACE> -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'Here:
<PVC_NAME>— the name of the PVC you want to migrate.<PVC_NAMESPACE>— the namespace the PVC is located in.
-
Clone the disk associated with the PVC.
-
(Optional) Keep the old PVC until the migration work is complete by renaming it to
<PVC_NAME>-old, so that you can revert to it if needed.When you create the new PVC, use
<PVC_NAME>— this way it will receive the name of the original migrated one.You can rename the PVC manually or using the
rename-pvcutility. Example command forrename-pvc:rename-pvc -n <PVC_NAMESPACE> <OLD_PVC_NAME> <NEW_PVC_NAME>
Several PVC migration methods are available:
- using a disk snapshot (via the VK Cloud management console or via the Cinder CSI plugin);
- using the
pv-migrateutility.
-
Go to the VK Cloud management console.
-
Create a disk snapshot associated with the PVC.
-
Create a new disk based on this snapshot, specifying the following parameters:
- Source: the disk snapshot created in the previous step;
- Availability zone: the availability zone you are migrating to.
- Specify the remaining parameters as you prefer.
-
Wait for the disk to be created and remember its ID.
-
Create a PV referencing the ID of the disk created from the snapshot, and a PVC based on it:
-
Wait for the created PV to reach the
Boundstatus. -
Migrate the corresponding workload to the new PVC.
-
Start the workload and make sure it works correctly after the migration.
- Wait for the resource migration work on the VK Cloud side to be completed.
- Start the workload and make sure that it works correctly after the migration.
-
Check whether the binding to availability zones was additionally set through Helm charts or when editing the addon code in the VK Cloud management console.
-
If so, manually specify the availability zone the migration was performed to for them. This is needed so that the migration results are not overwritten the next time the manifests are updated.
Helm chartsAddons-
Specify the availability zone the migration was performed to in
values.yaml:topology:zone: <AVAILABILITY_ZONE> -
Apply the update. For more information, see the official Helm documentation.
-
If you manage cluster resources through Terraform, synchronize the Terraform State with the actual infrastructure after migrating the cluster:
-
Add a
lifecycleblock to thevkcs_kubernetes_cluster_v2cluster resource:lifecycle {prevent_destroy = true}This will prevent the cluster from being recreated or deleted if unsynchronized fields remain in the configuration file or the Terraform State.
-
Bind the cluster to the
vkcs_kubernetes_cluster_v2resource, using the data from the Terraform configuration file:terraform import vkcs_kubernetes_cluster_v2.<RESOURCE_NAME> <CLUSTER_ID> -
Bind each node group to the
vkcs_kubernetes_node_group_v2resource:terraform import vkcs_kubernetes_node_group_v2.<RESOURCE_NAME> <NODE_GROUP_ID> -
Check which parameters appeared on the new resources, and add them to the Terraform configuration file:
terraform state show <RESOURCE>.<RESOURCE_NAME> -
If autoscaling is enabled for the node group, the value of the
node_countparameter in the Terraform file may differ from the actually used number of nodes. Add alifecycleblock to the node group resource:lifecycle {ignore_changes = [node_count]}This way Terraform will not track changes to the
node_countparameter, which changes automatically due to autoscaling. -
Generate a changes file and make sure it does not contain any instructions to recreate resources:
terraform planIf the command output shows
No changes, the configuration file and the Terraform State are fully synchronized. If the output still shows changes, continue synchronizing until the command output showsNo changes.If the command output suggests resource recreation operations for the
availability_zoneoravailability_zonesfields:-
Find out the current value of the field in the configuration using the command:
terraform state show <RESOURCE>.<RESOURCE_NAME> -
Update the value in the Terraform configuration file.
-
-
Remove the previously added
lifecycleblock with theprevent_destroy = trueparameter from the cluster resource.
For more information, see the Using Terraform section and the Terraform provider documentation.
Running resources in the cluster are charged for and consume compute resources. If you do not plan to use the PVCs and disk snapshots left after the migration, or the availability zone itself, delete them and their associated resources:
-
Delete the old PVC and the disk snapshots.
-
Delete the storage class (StorageClass) of the old availability zone:
kubectl get sc -o name | grep <OLD_AVAILABILITY_ZONE> | xargs -I {} kubectl delete {}Here
<OLD_AVAILABILITY_ZONE>— the name of the availability zone the migration was performed from, specified in lowercase.