Migrating workload after resource migration
This guide will help you complete the migration of your workload after VK Cloud employees have completed the resource migration work of cluster resources between availability zones. You will perform the following steps:
- Determine whether your workload depends on the old availability zone, since in that case your workload metadata will continue to reference it.
- Adapt the workload on the already migrated worker node groups to the new availability zone.
- Migrate the stateful workload to new PVCs.
- If the availability zone binding was set in Helm charts or when configuring add-ons, update these values to the new availability zone.
-
Go to the Kubernetes cluster whose resources have been migrated to another availability zone.
-
Install and configure
kubectl, if you have not already done so. -
Connect to the cluster using
kubectl. -
(Optional) Install the
rename-pvcutility for renaming PVCs.
Determine whether the migration will affect the workload of your clusters. To do this, find out whether the availability zone is specified:
-
in the pod's scheduling rules:
- in the
nodeAffinity,topologySpreadConstraints, ornodeSelectorparameters; - in the labels
topology.kubernetes.io/zone,failure-domain.beta.kubernetes.io/zone, andtopology.cinder.csi.openstack.org/zone.
- in the
-
in the storage class (
StorageClass) for StatefulSet pods.
If the availability zone is not specified, migrating to a different availability zone will not affect your workload in any way, and no additional migration actions will be required from you.
-
Wait for the VK Cloud resource migration work to be completed.
Nodes of the migrated worker node groups (i.e., those created before the migration) will have the following labels pointing to the old availability zone:
topology.kubernetes.io/zone,failure-domain.beta.kubernetes.io/zone.
These labels need to be updated, since the node itself will physically be located in the new availability zone.
The
topology.cinder.csi.openstack.org/zonelabel does not need to be updated: the new availability zone will be set in it for nodes created both before and after the migration. -
Update the availability zones in the labels:
For individual nodesFor all nodeskubectl label node ≠ \topology.kubernetes.io/zone=<NEW_AVAILABILITY_ZONE> \failure-domain.beta.kubernetes.io/zone=<NEW_AVAILABILITY_ZONE> \--overwriteHere
<NEW_AVAILABILITY_ZONE>is the availability zone after the migration.
Since the resource migration work by VK Cloud has already been completed, the disks on which the PVs are located have already physically changed their zone. Now you need to update their metadata in the cluster.
To migrate the stateful workload to a PVC in the new availability zone,
To prevent accidental deletion of data from the PVC when migrating the workload:
-
Change the PV's reclaim policy to
Retain:pv_name=$(kubectl get pvc <PVC_NAME> -n <PVC_NAMESPACE> -o jsonpath='{.spec.volumeName}')kubectl patch pv "$pv_name" -n <PVC_NAMESPACE> -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'Here:
<PVC_NAME>— the name of the PVC you want to migrate.<PVC_NAMESPACE>— the namespace the PVC is located in.
-
Clone the disk associated with the PVC.
-
(Optional) Keep the old PVC until the migration work is complete by renaming it to
<PVC_NAME>-old, so that you can revert to it if needed.When you create the new PVC, use
<PVC_NAME>— this way it will receive the name of the original migrated one.You can rename the PVC manually or using the
rename-pvcutility. Example command forrename-pvc:rename-pvc -n <PVC_NAMESPACE> <OLD_PVC_NAME> <NEW_PVC_NAME>
Several PVC migration methods are available:
- using a disk snapshot (via the VK Cloud management console or via the Cinder CSI plugin);
- using the
pv-migrateutility; - using the disk ID.
-
Go to the VK Cloud management console.
-
Create a disk snapshot associated with the PVC.
-
Create a new disk based on this snapshot, specifying the following parameters:
- Source: the disk snapshot created in the previous step;
- Availability zone: the availability zone you are migrating to.
- Specify the remaining parameters as you prefer.
-
Wait for the disk to be created and remember its ID.
-
Create a PV referencing the ID of the disk created from the snapshot, and a PVC based on it:
-
Wait for the created PV to reach the
Boundstatus. -
Migrate the corresponding workload to the new PVC.
-
Start the workload and make sure it works correctly after the migration.
-
Check whether the binding to availability zones was additionally set through Helm charts or when editing the addon code in the VK Cloud management console.
-
If so, manually specify the availability zone the migration was performed to for them. This is needed so that the migration results are not overwritten the next time the manifests are updated.
Helm chartsAddons-
Specify the availability zone the migration was performed to in
values.yaml:topology:zone: <AVAILABILITY_ZONE> -
Apply the update. For more information, see the official Helm documentation.
-
If you manage cluster resources through Terraform, synchronize the Terraform State with the actual infrastructure after migrating the cluster:
-
Add a
lifecycleblock to thevkcs_kubernetes_clustercluster resource:lifecycle {prevent_destroy = true}This will prevent the cluster from being recreated or deleted if unsynchronized fields remain in the configuration file or the Terraform State.
-
Bind the cluster to the
vkcs_kubernetes_clusterresource, using the data from the Terraform configuration file:terraform import vkcs_kubernetes_cluster.<RESOURCE_NAME> <CLUSTER_ID> -
Bind each node group to the
vkcs_kubernetes_node_groupresource:terraform import vkcs_kubernetes_node_group.<RESOURCE_NAME> <NODE_GROUP_ID> -
Check which parameters appeared on the new resources, and add them to the Terraform configuration file:
terraform state show <RESOURCE>.<RESOURCE_NAME> -
If autoscaling is enabled for the node group, the value of the
node_countparameter in the Terraform file may differ from the actually used number of nodes. Add alifecycleblock to the node group resource:lifecycle {ignore_changes = [node_count]}This way Terraform will not track changes to the
node_countparameter, which changes automatically due to autoscaling. -
Generate a changes file and make sure it does not contain any instructions to recreate resources:
terraform planIf the command output shows
No changes, the configuration file and the Terraform State are fully synchronized. If the output still shows changes, continue synchronizing until the command output showsNo changes.If the command output suggests resource recreation operations for the
availability_zoneoravailability_zonesfields:-
Find out the current value of the field in the configuration using the command:
terraform state show <RESOURCE>.<RESOURCE_NAME> -
Update the value in the Terraform configuration file.
-
-
Remove the previously added
lifecycleblock with theprevent_destroy = trueparameter from the cluster resource.
For more information, see the Using Terraform section and the Terraform provider documentation.
Running resources in the cluster are charged for and consume compute resources. If you do not plan to use the PVCs and disk snapshots left after the migration, or the availability zone itself, delete them and their associated resources:
-
Delete the old PVC and the disk snapshots.
-
Delete the storage class (StorageClass) of the old availability zone:
kubectl get sc -o name | grep <OLD_AVAILABILITY_ZONE> | xargs -I {} kubectl delete {}Here
<OLD_AVAILABILITY_ZONE>— the name of the availability zone the migration was performed from, specified in lowercase.