VK Cloud logo

Migrating Workload Before Resource Migration

This guide will help you prepare in advance for the migration of cluster resources between availability zones, which will be performed by VK Cloud staff. You will perform the following steps:

  1. Determine whether your workload depends on the current availability zone.
  2. Create a new worker node group in the availability zone to which the migration will take place, and move your workload there.
  3. Move your stateful workload to new PVCs.
  4. Wait for the resource migration work by VK Cloud to be completed.
  5. If the availability zone binding was set in Helm charts or when configuring add-ons, update these values to the new availability zone.

Preparatory steps

  1. Determine which Kubernetes cluster needs to be moved to a new availability zone.

  2. Install and configure kubectl, if you have not already done so.

  3. Connect to the cluster using kubectl.

  4. (Optional) Install the rename-pvc utility for renaming PVCs.

1. Determine whether the workload depends on the availability zone

Determine whether the migration will affect the workload of your clusters. To do this, find out whether the availability zone is specified:

  • in the pod's scheduling rules:

    • in the nodeAffinity, topologySpreadConstraints, or nodeSelector parameters;
    • in the labels topology.kubernetes.io/zone, failure-domain.beta.kubernetes.io/zone, and topology.cinder.csi.openstack.org/zone.
  • in the storage class (StorageClass) for StatefulSet pods.

If the availability zone is not specified, migrating to a different availability zone will not affect your workload in any way, and no additional migration actions will be required from you.

2. Move the workload to new node groups

  1. Add a worker node group and specify the availability zone to which the migration is being performed for it.
  2. Sequentially move your workload to this node group. For more details, see the official Kubernetes documentation.
  3. Delete the node group with the old availability zone or reduce the number of nodes in it to zero so that you can use it later.

3. Move the stateful workload to a new PVC

Since the cluster resources have not been moved yet, the PVC used by the stateful workload is physically and metadata-wise located in the old availability zone.

To move the stateful workload to a PVC in the new availability zone:

  1. Prepare the PVC.
  2. Move the workload using one of the proposed methods.

3.1. Prepare the PVC for migration

To prevent accidental deletion of data from the PVC when migrating the workload:

  1. Change the PV's reclaim policy to Retain:

    pv_name=$(kubectl get pvc <PVC_NAME> -n <PVC_NAMESPACE> -o jsonpath='{.spec.volumeName}') kubectl patch pv "$pv_name" -n <PVC_NAMESPACE> -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'

    Here:

    • <PVC_NAME> — the name of the PVC you want to migrate.
    • <PVC_NAMESPACE> — the namespace the PVC is located in.
  2. Clone the disk associated with the PVC.

  3. (Optional) Keep the old PVC until the migration work is complete by renaming it to <PVC_NAME>-old, so that you can revert to it if needed.

    When you create the new PVC, use <PVC_NAME> — this way it will receive the name of the original migrated one.

    You can rename the PVC manually or using the rename-pvc utility. Example command for rename-pvc:

    rename-pvc -n <PVC_NAMESPACE> <OLD_PVC_NAME> <NEW_PVC_NAME>

3.2. Choose a migration method

Several methods are available for migrating a PVC:

  • using a disk snapshot (via the VK Cloud management console or via the Cinder CSI plugin);
  • using the pv-migrate utility.
  1. Go to the VK Cloud management console.

  2. Create a disk snapshot associated with the PVC.

  3. Create a new disk based on this snapshot, specifying the following parameters:

    • Source: the disk snapshot created in the previous step;
    • Availability zone: the availability zone you are migrating to.
    • Specify the remaining parameters as you prefer.
  4. Wait for the disk to be created and remember its ID.

  5. Create a PV referencing the ID of the disk created from the snapshot, and a PVC based on it:

  6. Wait for the created PV to reach the Bound status.

  7. Migrate the corresponding workload to the new PVC.

  8. Start the workload and make sure it works correctly after the migration.

4. Complete the resource migration

  1. Wait for the resource migration work by VK Cloud to be completed.
  2. Start the workload and make sure that it works correctly after the migration.

5. Update Helm charts and add-on settings

  1. Check whether the binding to availability zones was additionally set through Helm charts or when editing the addon code in the VK Cloud management console.

  2. If so, manually specify the availability zone the migration was performed to for them. This is needed so that the migration results are not overwritten the next time the manifests are updated.

    1. Specify the availability zone the migration was performed to in values.yaml:

      topology:  zone: <AVAILABILITY_ZONE>
    2. Apply the update. For more information, see the official Helm documentation.

6. (Optional) Synchronize Terraform State

If you manage cluster resources through Terraform, synchronize the Terraform State with the actual infrastructure after migrating the cluster:

  1. Add a lifecycle block to the vkcs_kubernetes_cluster cluster resource:

    lifecycle {  prevent_destroy = true}

    This will prevent the cluster from being recreated or deleted if unsynchronized fields remain in the configuration file or the Terraform State.

  2. Bind the cluster to the vkcs_kubernetes_cluster resource, using the data from the Terraform configuration file:

    terraform import vkcs_kubernetes_cluster.<RESOURCE_NAME> <CLUSTER_ID>
  3. Bind each node group to the vkcs_kubernetes_node_group resource:

    terraform import vkcs_kubernetes_node_group.<RESOURCE_NAME> <NODE_GROUP_ID>
  4. Check which parameters appeared on the new resources, and add them to the Terraform configuration file:

    terraform state show <RESOURCE>.<RESOURCE_NAME>
  5. If autoscaling is enabled for the node group, the value of the node_count parameter in the Terraform file may differ from the actually used number of nodes. Add a lifecycle block to the node group resource:

    lifecycle {  ignore_changes = [node_count]}

    This way Terraform will not track changes to the node_count parameter, which changes automatically due to autoscaling.

  6. Generate a changes file and make sure it does not contain any instructions to recreate resources:

    terraform plan

    If the command output shows No changes, the configuration file and the Terraform State are fully synchronized. If the output still shows changes, continue synchronizing until the command output shows No changes.

    If the command output suggests resource recreation operations for the availability_zone or availability_zones fields:

    1. Find out the current value of the field in the configuration using the command:

      terraform state show <RESOURCE>.<RESOURCE_NAME>
    2. Update the value in the Terraform configuration file.

  7. Remove the previously added lifecycle block with the prevent_destroy = true parameter from the cluster resource.

For more information, see the Using Terraform section and the Terraform provider documentation.

Delete unused resources

Running resources in the cluster are charged for and consume compute resources. If you do not plan to use the PVCs and disk snapshots left after the migration, or the availability zone itself, delete them and their associated resources:

  1. Delete the old PVC and the disk snapshots.

  2. Delete the storage class (StorageClass) of the old availability zone:

    kubectl get sc -o name | grep <OLD_AVAILABILITY_ZONE> | xargs -I {} kubectl delete {}

    Here <OLD_AVAILABILITY_ZONE> — the name of the availability zone the migration was performed from, specified in lowercase.

Was this article helpful?