VK Cloud logo

Migrating workload after resource migration

This guide will help you complete the migration of your workload after VK Cloud employees have completed the resource migration work of cluster resources between availability zones. You will perform the following steps:

  1. Determine whether your workload depends on the old availability zone, since in that case your workload metadata will continue to reference it.
  2. Adapt the workload on the already migrated worker node groups to the new availability zone.
  3. Migrate the stateful workload to new PVCs.
  4. If the availability zone binding was set in Helm charts or when configuring add-ons, update these values to the new availability zone.

Preparatory steps

  1. Go to the Kubernetes cluster whose resources have been migrated to another availability zone.

  2. Install and configure kubectl, if you have not already done so.

  3. Connect to the cluster using kubectl.

  4. (Optional) Install the rename-pvc utility for renaming PVCs.

1. Determine whether the workload depends on the availability zone

Determine whether the migration will affect the workload of your clusters. To do this, find out whether the availability zone is specified:

  • in the pod's scheduling rules:

    • in the nodeAffinity, topologySpreadConstraints, or nodeSelector parameters;
    • in the labels topology.kubernetes.io/zone, failure-domain.beta.kubernetes.io/zone, and topology.cinder.csi.openstack.org/zone.
  • in the storage class (StorageClass) for StatefulSet pods.

If the availability zone is not specified, migrating to a different availability zone will not affect your workload in any way, and no additional migration actions will be required from you.

2. Update the configurations of the migrated node groups

  1. Wait for the VK Cloud resource migration work to be completed.

    Nodes of the migrated worker node groups (i.e., those created before the migration) will have the following labels pointing to the old availability zone:

    • topology.kubernetes.io/zone,
    • failure-domain.beta.kubernetes.io/zone.

    These labels need to be updated, since the node itself will physically be located in the new availability zone.

    The topology.cinder.csi.openstack.org/zone label does not need to be updated: the new availability zone will be set in it for nodes created both before and after the migration.

  2. Update the availability zones in the labels:

    kubectl label node ≠ \       topology.kubernetes.io/zone=<NEW_AVAILABILITY_ZONE> \       failure-domain.beta.kubernetes.io/zone=<NEW_AVAILABILITY_ZONE> \       --overwrite

    Here <NEW_AVAILABILITY_ZONE> is the availability zone after the migration.

3. Migrate the stateful workload to a new PVC

Since the resource migration work by VK Cloud has already been completed, the disks on which the PVs are located have already physically changed their zone. Now you need to update their metadata in the cluster.

To migrate the stateful workload to a PVC in the new availability zone,

  1. Prepare the PVC.
  2. Migrate the workload using one of the suggested methods.

3.1. Prepare the PVC for migration

To prevent accidental deletion of data from the PVC when migrating the workload:

  1. Change the PV's reclaim policy to Retain:

    pv_name=$(kubectl get pvc <PVC_NAME> -n <PVC_NAMESPACE> -o jsonpath='{.spec.volumeName}') kubectl patch pv "$pv_name" -n <PVC_NAMESPACE> -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'

    Here:

    • <PVC_NAME> — the name of the PVC you want to migrate.
    • <PVC_NAMESPACE> — the namespace the PVC is located in.
  2. Clone the disk associated with the PVC.

  3. (Optional) Keep the old PVC until the migration work is complete by renaming it to <PVC_NAME>-old, so that you can revert to it if needed.

    When you create the new PVC, use <PVC_NAME> — this way it will receive the name of the original migrated one.

    You can rename the PVC manually or using the rename-pvc utility. Example command for rename-pvc:

    rename-pvc -n <PVC_NAMESPACE> <OLD_PVC_NAME> <NEW_PVC_NAME>

3.2. Choose a migration method

Several PVC migration methods are available:

  • using a disk snapshot (via the VK Cloud management console or via the Cinder CSI plugin);
  • using the pv-migrate utility;
  • using the disk ID.
  1. Go to the VK Cloud management console.

  2. Create a disk snapshot associated with the PVC.

  3. Create a new disk based on this snapshot, specifying the following parameters:

    • Source: the disk snapshot created in the previous step;
    • Availability zone: the availability zone you are migrating to.
    • Specify the remaining parameters as you prefer.
  4. Wait for the disk to be created and remember its ID.

  5. Create a PV referencing the ID of the disk created from the snapshot, and a PVC based on it:

  6. Wait for the created PV to reach the Bound status.

  7. Migrate the corresponding workload to the new PVC.

  8. Start the workload and make sure it works correctly after the migration.

4. Update Helm charts and add-on settings

  1. Check whether the binding to availability zones was additionally set through Helm charts or when editing the addon code in the VK Cloud management console.

  2. If so, manually specify the availability zone the migration was performed to for them. This is needed so that the migration results are not overwritten the next time the manifests are updated.

    1. Specify the availability zone the migration was performed to in values.yaml:

      topology:  zone: <AVAILABILITY_ZONE>
    2. Apply the update. For more information, see the official Helm documentation.

5. (Optional) Synchronize Terraform State

If you manage cluster resources through Terraform, synchronize the Terraform State with the actual infrastructure after migrating the cluster:

  1. Add a lifecycle block to the vkcs_kubernetes_cluster cluster resource:

    lifecycle {  prevent_destroy = true}

    This will prevent the cluster from being recreated or deleted if unsynchronized fields remain in the configuration file or the Terraform State.

  2. Bind the cluster to the vkcs_kubernetes_cluster resource, using the data from the Terraform configuration file:

    terraform import vkcs_kubernetes_cluster.<RESOURCE_NAME> <CLUSTER_ID>
  3. Bind each node group to the vkcs_kubernetes_node_group resource:

    terraform import vkcs_kubernetes_node_group.<RESOURCE_NAME> <NODE_GROUP_ID>
  4. Check which parameters appeared on the new resources, and add them to the Terraform configuration file:

    terraform state show <RESOURCE>.<RESOURCE_NAME>
  5. If autoscaling is enabled for the node group, the value of the node_count parameter in the Terraform file may differ from the actually used number of nodes. Add a lifecycle block to the node group resource:

    lifecycle {  ignore_changes = [node_count]}

    This way Terraform will not track changes to the node_count parameter, which changes automatically due to autoscaling.

  6. Generate a changes file and make sure it does not contain any instructions to recreate resources:

    terraform plan

    If the command output shows No changes, the configuration file and the Terraform State are fully synchronized. If the output still shows changes, continue synchronizing until the command output shows No changes.

    If the command output suggests resource recreation operations for the availability_zone or availability_zones fields:

    1. Find out the current value of the field in the configuration using the command:

      terraform state show <RESOURCE>.<RESOURCE_NAME>
    2. Update the value in the Terraform configuration file.

  7. Remove the previously added lifecycle block with the prevent_destroy = true parameter from the cluster resource.

For more information, see the Using Terraform section and the Terraform provider documentation.

Delete unused resources

Running resources in the cluster are charged for and consume compute resources. If you do not plan to use the PVCs and disk snapshots left after the migration, or the availability zone itself, delete them and their associated resources:

  1. Delete the old PVC and the disk snapshots.

  2. Delete the storage class (StorageClass) of the old availability zone:

    kubectl get sc -o name | grep <OLD_AVAILABILITY_ZONE> | xargs -I {} kubectl delete {}

    Here <OLD_AVAILABILITY_ZONE> — the name of the availability zone the migration was performed from, specified in lowercase.

Was this article helpful?