VK Cloud logo

Migrating data from AWS S3 to VK Cloud object storage

Using the VK Object Storage service, you can transfer data from AWS S3, reducing storage costs and avoiding vendor lock-in. The migration is performed with AWS CLI or rclone tools, followed by data integrity validation and switching applications to the new endpoint.

The following sample data is used below for illustration:

  • Source: bucket my-aws-bucket in AWS S3, region us-east-1.
  • Destination: bucket my-vk-bucket in VK Object Storage, region ru-msk.
  • AWS CLI profile for the source: aws-source.
  • AWS CLI profile for the destination: vk-dest.
  • VK Object Storage endpoint: https://hb.ru-msk.vkcloud-storage.ru.

Replace the sample values with your actual data.

Preparation steps

  1. Create an account for the VK Object Storage service, if not already done:

    1. Go to the VK Cloud personal dashboard.

    2. Select a project.

    3. Go to the Object Storage → Аккаунты section.

    4. Click the Добавить аккаунт or Добавить button.

    5. Set the account name.

    6. Click the Создать button.

    7. In the window that opens, copy and save the access key ID (Access Key ID) and the secret key (Secret Key).

  2. Make sure AWS CLI is installed and configured to work with the created VK Object Storage account.

  3. (Optional) Install rclone if you plan to use it for the migration instead of AWS CLI.

1. Choose a migration strategy

Two migration strategies are available:

  • Big Bang — a full data transfer in a single pass, followed by switching all applications to the new endpoint. Suitable for small data volumes or systems with an acceptable downtime window. All changes in the source bucket made after the migration starts must be transferred again.
  • Phased migration (Dual write) — applications write data to both storages in parallel, while a background process transfers historical data. After the transfer is complete, applications are switched to VK Object Storage. Suitable for systems with no acceptable downtime and large data volumes.

Below we describe migration using the Big Bang strategy with aws s3 sync. For phased migration, use the same tool, but run the synchronization repeatedly until applications are fully switched.

2. Configure AWS CLI profiles for the source and destination

For migration with aws s3 sync, you will need two AWS CLI profiles: one for reading from AWS S3 and one for writing to VK Object Storage.

  1. Configure the aws-source profile for access to AWS S3, if not already configured:

    aws configure --profile aws-source

    Set the values:

    • AWS Access Key ID — the access key for AWS S3.
    • AWS Secret Access Key — the AWS S3 secret key.
    • Default region name — the AWS S3 region, for example us-east-1.
    • Default output format — json.
  2. Configure the vk-dest profile for access to VK Object Storage:

    aws configure --profile vk-dest

    Set the values:

    • AWS Access Key ID — the previously saved access key ID (Access Key ID) of the VK Object Storage account.
    • AWS Secret Access Key — the previously saved secret key (Secret Key) of the VK Object Storage account.
    • Default region name — ru-msk.
    • Default output format — json.
  3. Verify access to both storages:

    aws s3 ls --profile aws-sourceaws s3 ls --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-dest

    The first command should output the list of AWS S3 buckets, the second — the list of VK Object Storage buckets.

3. Create the target bucket in VK Cloud

Create a bucket my-vk-bucket in VK Object Storage via the personal dashboard or using AWS CLI:

aws s3api create-bucket \    --bucket my-vk-bucket \    --endpoint-url https://hb.ru-msk.vkcloud-storage.ru \    --region ru-msk \    --profile vk-dest

4. Perform the data migration

The aws s3 sync command copies objects from the source bucket to the target one, skipping already copied files with matching ETag and size.

  1. Run the synchronization:

    aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \    --source-region us-east-1 \    --region ru-msk \    --endpoint-url https://hb.ru-msk.vkcloud-storage.ru \    --profile vk-dest

    Where:

    • s3://my-aws-bucket — the source AWS S3 bucket.
    • s3://my-vk-bucket — the target VK Object Storage bucket.
    • --source-region us-east-1 — the source region.
    • --region ru-msk — the destination region.
    • --endpoint-url — the VK Object Storage endpoint.
    • --profile vk-dest — the profile with access keys for VK Object Storage.
  2. Wait for the synchronization to complete. For large data volumes, this process may take considerable time.

5. Verify the integrity of the transferred data

After the migration is complete, make sure all objects have been transferred correctly.

  1. Compare the number of objects in the source and target buckets:

    aws s3api list-objects-v2 --bucket my-aws-bucket --profile aws-source --query "KeyCount" --output textaws s3api list-objects-v2 --bucket my-vk-bucket --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-dest --query "KeyCount" --output text

    The values must match.

  2. Compare the object ETags (checksums). Get the object lists from both buckets:

    aws s3api list-objects-v2 --bucket my-aws-bucket --profile aws-source --query "Contents[*].[Key,ETag,Size]" --output text > aws-objects.txtaws s3api list-objects-v2 --bucket my-vk-bucket --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-dest --query "Contents[*].[Key,ETag,Size]" --output text > vk-objects.txt
  3. Compare the resulting files:

    diff aws-objects.txt vk-objects.txt

    If the output is empty, all objects have been transferred correctly. If there are differences, run the synchronization again:

    aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \    --source-region us-east-1 \    --region ru-msk \    --endpoint-url https://hb.ru-msk.vkcloud-storage.ru \    --profile vk-dest

6. Switch applications to the new endpoint

  1. Update the application configuration to use the VK Object Storage endpoint:

    • Endpoint URL: https://hb.ru-msk.vkcloud-storage.ru
    • Region: ru-msk
    • Access Key ID and Secret Key — the keys of the VK Object Storage account.
    • Bucket name: my-vk-bucket
  2. If applications access the bucket by domain name, update the DNS records. For public objects, use the URL format https://<ИМЯ_БАКЕТА>.hb.ru-msk.vkcloud-storage.ru/<КЛЮЧ_ОБЪЕКТА>.

  3. If the Dual write strategy is used, stop writing to AWS S3 after confirming that the applications work correctly with VK Object Storage.

  4. Perform a final synchronization to transfer objects modified after the main migration:

    aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \    --source-region us-east-1 \    --region ru-msk \    --endpoint-url https://hb.ru-msk.vkcloud-storage.ru \    --profile vk-dest

Delete unused resources

After the migration is complete and the applications are confirmed to work correctly with VK Object Storage, delete the data and the bucket in AWS S3 as described in the AWS documentation.

The bucket and the objects uploaded to it in VK Object Storage are billed. If you no longer need them:

  1. Delete the objects.
  2. Delete the bucket.
Was this article helpful?