Migrating data from AWS S3 to VK Cloud object storage
Using the VK Object Storage service, you can transfer data from AWS S3, reducing storage costs and avoiding vendor lock-in. The migration is performed with AWS CLI or rclone tools, followed by data integrity validation and switching applications to the new endpoint.
The following sample data is used below for illustration:
- Source: bucket
my-aws-bucketin AWS S3, regionus-east-1. - Destination: bucket
my-vk-bucketin VK Object Storage, regionru-msk. - AWS CLI profile for the source:
aws-source. - AWS CLI profile for the destination:
vk-dest. - VK Object Storage endpoint:
https://hb.ru-msk.vkcloud-storage.ru.
Replace the sample values with your actual data.
-
Create an account for the VK Object Storage service, if not already done:
-
Go to the VK Cloud personal dashboard.
-
Select a project.
-
Go to the Object Storage → Аккаунты section.
-
Click the Добавить аккаунт or Добавить button.
-
Set the account name.
-
Click the Создать button.
-
In the window that opens, copy and save the access key ID (Access Key ID) and the secret key (Secret Key).
-
-
Make sure AWS CLI is installed and configured to work with the created VK Object Storage account.
-
(Optional) Install rclone if you plan to use it for the migration instead of AWS CLI.
Two migration strategies are available:
- Big Bang — a full data transfer in a single pass, followed by switching all applications to the new endpoint. Suitable for small data volumes or systems with an acceptable downtime window. All changes in the source bucket made after the migration starts must be transferred again.
- Phased migration (Dual write) — applications write data to both storages in parallel, while a background process transfers historical data. After the transfer is complete, applications are switched to VK Object Storage. Suitable for systems with no acceptable downtime and large data volumes.
Below we describe migration using the Big Bang strategy with aws s3 sync. For phased migration, use the same tool, but run the synchronization repeatedly until applications are fully switched.
For migration with aws s3 sync, you will need two AWS CLI profiles: one for reading from AWS S3 and one for writing to VK Object Storage.
-
Configure the
aws-sourceprofile for access to AWS S3, if not already configured:aws configure --profile aws-sourceSet the values:
AWS Access Key ID— the access key for AWS S3.AWS Secret Access Key— the AWS S3 secret key.Default region name— the AWS S3 region, for exampleus-east-1.Default output format—json.
-
Configure the
vk-destprofile for access to VK Object Storage:aws configure --profile vk-destSet the values:
AWS Access Key ID— the previously saved access key ID (Access Key ID) of the VK Object Storage account.AWS Secret Access Key— the previously saved secret key (Secret Key) of the VK Object Storage account.Default region name—ru-msk.Default output format—json.
-
Verify access to both storages:
aws s3 ls --profile aws-sourceaws s3 ls --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-destThe first command should output the list of AWS S3 buckets, the second — the list of VK Object Storage buckets.
Create a bucket my-vk-bucket in VK Object Storage via the personal dashboard or using AWS CLI:
aws s3api create-bucket \--bucket my-vk-bucket \--endpoint-url https://hb.ru-msk.vkcloud-storage.ru \--region ru-msk \--profile vk-dest
The aws s3 sync command copies objects from the source bucket to the target one, skipping already copied files with matching ETag and size.
-
Run the synchronization:
aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \--source-region us-east-1 \--region ru-msk \--endpoint-url https://hb.ru-msk.vkcloud-storage.ru \--profile vk-destWhere:
s3://my-aws-bucket— the source AWS S3 bucket.s3://my-vk-bucket— the target VK Object Storage bucket.--source-region us-east-1— the source region.--region ru-msk— the destination region.--endpoint-url— the VK Object Storage endpoint.--profile vk-dest— the profile with access keys for VK Object Storage.
-
Wait for the synchronization to complete. For large data volumes, this process may take considerable time.
After the migration is complete, make sure all objects have been transferred correctly.
-
Compare the number of objects in the source and target buckets:
aws s3api list-objects-v2 --bucket my-aws-bucket --profile aws-source --query "KeyCount" --output textaws s3api list-objects-v2 --bucket my-vk-bucket --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-dest --query "KeyCount" --output textThe values must match.
-
Compare the object ETags (checksums). Get the object lists from both buckets:
aws s3api list-objects-v2 --bucket my-aws-bucket --profile aws-source --query "Contents[*].[Key,ETag,Size]" --output text > aws-objects.txtaws s3api list-objects-v2 --bucket my-vk-bucket --endpoint-url https://hb.ru-msk.vkcloud-storage.ru --profile vk-dest --query "Contents[*].[Key,ETag,Size]" --output text > vk-objects.txt -
Compare the resulting files:
diff aws-objects.txt vk-objects.txtIf the output is empty, all objects have been transferred correctly. If there are differences, run the synchronization again:
aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \--source-region us-east-1 \--region ru-msk \--endpoint-url https://hb.ru-msk.vkcloud-storage.ru \--profile vk-dest
-
Update the application configuration to use the VK Object Storage endpoint:
- Endpoint URL:
https://hb.ru-msk.vkcloud-storage.ru - Region:
ru-msk - Access Key ID and Secret Key — the keys of the VK Object Storage account.
- Bucket name:
my-vk-bucket
- Endpoint URL:
-
If applications access the bucket by domain name, update the DNS records. For public objects, use the URL format
https://<ИМЯ_БАКЕТА>.hb.ru-msk.vkcloud-storage.ru/<КЛЮЧ_ОБЪЕКТА>. -
If the Dual write strategy is used, stop writing to AWS S3 after confirming that the applications work correctly with VK Object Storage.
-
Perform a final synchronization to transfer objects modified after the main migration:
aws s3 sync s3://my-aws-bucket s3://my-vk-bucket \--source-region us-east-1 \--region ru-msk \--endpoint-url https://hb.ru-msk.vkcloud-storage.ru \--profile vk-dest
After the migration is complete and the applications are confirmed to work correctly with VK Object Storage, delete the data and the bucket in AWS S3 as described in the AWS documentation.
The bucket and the objects uploaded to it in VK Object Storage are billed. If you no longer need them: