Data backup to object storage using rclone
Using the VK Object Storage service, you can organize data backup by choosing a tool suitable for a specific task. For command-line work, rclone is a good fit — a utility that supports S3-compatible storage, on-the-fly encryption, and flexible storage policies.
The following data is used as an example below:
- Bucket name:
backup-bucket - Region:
ru-msk - VK Object Storage endpoint:
https://hb.ru-msk.vkcloud-storage.ru
-
Create an account for the VK Object Storage service if this has not been done yet:
-
Go to the VK Cloud management console.
-
Select a project.
-
Go to the Object Storage → Accounts section.
-
Click the Add account or Add button.
-
Set the account name.
-
Click the Create button.
-
In the window that opens, copy and save the access key ID (Access Key ID) and the secret key (Secret Key).
-
-
Make sure that AWS CLI is installed and configured to work with the created VK Object Storage account. This will be needed if you plan to manage the object lifecycle via the CLI.
-
Create a bucket named
backup-bucket. -
Install the rclone utility by following the official documentation.
-
Run the interactive rclone configuration:
rclone config -
Select the
n(new remote) option to create a new connection and set the parameters:- Specify the connection name, for example
vkcloud. - Select the storage type
s3(Storage). - Select
Otheras the provider (any S3-compatible provider). - Set the connection parameters.
The resulting configuration in the
~/.config/rclone/rclone.conffile should look like this:[vkcloud]type = s3provider = Otherenv_auth = falseaccess_key_id = <ACCESS_KEY>secret_access_key = <SECRET_KEY>region = ru-mskendpoint = https://hb.ru-msk.vkcloud-storage.ruHere:
<ACCESS_KEY>— the Access Key ID value obtained when creating the VK Object Storage account.<SECRET_KEY>— the Secret Key value obtained when creating the VK Object Storage account.ru-msk— the region code of the account. For the Kazakhstan region, usekz-ast.https://hb.ru-msk.vkcloud-storage.ru— the endpoint of the VK Object Storage service in the Moscow region. For the Kazakhstan region, usehttps://hb.kz-ast.vkcloud-storage.ru.
- Specify the connection name, for example
-
Verify the connection by running the command:
rclone lsd vkcloud:The created bucket
backup-bucketshould appear in the output.
When performing a backup, you can exclude temporary and unnecessary files to reduce the amount of data transferred and the operation time. To do this, create a filter file filter.txt and specify in it the files that should be excluded:
# filter.txt- *.tmp- node_modules/**+ *
The name of this file must be specified with the --filter-from flag when calling the command for synchronizing the local and remote storage.
The + sign before a file name means that the file should be included in the backup, and the - sign means it should be ignored. In the example file above, the filters will work as follows:
- exclude all files ending with
.tmp(at any nesting level); - exclude the
node_modulesfolder and all its contents recursively; - include everything else that did not match the previous rules.
If your case is complex or there are only a few files to filter, you can specify them directly in the sync command via the --exclude or --include flags.
The rclone utility supports data encryption before sending it to the storage using a crypt connection type. This allows you to store data in an encrypted form even if the bucket does not have server-side encryption.
To configure encryption when synchronizing the local and remote storage:
-
Run the interactive rclone configuration:
rclone config -
Select the
n(new remote) option to create a new connection and set the parameters:- Specify the name of the connection with encryption, for example
vkcloud-encrypted. - Select the
crypttype. - Specify
vkcloud:backup-bucketas the connection source. - Set the main and additional passwords for encryption.
The resulting configuration in the
~/.config/rclone/rclone.conffile should look like this:[vkcloud-encrypted]type = cryptremote = vkcloud:backup-bucketpassword = <PASSWORD>password2 = <PASSWORD_SALT>Here:
vkcloud:backup-bucket— the full path to the remote storage, which consists of the name of the remote storage configured earlier viarclone configand the name of the bucket (storage) in the VK Object Storage service.<PASSWORD>— the encrypted password.<PASSWORD_SALT>— the additional encrypted password that the rclone utility's algorithm combines with the main password to compute the final cryptographic key.
- Specify the name of the connection with encryption, for example
To automatically delete outdated backups, use the built-in rclone capabilities:
-
Delete files older than a specified number of days:
rclone delete vkcloud:backup-bucket --min-age 30d
Here:
vkcloud:backup-bucket— the full path to the remote storage, which consists of the name of the remote storage configured earlier viarclone configand the name of the bucket (storage) in the VK Object Storage service.--min-age— an option for filtering files by age, which allows copying or processing files no older than the specified time, for example, no older than 30 days.
For incremental backups while keeping previous versions, use the --backup-dir flag directly in the sync command.
-
Create a test file:
mkdir -p /dataecho "test backup data" > /data/test-file.txt -
Perform the backup with encryption, the configured filters, and the configured backup retention policies:
rclone sync /data vkcloud-encrypted:/current \--filter-from filter.txt \--verbose \--backup-dir vkcloud:backup-bucket/archive/$(date +%Y-%m-%d)Here:
/data— the source from which the data needs to be transferred.vkcloud-encrypted:/current— a subconnection to the/currentfolder of the remote storage (VK Object Storage) with data encryption.--filter-from filter.txt— the filter file with rules for filtering data during synchronization.--backup-dir— an option to override the behavior of thesynccommand: on each run, the current version of the files from thecurrentfolder will be moved to thearchive/<DATE>folder, and new files will be uploaded tocurrent.--verbose— an option to enable detailed output: file names, transfer status, skips, errors, statistics, and per-thread progress.
-
Verify that the file has appeared in the bucket:
rclone ls vkcloud-encrypted:The
test-file.txtfile should be present in the output. -
Make sure the data in the bucket is encrypted — list the objects in the VK Object Storage bucket directly:
rclone ls vkcloud:backup-bucketThe file names and their contents should differ from the originals, since
cryptencrypts both the names and the data. -
Restore the file from the backup:
mkdir -p /restored-datarclone copy vkcloud-encrypted:test-file.txt /restored-data/ -
Check the contents of the restored file:
cat /restored-data/test-file.txtThe output should contain:
test backup data.
The bucket and the objects uploaded to it are billed. If you no longer need them:
- Delete the objects from the bucket.
- Delete the bucket.