VK Cloud logo

Data backup to object storage using rclone

Using the VK Object Storage service, you can organize data backup by choosing a tool suitable for a specific task. For command-line work, rclone is a good fit — a utility that supports S3-compatible storage, on-the-fly encryption, and flexible storage policies.

The following data is used as an example below:

  • Bucket name: backup-bucket
  • Region: ru-msk
  • VK Object Storage endpoint: https://hb.ru-msk.vkcloud-storage.ru

Preparation steps

  1. Create an account for the VK Object Storage service if this has not been done yet:

    1. Go to the VK Cloud management console.

    2. Select a project.

    3. Go to the Object Storage → Accounts section.

    4. Click the Add account or Add button.

    5. Set the account name.

    6. Click the Create button.

    7. In the window that opens, copy and save the access key ID (Access Key ID) and the secret key (Secret Key).

  2. Make sure that AWS CLI is installed and configured to work with the created VK Object Storage account. This will be needed if you plan to manage the object lifecycle via the CLI.

  3. Create a bucket named backup-bucket.

  4. Install the rclone utility by following the official documentation.

1. Configure rclone to work with VK Object Storage

  1. Run the interactive rclone configuration:

    rclone config
  2. Select the n (new remote) option to create a new connection and set the parameters:

    • Specify the connection name, for example vkcloud.
    • Select the storage type s3 (Storage).
    • Select Other as the provider (any S3-compatible provider).
    • Set the connection parameters.

    The resulting configuration in the ~/.config/rclone/rclone.conf file should look like this:

    [vkcloud]type = s3provider = Otherenv_auth = falseaccess_key_id = <ACCESS_KEY>secret_access_key = <SECRET_KEY>region = ru-mskendpoint = https://hb.ru-msk.vkcloud-storage.ru

    Here:

    • <ACCESS_KEY> — the Access Key ID value obtained when creating the VK Object Storage account.
    • <SECRET_KEY> — the Secret Key value obtained when creating the VK Object Storage account.
    • ru-msk — the region code of the account. For the Kazakhstan region, use kz-ast.
    • https://hb.ru-msk.vkcloud-storage.ru — the endpoint of the VK Object Storage service in the Moscow region. For the Kazakhstan region, use https://hb.kz-ast.vkcloud-storage.ru.
  3. Verify the connection by running the command:

    rclone lsd vkcloud:

    The created bucket backup-bucket should appear in the output.

2. (Optional) Configure filters to optimize traffic

When performing a backup, you can exclude temporary and unnecessary files to reduce the amount of data transferred and the operation time. To do this, create a filter file filter.txt and specify in it the files that should be excluded:

# filter.txt- *.tmp- node_modules/**+ *

The name of this file must be specified with the --filter-from flag when calling the command for synchronizing the local and remote storage.

The + sign before a file name means that the file should be included in the backup, and the - sign means it should be ignored. In the example file above, the filters will work as follows:

  • exclude all files ending with .tmp (at any nesting level);
  • exclude the node_modules folder and all its contents recursively;
  • include everything else that did not match the previous rules.

If your case is complex or there are only a few files to filter, you can specify them directly in the sync command via the --exclude or --include flags.

3. (Optional) Configure on-the-fly data encryption

The rclone utility supports data encryption before sending it to the storage using a crypt connection type. This allows you to store data in an encrypted form even if the bucket does not have server-side encryption.

To configure encryption when synchronizing the local and remote storage:

  1. Run the interactive rclone configuration:

    rclone config
  2. Select the n (new remote) option to create a new connection and set the parameters:

    • Specify the name of the connection with encryption, for example vkcloud-encrypted.
    • Select the crypt type.
    • Specify vkcloud:backup-bucket as the connection source.
    • Set the main and additional passwords for encryption.

    The resulting configuration in the ~/.config/rclone/rclone.conf file should look like this:

    [vkcloud-encrypted]type = cryptremote = vkcloud:backup-bucketpassword = <PASSWORD>password2 = <PASSWORD_SALT>

    Here:

    • vkcloud:backup-bucket — the full path to the remote storage, which consists of the name of the remote storage configured earlier via rclone config and the name of the bucket (storage) in the VK Object Storage service.
    • <PASSWORD> — the encrypted password.
    • <PASSWORD_SALT> — the additional encrypted password that the rclone utility's algorithm combines with the main password to compute the final cryptographic key.

4. (Optional) Configure backup retention policies

To automatically delete outdated backups, use the built-in rclone capabilities:

  1. Delete files older than a specified number of days:

    rclone delete vkcloud:backup-bucket --min-age 30d

Here:

  • vkcloud:backup-bucket — the full path to the remote storage, which consists of the name of the remote storage configured earlier via rclone config and the name of the bucket (storage) in the VK Object Storage service.
  • --min-age — an option for filtering files by age, which allows copying or processing files no older than the specified time, for example, no older than 30 days.

For incremental backups while keeping previous versions, use the --backup-dir flag directly in the sync command.

5. Verify that the backup works

  1. Create a test file:

    mkdir -p /dataecho "test backup data" > /data/test-file.txt
  2. Perform the backup with encryption, the configured filters, and the configured backup retention policies:

    rclone sync /data vkcloud-encrypted:/current \  --filter-from filter.txt \  --verbose \  --backup-dir vkcloud:backup-bucket/archive/$(date +%Y-%m-%d)

    Here:

    • /data — the source from which the data needs to be transferred.
    • vkcloud-encrypted:/current — a subconnection to the /current folder of the remote storage (VK Object Storage) with data encryption.
    • --filter-from filter.txt — the filter file with rules for filtering data during synchronization.
    • --backup-dir — an option to override the behavior of the sync command: on each run, the current version of the files from the current folder will be moved to the archive/<DATE> folder, and new files will be uploaded to current.
    • --verbose — an option to enable detailed output: file names, transfer status, skips, errors, statistics, and per-thread progress.
  3. Verify that the file has appeared in the bucket:

    rclone ls vkcloud-encrypted:

    The test-file.txt file should be present in the output.

  4. Make sure the data in the bucket is encrypted — list the objects in the VK Object Storage bucket directly:

    rclone ls vkcloud:backup-bucket

    The file names and their contents should differ from the originals, since crypt encrypts both the names and the data.

  5. Restore the file from the backup:

    mkdir -p /restored-datarclone copy vkcloud-encrypted:test-file.txt /restored-data/
  6. Check the contents of the restored file:

    cat /restored-data/test-file.txt

    The output should contain: test backup data.

Delete unused resources

The bucket and the objects uploaded to it are billed. If you no longer need them:

  1. Delete the objects from the bucket.
  2. Delete the bucket.
Was this article helpful?