VK Cloud logo

Downloading a copy via OpenStack CLI and API

With this method, you get backup copy IDs via OpenStack CLI or the Karboii API. Then you download the copy files directly from VK Object Storage via AWS CLI. Use this method to automate downloads or if the management console is unavailable. You can also download the copy via the management console.

Before you download a copy

  1. Make sure that API access is activated.

  2. Install the OpenStack client and authenticate in the project that stores the backups. For authentication, use the OpenStack RC file or another way to get a token.

  3. Make note of the Project ID of the project. You will need it when you work via API and when you download files from VK Object Storage.

    The Project ID differs from the project ID in the management console. If you authenticated with the OpenStack RC file, the value is stored in the OS_PROJECT_ID variable:

    echo $OS_PROJECT_ID

    If the variable is empty, get the Project ID with the command:

    openstack token issue -c project_id -f value

    You can also get the Project ID in the management console. Save it to a variable:

    export OS_PROJECT_ID=<PROJECT_ID>

    Here <PROJECT_ID> is the Project ID of the project.

  4. To work via API, install jq and make sure that curl is installed.

  5. Create a VK Object Storage account and get its access key (Access Key ID and Secret Key) if you have not done so yet.

  6. Install AWS CLI and configure the connection to VK Object Storage.

  7. Get the VK Object Storage endpoint address for the region of your project. For more information, see Connecting to VK Object Storage via CLI.

  8. Make sure that all the access you set up works by testing it on a test disk.

1. Determine the copies to download

  1. Get the list of the project backups:

    openstack volume backup list

    The ID column shows the ID of each copy. To get the copies of only one disk:

    1. Get the VM disk ID with the command:

      openstack volume list

      The disk ID is also available in the management console under Cloud Servers → Disks.

    2. Run the command:

      openstack volume backup list --volume <DISK_ID>
  2. Get the status, size, and type of the copy with the command:

    openstack volume backup show <COPY_ID>

    Here <COPY_ID> is the backup ID from the ID column of the list you got in the previous step.

    The command output contains information about the backup. Determine its type:

    • Full copy: the name field has the _full suffix. You only need to download this copy of the disk.
    • Incremental copy: the is_incremental parameter is displayed. You need to download the entire chain: the base full copy and all incremental copies after it, up to the required one.
  3. Make note of the IDs of all copies in the chain, in order from the full copy to the selected one. For a full copy, it's one ID. Use these IDs instead of <COPY_ID> when you download the files.

  4. Make note of the exact disk size in GB. You will need it to assemble the disk. You can do this in one of the following ways:

    • By copy ID:

      openstack volume backup show <COPY_ID> -c size -f value
    • By disk ID:

      openstack volume show <DISK_ID> -c size -f value

2. Prepare a machine for the copy files

  1. Select a Linux machine to download the files to.
  2. Make sure that the machine has enough free space. You need space for the compressed content file of each copy and for the resulting RAW file. The RAW file is as large as the disk.

Right after you assemble the disk, the RAW file may take up less space. When you copy or convert it, it takes the full disk size.

3. Download the copy files from VK Object Storage

The copy files are stored in the volumebackups-<PROJECT_ID> system bucket. The folder name matches the copy ID.

For each copy, run the command:

aws --endpoint-url <S3_ADDRESS> s3 cp s3://volumebackups-<PROJECT_ID>/<COPY_ID>/ ./<COPY_ID>/ --recursive

Here:

  • <S3_ADDRESS> is the VK Object Storage endpoint address, for example, https://hb.ru-msk.vkcloud-storage.ru.

  • <PROJECT_ID> is the Project ID of the project that you obtained earlier.

  • <COPY_ID> is the backup ID:

    • For OpenStack CLI, the value from the ID column.
    • For API, the external_id value.

The files are saved in the ./<COPY_ID>/<file> structure. The command downloads five files for each copy:

  • content
  • object_segments
  • volume_segments
  • sha_list
  • volume_metadata

For more information, see Migrating VM disk backups outside VK Cloud.

If you are downloading a chain of copies, run the command for each copy in the chain.

4. Verify the downloaded files

  1. Make sure that the folder of each copy contains all five files (content, object_segments, volume_segments, sha_list, and volume_metadata). To do this, run the command:

    ls -l ./<COPY_FOLDER>/

    Here <COPY_FOLDER> is the folder that contains the backup copy files.

  2. If the content file was downloaded in parts (content1, content2, and so on), join them via natural sorting:

    cat $(ls ./<COPY_FOLDER>/content[0-9]* | sort -V) > ./<COPY_FOLDER>/content

    Here <COPY_FOLDER> is the folder that contains the backup copy files.

  3. Make sure that the content file was downloaded completely. To do this, take the last element in the object_segments file. Compare the sum of object_offset and object_length with the size of the content file:

    python3 -I - <<'EOF'import json, osd = "./<COPY_FOLDER>"seg = json.load(open(os.path.join(d, "object_segments")))last = seg[-1]print("by map:", last["object_offset"] + last["object_length"],      "file:", os.path.getsize(os.path.join(d, "content")))EOF

    Here <COPY_FOLDER> is the folder that contains the backup copy files.

    If the numbers do not match, the file was not downloaded completely. Download the copy again.

Was this article helpful?