Downloading a copy via OpenStack CLI and API
With this method, you get backup copy IDs via OpenStack CLI or the Karboii API. Then you download the copy files directly from VK Object Storage via AWS CLI. Use this method to automate downloads or if the management console is unavailable. You can also download the copy via the management console.
-
Make sure that API access is activated.
-
Install the OpenStack client and authenticate in the project that stores the backups. For authentication, use the OpenStack RC file or another way to get a token.
-
Make note of the Project ID of the project. You will need it when you work via API and when you download files from VK Object Storage.
The Project ID differs from the project ID in the management console. If you authenticated with the OpenStack RC file, the value is stored in the
OS_PROJECT_IDvariable:echo $OS_PROJECT_IDIf the variable is empty, get the Project ID with the command:
openstack token issue -c project_id -f valueYou can also get the Project ID in the management console. Save it to a variable:
export OS_PROJECT_ID=<PROJECT_ID>Here
<PROJECT_ID>is the Project ID of the project. -
To work via API, install
jqand make sure thatcurlis installed. -
Create a VK Object Storage account and get its access key (
Access Key IDandSecret Key) if you have not done so yet. -
Install AWS CLI and configure the connection to VK Object Storage.
-
Get the VK Object Storage endpoint address for the region of your project. For more information, see Connecting to VK Object Storage via CLI.
-
Make sure that all the access you set up works by testing it on a test disk.
-
Get the list of the project backups:
openstack volume backup listThe
IDcolumn shows the ID of each copy. To get the copies of only one disk:-
Get the VM disk ID with the command:
openstack volume listThe disk ID is also available in the management console under Cloud Servers → Disks.
-
Run the command:
openstack volume backup list --volume <DISK_ID>
-
-
Get the status, size, and type of the copy with the command:
openstack volume backup show <COPY_ID>Here
<COPY_ID>is the backup ID from theIDcolumn of the list you got in the previous step.The command output contains information about the backup. Determine its type:
- Full copy: the
namefield has the_fullsuffix. You only need to download this copy of the disk. - Incremental copy: the
is_incrementalparameter is displayed. You need to download the entire chain: the base full copy and all incremental copies after it, up to the required one.
- Full copy: the
-
Make note of the IDs of all copies in the chain, in order from the full copy to the selected one. For a full copy, it's one ID. Use these IDs instead of
<COPY_ID>when you download the files. -
Make note of the exact disk size in GB. You will need it to assemble the disk. You can do this in one of the following ways:
-
By copy ID:
openstack volume backup show <COPY_ID> -c size -f value -
By disk ID:
openstack volume show <DISK_ID> -c size -f value
-
- Select a Linux machine to download the files to.
- Make sure that the machine has enough free space. You need space for the compressed
contentfile of each copy and for the resulting RAW file. The RAW file is as large as the disk.
Right after you assemble the disk, the RAW file may take up less space. When you copy or convert it, it takes the full disk size.
The copy files are stored in the volumebackups-<PROJECT_ID> system bucket. The folder name matches the copy ID.
For each copy, run the command:
aws --endpoint-url <S3_ADDRESS> s3 cp s3://volumebackups-<PROJECT_ID>/<COPY_ID>/ ./<COPY_ID>/ --recursive
Here:
-
<S3_ADDRESS>is the VK Object Storage endpoint address, for example,https://hb.ru-msk.vkcloud-storage.ru. -
<PROJECT_ID>is the Project ID of the project that you obtained earlier. -
<COPY_ID>is the backup ID:- For OpenStack CLI, the value from the
IDcolumn. - For API, the
external_idvalue.
- For OpenStack CLI, the value from the
The files are saved in the ./<COPY_ID>/<file> structure. The command downloads five files for each copy:
contentobject_segmentsvolume_segmentssha_listvolume_metadata
For more information, see Migrating VM disk backups outside VK Cloud.
If you are downloading a chain of copies, run the command for each copy in the chain.
-
Make sure that the folder of each copy contains all five files (
content,object_segments,volume_segments,sha_list, andvolume_metadata). To do this, run the command:ls -l ./<COPY_FOLDER>/Here
<COPY_FOLDER>is the folder that contains the backup copy files. -
If the
contentfile was downloaded in parts (content1,content2, and so on), join them via natural sorting:cat $(ls ./<COPY_FOLDER>/content[0-9]* | sort -V) > ./<COPY_FOLDER>/contentHere
<COPY_FOLDER>is the folder that contains the backup copy files. -
Make sure that the
contentfile was downloaded completely. To do this, take the last element in theobject_segmentsfile. Compare the sum ofobject_offsetandobject_lengthwith the size of thecontentfile:python3 -I - <<'EOF'import json, osd = "./<COPY_FOLDER>"seg = json.load(open(os.path.join(d, "object_segments")))last = seg[-1]print("by map:", last["object_offset"] + last["object_length"],"file:", os.path.getsize(os.path.join(d, "content")))EOFHere
<COPY_FOLDER>is the folder that contains the backup copy files.If the numbers do not match, the file was not downloaded completely. Download the copy again.