VK Object Storage in the Data Lakehouse architecture
VK Object Storage is a cost-effective object storage for unstructured data from VK Cloud. It is a key component of DLH, as it provides flexibility, scalability, and cost-effective data storage.
This component performs the following functions:
-
Distributed storage of objects in various formats.
-
Storage metadata management at the entity level:
Projects— projects to which S3 files belong.Buckets— local disks connected to S3 within projects.Objects— key parameters for processing S3 objects (files).
-
Defining rules for uploading, storing, processing, and downloading files.
-
Managing the role-based access model for data with support for Access Control List (ACL) — who can access a file and which specific operations are allowed.
-
Support for the standard Amazon S3 API for data access.
-
Organizing incoming and outgoing content transfer streams to and from the storage.
-
Organizing fault-tolerant uploads of large files in multiple streams (multipart), including:
- retry policy;
- file splitting logic.
-
Monitoring upload success and detecting corrupted files in S3.
-
Fast object search in S3.
-
Providing S3 data upon requests from external services.