VK Cloud logo

VK Object Storage in the Data Lakehouse architecture

VK Object Storage is a cost-effective object storage for unstructured data from VK Cloud. It is a key component of DLH, as it provides flexibility, scalability, and cost-effective data storage.

This component performs the following functions:

  • Distributed storage of objects in various formats.

  • Storage metadata management at the entity level:

    • Projects — projects to which S3 files belong.
    • Buckets — local disks connected to S3 within projects.
    • Objects — key parameters for processing S3 objects (files).
  • Defining rules for uploading, storing, processing, and downloading files.

  • Managing the role-based access model for data with support for Access Control List (ACL) — who can access a file and which specific operations are allowed.

  • Support for the standard Amazon S3 API for data access.

  • Organizing incoming and outgoing content transfer streams to and from the storage.

  • Organizing fault-tolerant uploads of large files in multiple streams (multipart), including:

    • retry policy;
    • file splitting logic.
  • Monitoring upload success and detecting corrupted files in S3.

  • Fast object search in S3.

  • Providing S3 data upon requests from external services.

Was this article helpful?