VK Cloud logo

About the service

Data Lakehouse (DLH) is a hybrid data management architecture that can be organized on the VK Cloud platform. It combines the scalability and flexibility of a Data Lake with the reliability and structure of a Data Warehouse. Such an architecture:

  • Provides a single space for storing, processing, and analyzing data of any volume and format.
  • Supports ACID transactions and version control.
  • Is compatible with modern analytical tools: Spark, Trino, BI systems.

In Data Lakehouse, the data storage layer is implemented on the basis of S3 storage, and a familiar SQL interface is used to access the data. This approach makes it possible to simplify the storage of "raw" unstructured data, make the infrastructure for it cheaper, and at the same time efficiently run analytical SQL queries without losing the quality of the results. Physically, both layers are separated and can be scaled horizontally independently of each other.

Unstructured data is saved to S3 storage from various sources in different data formats: TSV, CSV, XML, syslog, JSON, etc. For example, such data may include:

  • video recordings from outdoor surveillance cameras;
  • telemetry from sensors and devices;
  • graphic files;
  • data on website user behavior;
  • logs from information systems.

In their "raw" form, such data is unsuitable for daily analytics in BI systems, but can be used for rapid testing of new business hypotheses using machine learning algorithms or other Data Science methods.

To organize full access of analytical services to Data Lakehouse data, you need to annotate and catalog the information about the objects in the S3 storage in one of the commonly accepted open formats (for example, using the Cloud Iceberg Metastore service). Then, a SQL query execution engine based on Cloud Trino or Cloud Spark is connected to the annotated data via an API interface to organize end-to-end real-time event streaming.

Thus, Data Lakehouse allows you to use business analytics tools directly on the source data, increasing their relevance as well as reducing the latency and costs associated with the need to perform ETL operations.

What tasks is it suitable for

Data Lakehouse based on VK Cloud is an effective solution if you need to:

  • replace an outdated Hadoop data processing stack;
  • migrate from Greenplum;
  • implement a big data system based on your existing S3;
  • build a Data Office "from scratch".

How to get started?[id=dlh_how_to_start]

Depending on the task, you can use Data Lakehouse in different scenarios:

  • Integration of Data Lakehouse components with external data sources. Each time a request is made, information is transferred between the systems.
  • Creating a full Data Lakehouse architecture based on VK Cloud and migrating with a one-time transfer of data to the new system.

You can find some scenarios for working with Data Lakehouse in the Practical guides section.

Was this article helpful?