VK Cloud logo

Service architecture

Data Lakehouse (DLH) collects structured and unstructured data from various sources, processes it via Cloud Spark (ETL or ELT) or Cloud Trino (SQL queries), stores it in S3 storage with cataloging in Cloud Iceberg Metastore, manages metadata in DataHub, and provides data for analytics, BI, and ML via Cloud Trino, Cloud Spark, or Cloud ClickHouse.

Service architecture
  1. Structured data from operational (OLTP) and enterprise (CRM, ERP) systems arrives in Cloud Trino directly or via Cloud Spark for executing complex ETL and ELT processes.

  2. Semi-structured (JSON, XML, CSV) and unstructured (media files, documents) data is transferred to S3 storage as streams or in batches.

  3. Cloud Trino and Cloud Spark use the Iceberg Metastore API to exchange data with S3 storage.

    In Cloud Trino, the query processing workflow includes receiving (PULL model) and optimizing SQL queries by the Coordinator, which then distributes them among Workers, providing functionality similar to traditional DBMSs.

    Cloud Spark is used to run complex ETL and ELT processes. This service supports both batch and stream data processing:

    • In stream processing, data is processed continuously, in mini-batches or event-by-event, in real time. Stream processing is performed using the PUSH model.

    • In batch processing, data is accumulated in batches (files, tables) and processed all at once. The extraction process is started on a schedule or manually (PULL model).

  4. Cloud Iceberg Metastore catalogs the objects of S3 storage and provides an API for accessing it.

    Data storage is organized with support for ACID transactions via the Cloud Iceberg Metastore service, which allows representing S3 storage objects as database tables. Using S3 storage significantly reduces storage costs.

  5. The overall list of objects and the rules for processing them is published in the data catalog based on DataHub (with support for Data Discovery, Data Quality, Metadata Management, Data Governance, Data Lineage functionality) for access management and data lineage tracking.

  6. Processed data from Cloud Trino and Cloud Spark can be used in various external analytical systems: MLflow, Jupyter, ClickHouse. These systems allow performing in-depth data analysis, creating detailed visualizations, and managing the machine learning model lifecycle.

Data processing models

Depending on the task, data can be processed in the following ways:

  • PULL model — an approach in which data is extracted from the source on a schedule by running batch processes, which are convenient to manage in Cloud Airflow. This approach is used for simple ETL and ELT processes, analytics in S3 storage, BI integration, SQL queries to databases, and when the load on the source needs to be controlled.

  • PUSH model — an approach in which data is automatically sent from the source to Cloud Spark or Cloud Trino. This approach is used for tasks where continuous delivery and processing of large volumes of data are critical: complex ETL and ELT processes, data preparation for analytics, machine learning on big data.

When designing the Data Lakehouse architecture, it is important to determine which types of data will be processed and how:

Criterion

PULL model

PUSH model

Service

Cloud Airflow, Cloud Trino, Cloud Spark (batch processing)

Cloud Spark (stream processing, micro-batches)

Source support

API, databases, files

For each source, a component must be developed that will send data via the Producer API

Update frequency

On a schedule

In real time

Data latency

5–60 minutes

< 1 second

Load on the source

High (50–500 connections per hour)

Low (1–5 permanent connections)

Scalability

Limited (up to 100 parallel operations per second)

High (up to 10,000 parallel operations per second)

Integration complexity

Low (1–2 days for configuration)

High (Producer development required)

Was this article helpful?