Quick Start
The quick start covers a basic scenario for deploying the Data Lakehouse (DLH) service based on three components:
-
VK Object Storage — S3 storage used as an inexpensive, infinitely scalable storage for unstructured data received from various databases, such as PostgreSQL.
-
Cloud Iceberg Metastore — used to organize a metadata catalog based on Apache Iceberg. It ensures the consistency of operations between the various DLH service components.
-
Cloud Trino — in combination with Cloud Iceberg Metastore, provides customers with the familiar SQL interface for performing the following operations:
- data transfer (ETL);
- manipulation of captured data (ELT);
- combining data from different sources (Data Mesh);
- other operations with the catalog and data.
After completing all the quick start steps, you will:
- Create instances of the Cloud Iceberg Metastore and Cloud Trino services.
- Create a PostgreSQL DB instance and configure a connection to it in Cloud Trino.
The S3 storage bucket will be created automatically when configuring connections for the Cloud Trino service instance.
-
Go to the Data Platform → Service Instances section.
-
Click the Create Instance button.
-
At the Configuration step:
- Select the
Cloud Iceberg Metastoreservice type. - Leave the default value in the Version field.
- Select the
Clusterconfiguration.
- Select the
-
Click the Next Step button.
-
At the Parameters step:
- Write down the database name generated automatically. You will need it to connect to Cloud Trino.
- Enable the Assign External IP option.
- Select the Create a New Cluster option.
- Select one of the available networks from the list or create a new one by clicking Create a New Network. When creating a new subnet, specify the SDN and the subnet address.
- Select an availability zone, for example:
Москва (ME1). - Select the
Basicsetup option and theSmallcluster template.
-
Click the Next Step button.
-
At the Maintenance step:
-
Select the days of the week and the start time for maintenance. During this time, maintenance work may be performed on the server, including backup creation — the service may be temporarily unavailable.
-
Leave the default values for the remaining fields.
-
-
Click the Next Step button.
-
At the Credentials step:
-
Leave the default value in the Login field. Write down the administrator login for accessing Cloud Iceberg Metastore; you will need it later.
-
In the Password field, click the Generate button to create an administrator password. Save the password; you will need it later.
-
-
Click the Create button.
Wait for the Cloud Iceberg Metastore service instance creation process to complete.
The PostgreSQL DB in the basic DLH service configuration acts as an external data source. To deploy a PostgreSQL instance:
-
Go to the Databases → Database Instances section.
-
If there are no DB instances in the selected project, click the Create Database button.
Otherwise, click the Add button.
-
At the Configuration step:
- Select the
PostgreSQLdatabase type. - Leave the default value in the Version field.
- Select the
Singleconfiguration.
- Select the
-
Click the Next Step button.
-
At the Instance Creation step, specify:
-
Virtual Machine Type:
STD3-2-8. -
Availability Zone:
Москва (ME1). -
Disk Type:
SSD. -
Disk Size, GB:
10. -
Enable Disk Autoscaling: make sure this option is not selected.
-
Network: select the network you specified when creating the Cloud Iceberg Metastore service instance during the basic configuration deployment.
-
Assign External IP: make sure this option is enabled.
-
Firewall Settings: select
sshfrom the drop-down list.The resulting list of security groups should look as follows:
default,ssh. -
Create a Replica: make sure this option is not selected.
-
SSH Access Key:
Create a New Key. -
Backup:
Disabled.
-
-
Click the Next Step button.
-
At the Initialization step, specify:
- Creation Type:
New Database. - Username: leave the default value or come up with your own, for example:
tsuser1. - User Password: generate a password.
Save the resulting login and password values. You will need them later.
The database name to be created will be generated automatically.
- Creation Type:
-
Click the Create Database button.
Wait for the DB instance creation to complete.
Cloud Trino is used as a horizontally scalable SQL query engine for the external data source (PostgreSQL).
To deploy a Cloud Trino service instance and configure its PostgreSQL connections:
-
Go to the Data Platform → Service Instances section.
-
Click the Create Instance button.
-
At the Configuration step:
- Select the
Cloud Trinoservice type. - Leave the default value in the Version field.
- Select the
Singleconfiguration.
- Select the
-
Click the Next Step button.
-
At the Parameters step:
- In the Instance Name field, specify an instance name or use the default value.
- Enable the Assign External IP option.
- Select the cluster you specified when creating the Cloud Iceberg Metastore service instance during the basic configuration deployment.
- Select the
Basicsetup option and theSmallcluster template. - Click the Next Step button.
-
At the Maintenance step:
-
Maintenance Window: leave the default value. During this time, maintenance work may be performed on the server, including backup creation — the service may be temporarily unavailable.
-
Leave the default values for the remaining fields.
-
-
Click the Next Step button.
-
At the Connections step:
-
Add the PostgreSQL data source to the Cloud Trino service instance:
-
Click the Add Connection button.
-
Select the
PostgreSQLconnection type and specify:- Name: come up with a connection name, for example:
postgres_trino. - Database Name: the name of the database inside the instance, for example:
ad_db. It is available on the Database List tab after creating the PostgreSQL instance. - hostname:port: the connection address of the created PostgreSQL instance in the format:
<IP ADDRESS>:<PORT>. For example,10.0.1.2:5432. You can view the DB connection address in the instance card.
- Name: come up with a connection name, for example:
-
Login: the value specified when creating the PostgreSQL DB instance. In this example:
tsuser1. -
Password: the password of the database user account specified when creating the PostgreSQL DB instance.
-
-
Add the Apache Iceberg data source with VK Object Storage to the Cloud Trino service instance:
-
Click the Add Connection button.
-
Select the
Apache Iceberg with S3 VK Cloudconnection. -
Specify the connection parameters:
- Name: come up with a connection name, for example:
iceberg_s3. - hostname:port of the Iceberg Metastore service: the connection address of the created Iceberg Metastore instance in the format
<IP ADDRESS>:<PORT>. You can view the Iceberg Metastore service address in the instance card on the General Information tab in the Postgres primary connection string field. - Iceberg Metastore Service Login: the administrator login specified when creating the Iceberg Metastore service instance during the basic configuration deployment. In this example:
vkdata. - Iceberg Metastore Service Password: the password of the database user account specified when creating the Iceberg Metastore instance.
- Bucket: specify the bucket name for creating the S3 object storage, following the bucket naming rules.
- Name: come up with a connection name, for example:
-
-
-
Click the Next Step button.
-
At the Credentials step:
-
Leave the default value in the Login field. Write down the administrator login for accessing Cloud Trino; you will need it later.
-
In the Password field, click the Generate button to generate an administrator password. Save the password; you will need it later.
-
-
Click the Create button.
Wait for the Cloud Trino service instance creation process to complete.
After the Trino service is deployed, two connections to virtual databases will be automatically added to the configuration for generating large volumes of synthetic data and testing: otpcds and otpch.