Connecting Cloud Spark as a data processing tool
Including the Cloud Spark service in the basic architecture of a Data Lakehouse (DLH) makes it possible to perform complex data transformations before the data gets into S3 object storage, unlike Cloud Trino, which can implement relatively simple scenarios.
An analytical database, such as Cloud ClickHouse, can additionally be used to store operational information, although such an architecture makes the solution more complex and expensive.
Deploy the basic architecture of the Data Lakehouse service.
-
Navigate to the VK Cloud management console.
-
Select the project where you need to create a service instance.
-
Go to the Data Platform → Service Instances section.
-
Click the Create Instance button.
-
At the Configuration step:
- Select the
Cloud Sparkservice type. - In the Version field, leave the default value.
- Select the
-
Click the Next Step button.
-
At the Parameters step:
-
Set the instance parameters:
- Name: specify a name for the service instance.
- Description: add a description if necessary.
- Assign external IP: enable this option to make the Cloud ClickHouse instance accessible via the Internet.
- Kubernetes cluster: select from the drop-down list the cluster you specified when creating the Cloud Iceberg Metastore service instance at the stage of deploying the basic configuration.
-
In the Spark Connect field, specify the number of CPU cores and the amount of RAM to be allocated for real-time data processing.
-
In the Spark History Server field, specify the number of CPU cores and the amount of RAM to be allocated for logging the launches of Cloud Spark service jobs.
-
-
Click the Next Step button.
-
At the Maintenance step:
- Select the days of the week and the start time of maintenance work. During this time, maintenance work may be performed on the server, including backups — the service will be unavailable.
-
Click the Next Step button.
-
At the Settings step, click the Edit button and select the Cloud Spark version from the list of supported ones — it will be used to run jobs.
-
Click the Next Step button.
-
At the Connections step:
-
Add a connection to the PostgreSQL created when deploying the basic Data Lakehouse configuration:
-
Click the Add Connection button.
-
Select the PostgreSQL connection and specify:
- PostgreSQL: come up with a connection name, for example:
postgres_spark. - Database name: the name of the PostgreSQL source database that Cloud Spark will connect to for query execution and data integration, for example:
ad_db. It is available on the Database List tab after creating the PostgreSQL instance. - DB host name: a value in the format
<IP-ADDRESS>:<PORT>. The connection data must be copied on the PostgreSQL instance page on the General Information tab, from the value of the Postgres primary connection string field:postgres://<user>:<password>@<IP-ADDRESS>:<PORT>/<DB_NAME>. For example:10.0.1.2:5432. - Username: the login for the PostgreSQL created when deploying the basic Data Lakehouse configuration.
- User password: the password of the database user account specified when creating the PostgreSQL DB instance.
- PostgreSQL: come up with a connection name, for example:
-
-
Add a connection to the S3 object storage created when deploying the basic Data Lakehouse configuration:
-
Click the Add Connection button.
-
Select the
S3 VK Cloudconnection. -
Specify the connection parameters:
- Internal S3 connection name: come up with a connection name, for example:
iceberg_s3. - Bucket: the bucket name specified when creating VK Object Storage.
- Path in the S3 bucket: the address where files are located in the S3 storage. The parameter is available on the VK Object Storage instance page.
- Internal S3 connection name: come up with a connection name, for example:
-
-
-
Click the Next Step button.
-
At the Credentials step:
-
In the Login field, leave the default value. Write down the administrator login for accessing Cloud ClickHouse, you will need it later.
-
In the Password field, click the Generate button to generate an administrator password. Save the password, you will need it later.
-
-
Click the Create button.
Wait for the Cloud Spark service instance creation process to complete.