Creating a Spark cluster
This article provides an example of creating a Spark cluster on the VK Data Platform platform using Terraform.
The cluster creation uses:
- the vkcs_dataplatform_cluster resource;
- the vkcs_networking_network resource;
- the vkcs_networking_subnet resource;
- the vkcs_db_instance resource;
- the vkcs_db_database resource;
- the vkcs_db_user resource;
- the vkcs_dataplatform_product data source;
- the vkcs_networking_network data source.
For a full description of the parameters, see the Terraform provider documentation.
-
Check the quotas. Make sure that the selected region has enough resources to create a CDN resource. Different quotas may be configured for different regions.
If necessary, increase the quotas.
-
Install Terraform and configure the environment, if this has not been done yet.
-
Make sure that the
vkcs_provider.tffile specifies provider version 0.7.0 or higher. If the provider version is lower, update the provider.
Create a Terraform configuration file main.tf with the following content:
locals {s3_bucket = "my_bucket"}resource "vkcs_dataplatform_cluster" "basic_spark" {name = "tf-basic-spark"description = "Production Spark cluster for data processing"network_id = vkcs_networking_network.db.idsubnet_id = vkcs_networking_subnet.db.idproduct_name = "spark"product_version = "3.5.1"availability_zone = "MS1"configs = {settings = [{alias = "sparkproxy.spark_version"value = "spark-py-3.5.1:v3.5.1.2"}]maintenance = {start = "0 0 1 * *"}warehouses = [{name = "spark"connections = [{name = "s3_int"plug = "s3-int"settings = [{alias = "s3_bucket"value = local.s3_bucket},{alias = "s3_folder"value = "tfexample-folder"}]},{name = "postgres"plug = "postgresql"settings = [{alias = "db_name"value = vkcs_db_database.postgres_db.name},{alias = "hostname"value = "${vkcs_db_instance.db_instance.ip[0]}:5432"},{alias = "username"value = vkcs_db_user.postgres_user.name},{alias = "password"value = vkcs_db_user.postgres_user.password}]}]}]}pod_groups = [{name = "sparkconnect"count = 1resource = {cpu_request = "10"ram_request = "10"}},{name = "sparkhistory"count = 1resource = {cpu_request = "0.5"ram_request = "1"}volumes = {"data" = {storage_class_name = "ceph-ssd"storage = "5"count = 1}}}]}
Here:
-
network_id— the ID of the network where the cluster will be hosted. The cluster can be hosted in an existing network or in a new one. The ID can be specified in the manifest, or obtained from a data source or resource. -
product_name— the name of the Spark service on the VK Data Platform platform. -
product_version— the version of the Spark service on the VK Data Platform platform. -
availability_zone— the availability zone where the cluster is created. -
configs— the cluster configuration parameters:settings— general parameters. Thesparkproxy.spark_versionparameter defines the Spark version for running jobs.maintenance— the maintenance start time in Cron format.warehouses— connections to data sources.
-
pod_groups— the cluster compute resource groups:sparkconnect— resources for interactive data processing.sparkhistory— resources for storing the Spark job history.
Create a Terraform configuration file product.tf describing the Spark service on the VK Data Platform platform:
data "vkcs_dataplatform_product" "spark" {product_name = "spark"product_version = "3.5.1"}
Using this data source makes the configuration more flexible.
You will be able to reference data.vkcs_dataplatform_product.spark.product_name and data.vkcs_dataplatform_product.spark.product_version in main.tf.
For a full description of the parameters, see the Terraform provider documentation.
Create a Terraform configuration file network.tf describing the network infrastructure for the cluster:
Example of describing a data source for a network that already exists in the project:
data "vkcs_networking_network" "default" {name = "default"sdn = "neutron"}
Create a Terraform configuration file db.tf to describe the database that Spark connects to:
resource "vkcs_db_instance" "db_instance" {name = "db-instance"flavor_id = "STD3-2-8"size = 10volume_type = "ceph-ssd"network {uuid = vkcs_networking_network.db.id}datastore {type = "postgresql"version = "13"}}resource "vkcs_db_database" "postgres_db" {name = "mydatabase"dbms_id = vkcs_db_instance.db_instance.id}resource "vkcs_db_user" "postgres_user" {name = "myuser"password = "securepassword123"dbms_id = vkcs_db_instance.db_instance.id}
-
Place the Terraform configuration files in one directory:
vkcs_provider.tf;main.tf;product.tf(if created);network.tf(if created);db.tf(if created).
-
Navigate to this directory.
-
Make sure that the configuration files are correct and contain the required changes:
terraform validate && terraform plan -
Apply the changes:
terraform applyWhen prompted for confirmation, enter
yes. -
Wait for the operation to complete.
Make sure that the Spark cluster was created successfully:
- Go to the VK Cloud management console.
- Go to the Data Platform → Service instances section. Make sure that the Spark cluster is created and active.
If the resources created with Terraform are no longer needed, delete them:
-
Navigate to the directory with the Terraform configuration files.
-
Run the command:
terraform destroyWhen prompted for confirmation, enter
yes. -
Wait for the operation to complete.