VK Cloud logo

Creating a Spark cluster

This article provides an example of creating a Spark cluster on the VK Data Platform platform using Terraform.

The cluster creation uses:

For a full description of the parameters, see the Terraform provider documentation.

Preparatory steps

  1. Check the quotas. Make sure that the selected region has enough resources to create a CDN resource. Different quotas may be configured for different regions.

    If necessary, increase the quotas.

  2. Install Terraform and configure the environment, if this has not been done yet.

  3. Make sure that the vkcs_provider.tf file specifies provider version 0.7.0 or higher. If the provider version is lower, update the provider.

1. Create a file with the cluster description

Create a Terraform configuration file main.tf with the following content:

locals {  s3_bucket = "my_bucket"}resource "vkcs_dataplatform_cluster" "basic_spark" {  name            = "tf-basic-spark"  description     = "Production Spark cluster for data processing"  network_id      = vkcs_networking_network.db.id  subnet_id       = vkcs_networking_subnet.db.id  product_name    = "spark"  product_version = "3.5.1"  availability_zone = "MS1"  configs = {    settings = [      {        alias = "sparkproxy.spark_version"        value = "spark-py-3.5.1:v3.5.1.2"      }    ]    maintenance = {      start = "0 0 1 * *"    }    warehouses = [      {        name = "spark"        connections = [          {            name = "s3_int"            plug = "s3-int"            settings = [              {                alias = "s3_bucket"                value = local.s3_bucket              },              {                alias = "s3_folder"                value = "tfexample-folder"              }            ]          },          {            name = "postgres"            plug = "postgresql"            settings = [              {                alias = "db_name"                value = vkcs_db_database.postgres_db.name              },              {                alias = "hostname"                value = "${vkcs_db_instance.db_instance.ip[0]}:5432"              },              {                alias = "username"                value = vkcs_db_user.postgres_user.name              },              {                alias = "password"                value = vkcs_db_user.postgres_user.password              }            ]          }        ]      }    ]  }  pod_groups = [    {      name  = "sparkconnect"      count = 1      resource = {        cpu_request = "10"        ram_request = "10"      }    },    {      name  = "sparkhistory"      count = 1      resource = {        cpu_request = "0.5"        ram_request = "1"      }      volumes = {        "data" = {          storage_class_name = "ceph-ssd"          storage            = "5"          count              = 1        }      }    }  ]}

Here:

  • network_id — the ID of the network where the cluster will be hosted. The cluster can be hosted in an existing network or in a new one. The ID can be specified in the manifest, or obtained from a data source or resource.

  • product_name — the name of the Spark service on the VK Data Platform platform.

  • product_version — the version of the Spark service on the VK Data Platform platform.

  • availability_zone — the availability zone where the cluster is created.

  • configs — the cluster configuration parameters:

    • settings — general parameters. The sparkproxy.spark_version parameter defines the Spark version for running jobs.
    • maintenance — the maintenance start time in Cron format.
    • warehouses — connections to data sources.
  • pod_groups — the cluster compute resource groups:

    • sparkconnect — resources for interactive data processing.
    • sparkhistory — resources for storing the Spark job history.

2. (Optional) Define the Spark service on the VK Data Platform platform via a data source

Create a Terraform configuration file product.tf describing the Spark service on the VK Data Platform platform:

data "vkcs_dataplatform_product" "spark" {  product_name = "spark"  product_version = "3.5.1"}

Using this data source makes the configuration more flexible. You will be able to reference data.vkcs_dataplatform_product.spark.product_name and data.vkcs_dataplatform_product.spark.product_version in main.tf.

For a full description of the parameters, see the Terraform provider documentation.

3. (Optional) Create a file describing the network infrastructure for the cluster

Create a Terraform configuration file network.tf describing the network infrastructure for the cluster:

Example of describing a data source for a network that already exists in the project:

data "vkcs_networking_network" "default" {  name = "default"  sdn = "neutron"}

4. (Optional) Create a file describing the database for connecting to Spark

Create a Terraform configuration file db.tf to describe the database that Spark connects to:

resource "vkcs_db_instance" "db_instance" {  name        = "db-instance"  flavor_id   = "STD3-2-8"  size        = 10  volume_type = "ceph-ssd"    network {    uuid = vkcs_networking_network.db.id  }    datastore {    type    = "postgresql"    version = "13"  }}resource "vkcs_db_database" "postgres_db" {  name    = "mydatabase"  dbms_id = vkcs_db_instance.db_instance.id}resource "vkcs_db_user" "postgres_user" {  name     = "myuser"  password = "securepassword123"  dbms_id  = vkcs_db_instance.db_instance.id}

5. Create the required resources using Terraform

  1. Place the Terraform configuration files in one directory:

    • vkcs_provider.tf;
    • main.tf;
    • product.tf (if created);
    • network.tf (if created);
    • db.tf (if created).
  2. Navigate to this directory.

  3. Make sure that the configuration files are correct and contain the required changes:

    terraform validate && terraform plan
  4. Apply the changes:

    terraform apply

    When prompted for confirmation, enter yes.

  5. Wait for the operation to complete.

Verify the configuration was applied

Make sure that the Spark cluster was created successfully:

  1. Go to the VK Cloud management console.
  2. Go to the Data Platform → Service instances section. Make sure that the Spark cluster is created and active.

Delete unused resources

If the resources created with Terraform are no longer needed, delete them:

  1. Navigate to the directory with the Terraform configuration files.

  2. Run the command:

    terraform destroy

    When prompted for confirmation, enter yes.

  3. Wait for the operation to complete.

Was this article helpful?