VK Cloud logo

Creating a Spark cluster

This article provides an example of creating a Spark cluster on the VK Cloud platform using Terraform.

The cluster creation uses:

For a full description of the parameters, see the Terraform provider documentation.

Before creating the cluster

  1. Check the quotas. Make sure that the selected region has enough resources to create a CDN resource. Different quotas may be configured for different regions.

    If necessary, increase the quotas.

  2. Install Terraform and configure the provider, if this has not been done yet.

  3. Make sure that the vkcs_provider.tf file specifies provider version 0.7.0 or higher. If the provider version is lower, update the provider.

1. Create a file describing the cluster

In the example below, the cluster is created with the following configuration:

  • default region, availability zone ME1;
  • minimum number of worker nodes — 2, maximum number of worker nodes — 100;
  • cluster operating mode — DEV;
  • cluster transition to sleep mode after 120 minutes of inactivity;
  • cluster deletion after 1440 minutes of inactivity.

Create a Terraform configuration file main.tf with the following content:

locals {  spark_configuration = {    "spark.eventLog.dir"     = "s3a://spark-bucket"    "spark.eventLog.enabled" = "true"  }  spark_environment_variables = {    "ENV_VAR_1" : "env_var_1_value"    "ENV_VAR_2" : "env_var_2_value"  }}resource "vkcs_mlplatform_spark_k8s" "spark_k8s" {  name              = "tf-example"  availability_zone = "ME1"  network_id        = vkcs_networking_network.app.id  subnet_id         = vkcs_networking_subnet.app.id  node_groups = [    {      node_count          = 2      flavor_id           = data.vkcs_compute_flavor.basic.id      autoscaling_enabled = true      min_nodes           = 2      max_nodes           = 100    }  ]  cluster_mode = "DEV"  registry_id  = vkcs_mlplatform_k8s_registry.k8s_registry.id  ip_pool      = data.vkcs_networking_network.extnet.id  suspend_after_inactive_min = 120  delete_after_inactive_min  = 1440  spark_configuration   = yamlencode(local.spark_configuration)  environment_variables = yamlencode(local.spark_environment_variables)}

Here:

  • network_id — the ID of the network where the cluster will be placed. The cluster can be placed in an existing network or in a new one. The ID can be specified in the manifest or obtained from a data source or resource.

  • flavor_id — the ID of the VM flavor. The ID can be specified in the manifest or obtained from a data source.

  • registry_id — the ID of the Docker registry whose images will be used when running Spark jobs. You can create a new registry or use an existing one.

  • ip_pool — the ID of the external network for the cluster IP address pool.

  • spark_configuration — a list of properties responsible for Spark configuration.

  • environment_variables — a list of environment variables for Spark.

2. Create a file describing the K8S Docker Registry instance

Create a Terraform configuration file registry.tf with a description of the K8S Docker Registry instance:

resource "vkcs_mlplatform_k8s_registry" "k8s_registry" {  name              = "tf-example"  admin_name        = "admin"  admin_password    = "Password12!Password"  flavor_id         = data.vkcs_compute_flavor.basic.id  availability_zone = "ME1"  boot_volume = {    volume_type = "ceph-ssd"  }  networks = [    {      network_id = vkcs_networking_network.app.id      ip_pool = data.vkcs_networking_network.extnet.id    },  ]}

Here:

  • flavor_id — the ID of the VM flavor. The ID can be specified in the manifest or obtained from a data source.

  • network_id — the ID of the network where the instance will be placed. The instance can be placed in an existing network or in a new one. The ID can be specified in the manifest or obtained from a data source or resource.

  • ip_pool — the ID of the external network for the cluster IP address pool.

3. (Optional) Create a file describing the data source for the VM flavor

Create a Terraform configuration file flavor.tf with a description of the desired VM flavor:

data "vkcs_compute_flavor" "basic" {  name = "STD2-4-4"}

The VM flavor specified in the file will be used to create instances in your Terraform project.

4. (Optional) Create a file describing the network infrastructure for the cluster

Create a Terraform configuration file network.tf with a description of the network infrastructure for the cluster:

Example of a data source description for a network that already exists in the project:

data "vkcs_networking_network" "default" {  name = "default"  sdn = "neutron"}

5. Create the required resources using Terraform

  1. Place the Terraform configuration files in one directory:

    • vkcs_provider.tf;
    • main.tf;
    • registry.tf;
    • flavor.tf (if it was created);
    • network.tf (if it was created).
  2. Navigate to this directory.

  3. Make sure the configuration files are correct and contain the necessary changes:

    terraform validate && terraform plan
  4. Apply the changes:

    terraform apply

    When prompted for confirmation, enter yes.

  5. Wait for the operation to complete.

6. Verify the configuration has been applied

Make sure that the Spark cluster has been successfully created:

  1. Go to the VK Cloud management console.
  2. Go to the ML Platform → Instances section. Make sure the Spark cluster is created and active.

Delete unused resources

If the resources created with Terraform are no longer needed, delete them:

  1. Navigate to the directory with the Terraform configuration files.

  2. Run the command:

    terraform destroy

    When prompted for confirmation, enter yes.

  3. Wait for the operation to complete.

Was this article helpful?