Creating a Spark cluster
This article provides an example of creating a Spark cluster on the VK Cloud platform using Terraform.
The cluster creation uses:
- the vkcs_mlplatform_spark_k8s resource;
- the vkcs_mlplatform_k8s_registry resource;
- the vkcs_networking_network resource;
- the vkcs_compute_flavor data source;
- the vkcs_networking_network data source.
For a full description of the parameters, see the Terraform provider documentation.
-
Check the quotas. Make sure that the selected region has enough resources to create a CDN resource. Different quotas may be configured for different regions.
If necessary, increase the quotas.
-
Install Terraform and configure the provider, if this has not been done yet.
-
Make sure that the
vkcs_provider.tffile specifies provider version 0.7.0 or higher. If the provider version is lower, update the provider.
In the example below, the cluster is created with the following configuration:
- default region, availability zone
ME1; - minimum number of worker nodes — 2, maximum number of worker nodes — 100;
- cluster operating mode —
DEV; - cluster transition to sleep mode after 120 minutes of inactivity;
- cluster deletion after 1440 minutes of inactivity.
Create a Terraform configuration file main.tf with the following content:
locals {spark_configuration = {"spark.eventLog.dir" = "s3a://spark-bucket""spark.eventLog.enabled" = "true"}spark_environment_variables = {"ENV_VAR_1" : "env_var_1_value""ENV_VAR_2" : "env_var_2_value"}}resource "vkcs_mlplatform_spark_k8s" "spark_k8s" {name = "tf-example"availability_zone = "ME1"network_id = vkcs_networking_network.app.idsubnet_id = vkcs_networking_subnet.app.idnode_groups = [{node_count = 2flavor_id = data.vkcs_compute_flavor.basic.idautoscaling_enabled = truemin_nodes = 2max_nodes = 100}]cluster_mode = "DEV"registry_id = vkcs_mlplatform_k8s_registry.k8s_registry.idip_pool = data.vkcs_networking_network.extnet.idsuspend_after_inactive_min = 120delete_after_inactive_min = 1440spark_configuration = yamlencode(local.spark_configuration)environment_variables = yamlencode(local.spark_environment_variables)}
Here:
-
network_id— the ID of the network where the cluster will be placed. The cluster can be placed in an existing network or in a new one. The ID can be specified in the manifest or obtained from a data source or resource. -
flavor_id— the ID of the VM flavor. The ID can be specified in the manifest or obtained from a data source. -
registry_id— the ID of the Docker registry whose images will be used when running Spark jobs. You can create a new registry or use an existing one. -
ip_pool— the ID of the external network for the cluster IP address pool. -
spark_configuration— a list of properties responsible for Spark configuration. -
environment_variables— a list of environment variables for Spark.
Create a Terraform configuration file registry.tf with a description of the K8S Docker Registry instance:
resource "vkcs_mlplatform_k8s_registry" "k8s_registry" {name = "tf-example"admin_name = "admin"admin_password = "Password12!Password"flavor_id = data.vkcs_compute_flavor.basic.idavailability_zone = "ME1"boot_volume = {volume_type = "ceph-ssd"}networks = [{network_id = vkcs_networking_network.app.idip_pool = data.vkcs_networking_network.extnet.id},]}
Here:
-
flavor_id— the ID of the VM flavor. The ID can be specified in the manifest or obtained from a data source. -
network_id— the ID of the network where the instance will be placed. The instance can be placed in an existing network or in a new one. The ID can be specified in the manifest or obtained from a data source or resource. -
ip_pool— the ID of the external network for the cluster IP address pool.
Create a Terraform configuration file flavor.tf with a description of the desired VM flavor:
data "vkcs_compute_flavor" "basic" {name = "STD2-4-4"}
The VM flavor specified in the file will be used to create instances in your Terraform project.
Create a Terraform configuration file network.tf with a description of the network infrastructure for the cluster:
Example of a data source description for a network that already exists in the project:
data "vkcs_networking_network" "default" {name = "default"sdn = "neutron"}
-
Place the Terraform configuration files in one directory:
vkcs_provider.tf;main.tf;registry.tf;flavor.tf(if it was created);network.tf(if it was created).
-
Navigate to this directory.
-
Make sure the configuration files are correct and contain the necessary changes:
terraform validate && terraform plan -
Apply the changes:
terraform applyWhen prompted for confirmation, enter
yes. -
Wait for the operation to complete.
Make sure that the Spark cluster has been successfully created:
- Go to the VK Cloud management console.
- Go to the ML Platform → Instances section. Make sure the Spark cluster is created and active.
If the resources created with Terraform are no longer needed, delete them:
-
Navigate to the directory with the Terraform configuration files.
-
Run the command:
terraform destroyWhen prompted for confirmation, enter
yes. -
Wait for the operation to complete.