VK Cloud logo

Working with Spark jobs

These methods allow you to manage job execution on a Cloud Spark cluster:

  • create, run, and delete jobs;
  • get the list of current cluster jobs, job logs, and detailed job information;
  • get information about cluster events, including those related to job execution.

get_default_manifest

Get an example of a standard SparkApplication manifest with a job description for a Cloud Spark cluster and, if necessary, assign a name to this job.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

You can get the list of Cloud Spark clusters and their IDs using the get_clusters method

job_name

(optional)

str

Unique job name for the Cloud Spark cluster.

If the argument is omitted, the default value is used: None.

Return value

An object of the Manifest class that contains an example of a standard SparkApplication manifest. The value of the job_name argument passed in the method call is contained in the metadata.name field of the manifest.

Method signature and usage example

Manifest configuration examples

For more information about Spark configuration parameters, see the official documentation.

get_manifest_from_yaml_file

Get a SparkApplication manifest with a job description for a Cloud Spark cluster from a YAML file.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

yaml_file_path

(required)

str

Path to the YAML file

Return value

An object of the Manifest class.

Method signature and usage example

save_yaml

Save the manifest to a YAML file.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

file_path

(required)

str

Path to the file

Return value

No return value.

Method signature and usage example

spark_submit_job

Submit a job to the Cloud Spark cluster.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

You can get the list of Cloud Spark clusters and their IDs using the get_clusters method

manifest

(required)

Manifest

An object of the Manifest class containing a SparkApplication manifest with a job description.

The object can be obtained using the get_default_manifest or get_manifest_from_yaml_file method

pycode_file_path

(optional)

str

Local path to the file with Python code that is passed to the Cloud Spark cluster for execution.

If the argument is omitted, the default value is used: None

Return value

An object of the K8sSparkJobInfo class with information about the Spark job.

Field

Type

Description

job_name

str

Job name

status

str

Job status. On success — SUBMITTED

created at

Date and time in RFC3339 format

Job creation time

ui_url

str

Link to Spark History Server, which displays the job progress

Method signature and usage example

spark_jobs_list

Get the list of all jobs of the Cloud Spark cluster.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

You can get the list of Cloud Spark clusters and their IDs using the get_clusters method

Return value

A list of objects of the K8sSparkJobInfo class with information about the Cloud Spark cluster jobs.

Field

Type

Description

job_name

str

Job name

status

str

Job status. On success — SUBMITTED

created at

Date and time in RFC3339 format

Job creation time

ui_url

str

Link to Spark History Server, which displays the job progress

Method signature and usage example

spark_job_info

Get detailed information about the Spark job with the specified name.

Required token role: Administrator or User. Learn more about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method

job_name

(required)

str

Spark job name.

Spark job names can be obtained using the spark_jobs_list method

Return value

An object of the K8sSparkJobInfoDetail class with detailed information about the Spark job.

Method signature and usage example

spark_job_logs

Get logs of the Spark job with the specified name.

Required token role: Administrator or User. More about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method

job_name

(required)

str

Spark job name.

Spark job names can be obtained using the spark_jobs_list method

Return value

An object of the K8sSparkJobLogs class with the logs of the Spark job.

Method signature and usage example

describe_spark_job

Get information about a SparkApplication application running on a Cloud Spark cluster to execute the job with the specified name.

Required token role: Administrator or User. More about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method

job_name

(required)

str

Spark job name.

Spark job names can be obtained using the spark_jobs_list method

Return value

An object of the K8sSparkApplication class with detailed information about the SparkApplication application running on a Cloud Spark cluster to execute the job.

Method signature and usage example

spark_events

Get the list of events of a Cloud Spark cluster.

Required token role: Administrator or User. More about token roles.

Method arguments

Argument

Type

Description

cluster_id

(required)

str

Cloud Spark cluster ID.

The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method

Return value

An object of the K8sSparkEventsList class with the list of events of the Cloud Spark cluster.

Method signature and usage example

Was this article helpful?