Working with Spark jobs
These methods allow you to manage job execution on a Cloud Spark cluster:
- create, run, and delete jobs;
- get the list of current cluster jobs, job logs, and detailed job information;
- get information about cluster events, including those related to job execution.
Get an example of a standard SparkApplication manifest with a job description for a Cloud Spark cluster and, if necessary, assign a name to this job.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. You can get the list of Cloud Spark clusters and their IDs using the get_clusters method |
(optional) |
| Unique job name for the Cloud Spark cluster. If the argument is omitted, the default value is used: |
An object of the Manifest class that contains an example of a standard SparkApplication manifest. The value of the job_name argument passed in the method call is contained in the metadata.name field of the manifest.
For more information about Spark configuration parameters, see the official documentation.
Get a SparkApplication manifest with a job description for a Cloud Spark cluster from a YAML file.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Path to the YAML file |
An object of the Manifest class.
Save the manifest to a YAML file.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Path to the file |
No return value.
Submit a job to the Cloud Spark cluster.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. You can get the list of Cloud Spark clusters and their IDs using the get_clusters method |
(required) |
| An object of the The object can be obtained using the get_default_manifest or get_manifest_from_yaml_file method |
(optional) |
| Local path to the file with Python code that is passed to the Cloud Spark cluster for execution. If the argument is omitted, the default value is used: |
An object of the K8sSparkJobInfo class with information about the Spark job.
Field | Type | Description |
|---|---|---|
|
| Job name |
|
| Job status. On success — |
| Date and time in RFC3339 format | Job creation time |
|
| Link to Spark History Server, which displays the job progress |
Get the list of all jobs of the Cloud Spark cluster.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. You can get the list of Cloud Spark clusters and their IDs using the get_clusters method |
A list of objects of the K8sSparkJobInfo class with information about the Cloud Spark cluster jobs.
Field | Type | Description |
|---|---|---|
|
| Job name |
|
| Job status. On success — |
| Date and time in RFC3339 format | Job creation time |
|
| Link to Spark History Server, which displays the job progress |
Get detailed information about the Spark job with the specified name.
Required token role: Administrator or User. Learn more about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method |
(required) |
| Spark job name. Spark job names can be obtained using the spark_jobs_list method |
An object of the K8sSparkJobInfoDetail class with detailed information about the Spark job.
Get logs of the Spark job with the specified name.
Required token role: Administrator or User. More about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method |
(required) |
| Spark job name. Spark job names can be obtained using the spark_jobs_list method |
An object of the K8sSparkJobLogs class with the logs of the Spark job.
Get information about a SparkApplication application running on a Cloud Spark cluster to execute the job with the specified name.
Required token role: Administrator or User. More about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method |
(required) |
| Spark job name. Spark job names can be obtained using the spark_jobs_list method |
An object of the K8sSparkApplication class with detailed information about the SparkApplication application running on a Cloud Spark cluster to execute the job.
Get the list of events of a Cloud Spark cluster.
Required token role: Administrator or User. More about token roles.
Argument | Type | Description |
|---|---|---|
(required) |
| Cloud Spark cluster ID. The list of Cloud Spark clusters and their IDs can be obtained using the get_clusters method |
An object of the K8sSparkEventsList class with the list of events of the Cloud Spark cluster.