VK Cloud logo

About the service

Apache Spark is a service for distributed processing of large data. It consists of API interfaces for Java, Scala, Python, and R, as well as processing tools Spark SQL for SQL, Pandas API, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. Most often, Spark is used as part of a Hadoop cluster.

Cloud Spark is a solution based on Apache Spark Operator and Managed Containers from VK Cloud. It allows you to deploy Spark inside Kubernetes using an image from Docker Registry, without using a Hadoop cluster.

When to use this service

  • Distributed processing of large data.
  • Reading data from object storage and then exporting it to a DB for processing (ClickHouse, Greenplum, PostgreSQL). Data transfer from a DB to object storage is also possible.
  • Distributed ML model training using large data.
  • Graph computations using the GraphX component.

Service capabilities

  • Deploying a Spark cluster inside Kubernetes.

  • Automatic configuration of master nodes of the deployed clusters.

  • Tracking the execution history of Spark jobs using Spark History Server.

  • Performance analysis, error detection, and cluster state monitoring with the Cloud Logging, Cloud Alerting, and Cloud Monitoring services.

  • Storing objects in the VK Object Storage storage with S3 support.

  • Cluster management, application launching and debugging using the Cloud ML Platform Python library .

Was this article helpful?