VK Cloud logo

Cluster monitoring

You can monitor the state of a Spark cluster and analyze its operation using the following tools and services:

  • Cloud Logging;
  • Cloud Monitoring;
  • Cloud Alerting;
  • the cluster event log;
  • Spark History Server;
  • methods of the Cloud ML Platform library.

Using Cloud Logging

The Cloud Logging service collects, stores, and displays logs of user applications and VK Cloud services.

To be able to view cluster logs:

  1. Install the logging plugin on the cluster VM.
  2. Wait a few minutes for the data to accumulate.
  3. Go to the VK Cloud management console.
  4. Select the project where the desired cluster is located.
  5. Go to the Monitoring → Logging section.
  6. Configure log filtering in the table.

Using Cloud Monitoring

The Cloud Monitoring service allows you to monitor and analyze the cluster state in real time using specified metrics. The service displays collected metrics data on charts that can be configured to show only the information you need about cluster operation.

To add a cluster monitoring chart, follow the instruction. In the Resource type field, select the Spark service in k8s option.

Using Cloud Alerting

The Cloud Alerting service allows you to configure notifications about changes in key metrics of VK Cloud services.

To receive notifications about cluster operation:

  1. Create a notification channel.
  2. Configure the rules under which notifications will be sent via the created communication channel.

Using the event log

The event log contains messages about critical errors in cluster operation.

To view the event log:

  1. Go to the VK Cloud management console.
  2. Select the project where the desired cluster is located.
  3. Go to the Kubernetes Clusters → Kubernetes Clusters section.
  4. Click the name of the desired cluster.
  5. Go to the Event log tab.

If there were errors in cluster operation, they will be displayed on the tab.

Using Spark History Server

The Spark History Server service collects information about the job execution history and allows you to view application logs in a web interface. The service is connected for each Spark cluster.

To view logs in Spark History Server:

  1. Go to the VK Cloud management console.

  2. Select the project where the desired cluster is located.

  3. Go to the ML Platform → Spark in k8s section.

  4. Click the name of the desired cluster.

  5. On the General data tab, follow the link in the Spark History Server row.

  6. The Spark History Server instance will open. To authorize, enter your credentials:

    • the login and password of the VK Cloud management console — this method is used for old clusters;
    • access token — this method is used for new clusters.

The list of cluster events will be displayed.

Using the Cloud ML Platform library

The Cloud ML Platform library allows you to get lists of cluster events and logs.

Use the following methods:

  • spark_events — get the list of events that occurred in the cluster.
  • spark_job_logs — display the logs of the desired application.
Was this article helpful?