VK Cloud logo

Application codeBeta

To connect to a deployed Cloud Kafka instance from application code, use a client library for your programming language. The following examples cover Java, Scala, Python, Go, C / C++, .NET, and Node.js.

Prepare to connect

  1. Go to your VK Cloud account.

  2. Go to Data Platform → Service instances.

  3. Click the Cloud Kafka instance name.

  4. On the General information tab, copy the connection string for the option you need:

    • Bootstrap connection string with TLS — for a TLS connection;
    • Bootstrap connection string — for a connection without TLS.
  5. Download the certificate chain for a TLS connection: on the General information tab, click ••• → Download certificate chain in the certificate card.

  6. Have the username and password for your Cloud Kafka account ready. You can add an account on the service instance page.

Continue with the instructions for your programming language:

Java

To connect with Java, use the kafka-clients library. The minimum JDK version is 11.

  1. Add a dependency to your Java project using Maven or Gradle:

    <dependency>    <groupId>org.apache.kafka</groupId>    <artifactId>kafka-clients</artifactId>    <version>4.1.0</version></dependency>
  2. Create CheckKafka.java with the connection code in the Java source directory of your project. In a standard Maven or Gradle project, this is src/main/java. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    import java.util.Properties;import org.apache.kafka.clients.admin.AdminClient;public class CheckKafka {    public static void main(String[] args) throws Exception {        Properties props = new Properties();        props.put("bootstrap.servers", "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>");        props.put("security.protocol", "SASL_SSL");        props.put("sasl.mechanism", "SCRAM-SHA-512");        props.put("sasl.jaas.config",                "org.apache.kafka.common.security.scram.ScramLoginModule required " +                "username=\"<USERNAME>\" password=\"<PASSWORD>\";");        props.put("ssl.truststore.type", "PEM");        props.put("ssl.truststore.location", "<CERTIFICATE_CHAIN_PATH>");        props.put("ssl.endpoint.identification.algorithm", "");        try (AdminClient admin = AdminClient.create(props)) {            System.out.println("Брокеры: " + admin.describeCluster().nodes().get());        }    }}

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  3. Run the CheckKafka class using your IDE or build system. If the program displays broker information without an error, the client has connected to Cloud Kafka.

Scala

To connect with Scala, use the kafka-clients library. The minimum JDK version is 11.

  1. Add a dependency to build.sbt:

    libraryDependencies += "org.apache.kafka" % "kafka-clients" % "4.1.0"
  2. Create CheckKafka.scala with the connection code in the src/main/scala directory of your SBT project. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    import java.util.Propertiesimport org.apache.kafka.clients.admin.AdminClientobject CheckKafka {  def main(args: Array[String]): Unit = {    val props = new Properties()    props.put("bootstrap.servers", "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>")    props.put("security.protocol", "SASL_SSL")    props.put("sasl.mechanism", "SCRAM-SHA-512")    props.put("sasl.jaas.config",      "org.apache.kafka.common.security.scram.ScramLoginModule required " +        "username=\"<USERNAME>\" password=\"<PASSWORD>\";")    props.put("ssl.truststore.type", "PEM")    props.put("ssl.truststore.location", "<CERTIFICATE_CHAIN_PATH>")    props.put("ssl.endpoint.identification.algorithm", "")    val admin = AdminClient.create(props)    try {      println(s"Брокеры: ${admin.describeCluster().nodes().get()}")    } finally {      admin.close()    }  }}

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  3. Run the following command in a terminal from the project directory:

    $ sbt "runMain CheckKafka"

    If the program displays broker information without an error, the client has connected to Cloud Kafka.

Python

To connect with Python, use the confluent-kafka-python client library.

  1. Install the library in a terminal in the same Python environment in which you will run the example:

    $ pip install confluent-kafka
  2. Create check.py with the connection code in the project directory. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    from confluent_kafka.admin import AdminClientconf = {    "bootstrap.servers": "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>",    "security.protocol": "SASL_SSL",    "sasl.mechanism": "SCRAM-SHA-512",    "sasl.username": "<USERNAME>",    "sasl.password": "<PASSWORD>",    "ssl.ca.location": "<CERTIFICATE_CHAIN_PATH>",}metadata = AdminClient(conf).list_topics(timeout=10)print("Брокеров:", len(metadata.brokers))

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  3. Run the following command in a terminal from the directory containing the file:

    $ python check.py

    If the script displays a broker count greater than zero without an error, the client has connected to Cloud Kafka.

Go

To connect with Go, use the Sarama client library.

  1. In a terminal, go to the Go project directory.

  2. Install the dependencies:

    go get github.com/IBM/sarama github.com/xdg-go/scram
  3. Add scram_client.go to your project with code for username and password authentication through Sarama:

    package mainimport (    "crypto/sha512"    "github.com/IBM/sarama"    "github.com/xdg-go/scram")type scramClient struct {    *scram.Client    *scram.ClientConversation    scram.HashGeneratorFcn}func newSCRAMClient() sarama.SCRAMClient {    return &scramClient{HashGeneratorFcn: sha512.New}}func (c *scramClient) Begin(user, password, authzID string) (err error) {    c.Client, err = c.HashGeneratorFcn.NewClient(user, password, authzID)    if err != nil {        return err    }    c.ClientConversation = c.Client.NewConversation()    return nil}func (c *scramClient) Step(challenge string) (string, error) {    return c.ClientConversation.Step(challenge)}func (c *scramClient) Done() bool {    return c.ClientConversation.Done()}
  4. Create main.go with the connection code in the Go module directory. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    package mainimport (    "crypto/tls"    "crypto/x509"    "fmt"    "log"    "os"    "github.com/IBM/sarama")func main() {    config := sarama.NewConfig()    config.Net.SASL.Enable = true    config.Net.SASL.Mechanism = sarama.SASLTypeSCRAMSHA512    config.Net.SASL.User = "<USERNAME>"    config.Net.SASL.Password = "<PASSWORD>"    config.Net.SASL.SCRAMClientGeneratorFunc = newSCRAMClient    caPEM, err := os.ReadFile("<CERTIFICATE_CHAIN_PATH>")    if err != nil {        log.Fatal(err)    }    roots := x509.NewCertPool()    if !roots.AppendCertsFromPEM(caPEM) {        log.Fatal("Файл не содержит сертификатов PEM")    }    config.Net.TLS.Enable = true    config.Net.TLS.Config = &tls.Config{RootCAs: roots}    client, err := sarama.NewClient([]string{"<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>"}, config)    if err != nil {        log.Fatal(err)    }    defer client.Close()    fmt.Println("Брокеров:", len(client.Brokers()))}

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  5. Run the program:

    $ go run .

    If the program displays a broker count greater than zero without an error, the client has connected to Cloud Kafka.

C / C++

To connect with C or C++, use the librdkafka client library. The examples use its C API.

  1. Install the librdkafka development files with TLS and SASL SCRAM support. Run the command in a terminal for your operating system:

    sudo apt update && sudo apt install librdkafka-dev

    The build commands below require an environment where the compiler can access the librdkafka headers and library.

  2. Create check.c with the connection code. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    #include <stdio.h>#include <librdkafka/rdkafka.h>int main(void) {    char errstr[512];    rd_kafka_conf_t *conf = rd_kafka_conf_new();    rd_kafka_conf_set(conf, "bootstrap.servers", "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "security.protocol", "SASL_SSL", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "sasl.mechanism", "SCRAM-SHA-512", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "sasl.username", "<USERNAME>", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "sasl.password", "<PASSWORD>", errstr, sizeof(errstr));    rd_kafka_conf_set(conf, "ssl.ca.location", "<CERTIFICATE_CHAIN_PATH>", errstr, sizeof(errstr));    rd_kafka_t *client = rd_kafka_new(RD_KAFKA_PRODUCER, conf, errstr, sizeof(errstr));    if (client == NULL) {        fprintf(stderr, "Ошибка создания клиента: %s\n", errstr);        return 1;    }    const rd_kafka_metadata_t *metadata;    rd_kafka_resp_err_t err = rd_kafka_metadata(client, 0, NULL, &metadata, 10000);    if (err != RD_KAFKA_RESP_ERR_NO_ERROR) {        fprintf(stderr, "Ошибка подключения: %s\n", rd_kafka_err2str(err));        rd_kafka_destroy(client);        return 1;    }    printf("Брокеров: %d\n", metadata->broker_cnt);    rd_kafka_metadata_destroy(metadata);    rd_kafka_destroy(client);    return 0;}

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  3. In a terminal, compile check.c from its directory and run the resulting program:

    $ gcc check.c -o check -lrdkafka$ ./check

    If the program displays a broker count greater than zero without an error, the client has connected to Cloud Kafka.

.NET

To connect with .NET, use the Confluent.Kafka client library.

  1. Create a console application to check the connection and go to its project directory:

    $ dotnet new console -o kafka-check$ cd kafka-check
  2. Install the library in the project directory:

    $ dotnet add package Confluent.Kafka
  3. Replace the contents of Program.cs in the new project with the connection code. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    using System;using Confluent.Kafka;var config = new AdminClientConfig{    BootstrapServers = "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>",    SecurityProtocol = SecurityProtocol.SaslSsl,    SaslMechanism = SaslMechanism.ScramSha512,    SaslUsername = "<USERNAME>",    SaslPassword = "<PASSWORD>",    SslCaLocation = "<CERTIFICATE_CHAIN_PATH>",};using var admin = new AdminClientBuilder(config).Build();var metadata = admin.GetMetadata(TimeSpan.FromSeconds(10));Console.WriteLine($"Брокеров: {metadata.Brokers.Count}");

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  4. Run the following command in a terminal from the test project directory:

    $ dotnet run

    If the application displays a broker count greater than zero without an error, the client has connected to Cloud Kafka.

Node.js

To connect with Node.js, use the @confluentinc/kafka-javascript client library.

  1. Install the library in the project directory:

    $ npm install @confluentinc/kafka-javascript
  2. Create check.cjs with the connection code in the project directory. Use the TLS option (traffic is encrypted) or the non-TLS option (traffic is not encrypted):

    const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;const config = {    "bootstrap.servers": "<BOOTSTRAP_SERVER_ADDRESS_AND_PORT>",    "security.protocol": "SASL_SSL",    "sasl.mechanisms": "SCRAM-SHA-512",    "sasl.username": "<USERNAME>",    "sasl.password": "<PASSWORD>",    "ssl.ca.location": "<CERTIFICATE_CHAIN_PATH>",};async function checkConnection() {    const admin = new Kafka().admin(config);    await admin.connect();    try {        await admin.listTopics({ timeout: 10000 });        console.log("Подключение к Cloud Kafka установлено");    } finally {        await admin.disconnect();    }}checkConnection().catch((error) => {    console.error(error);    process.exitCode = 1;});

    Here:

    • <BOOTSTRAP_SERVER_ADDRESS_AND_PORT>: the connection string copied during preparation.
    • <USERNAME> and <PASSWORD>: the Cloud Kafka user's credentials.
    • <CERTIFICATE_CHAIN_PATH>: the path to the certificate chain file downloaded from the General information tab during preparation. Used only for TLS connections.
  3. Run the script in a terminal:

    $ node check.cjs

    If the script displays a connection established message without an error, the client has connected to Cloud Kafka.

Was this article helpful?