VK Cloud logo

Log replication to object storage using Fluent Bit

Fluent Bit allows you to configure aggregation and replication of logs from distributed systems to the VK Object Storage object storage. Logs are collected from machines, processed, and sent to the VK Object Storage bucket for long-term storage and subsequent analysis by downstream systems.

The following data is used as an example below:

  • Bucket name: fluent-bit-logs
  • Account region: ru-msk (Moscow)
  • VK Object Storage endpoint: https://hb.ru-msk.vkcloud-storage.ru
  • Log file path: /var/log/app/application.log

Preparation steps

  1. Create an account for the VK Object Storage service, if not done yet:

    1. Go to the VK Cloud personal dashboard.

    2. Select a project.

    3. Go to the Object Storage → Accounts section.

    4. Click the Add account or Add button.

    5. Set the account name.

    6. Click the Create button.

    7. In the window that opens, copy and save the access key identifier (Access Key ID) and the secret key (Secret Key).

  2. Create a VK Object Storage bucket named fluent-bit-logs.

  3. Make sure the AWS CLI is installed and configured to work with the created VK Object Storage account. This will be needed to verify the sent logs.

  4. Install Fluent Bit on the machine from which the logs will be collected. Installation instructions are available in the official Fluent Bit documentation.

1. Set up secure storage of credentials

To access VK Object Storage, the Fluent Bit agent uses the access key identifier and the secret key. Do not specify credentials directly in the configuration file — use environment variables.

  1. Create a file with environment variables, for example /etc/fluent-bit/s3-credentials.env:

    export S3_ACCESS_KEY_ID=<ACCESS_KEY_ID>export S3_SECRET_ACCESS_KEY=<SECRET_KEY>

    Here:

    • <ACCESS_KEY_ID> — the access key identifier saved when creating the VK Object Storage account.
    • <SECRET_KEY> — the secret key saved when creating the VK Object Storage account.
  2. Restrict access to the file with credentials:

    chmod 600 /etc/fluent-bit/s3-credentials.env
  3. Configure loading of environment variables before starting Fluent Bit. If Fluent Bit is started via a systemd plugin, add the following to the unit file:

    [Service]EnvironmentFile=/etc/fluent-bit/s3-credentials.env

2. Configure the Fluent Bit configuration

The Fluent Bit configuration consists of three main sections: Input (log source), Filter (processing), and Output (sending to VK Object Storage).

  1. Create or edit the file /etc/fluent-bit/fluent-bit.conf:

    [SERVICE]    Flush         5    Log_Level     info    storage.path  /var/log/flb-storage/    Parsers_File  /etc/fluent-bit/parsers.conf[INPUT]    Name              tail    Path              /var/log/app/application.log    Tag               app.logs    Refresh_Interval  5    Mem_Buf_Limit     50MB    Skip_Long_Lines   On[FILTER]    Name parser    Match app.logs    Key_Name log    Parser app_log_parser    Preserve_Key true    Reserve_Data true[OUTPUT]    Name            s3    Match           app.logs    region          ru-msk    bucket          fluent-bit-logs    endpoint_url    https://hb.ru-msk.vkcloud-storage.ru    access_key      ${S3_ACCESS_KEY_ID}    secret_key      ${S3_SECRET_ACCESS_KEY}    use_put_object  On    s3_key_format   /logs/%Y/%m/%d/%H/%M-%S-$UUID    total_file_size 50M    upload_timeout  10m

    Here:

    • Name — the plugin used. Instead of the tail plugin, you can use the systemd plugin to collect logs from journald. In this case, additional parameters must be added to the [INPUT] block.

    • Path — path to the log file. For example, /var/log/app/application.log is used.

    • Tag — tag for marking the log stream.

    • region — region code of the VK Object Storage account. For example, ru-msk is used.

    • bucket — name of the bucket to which the logs will be sent. For example, fluent-bit-logs is used.

    • endpoint_url — domain of the VK Object Storage service, must match the account region:

      • https://hb.ru-msk.vkcloud-storage.ru — for the Moscow region;
      • https://hb.kz-ast.vkcloud-storage.ru — for the Kazakhstan region.
    • access_key and secret_key — values from the environment variables configured earlier.

    • s3_key_format — format of the file name (object key) in the bucket.

  2. Create the parser file /etc/fluent-bit/parsers.conf:

    [PARSER]    Name        app_log_parser    Format      json    Time_Key    timestamp    Time_Format %Y-%m-%dT%H:%M:%S.%L

3. Set up buffering to handle network loss

If the network connection between Fluent Bit and VK Object Storage is lost, the logs must be stored in a local buffer and sent after the connection is restored. For this purpose, the file system is used as the buffer storage.

  1. Make sure the [SERVICE] section specifies the path to the buffer storage:

    [SERVICE]    storage.path  /var/log/flb-storage/
  2. Add the storage.type parameter to the [INPUT] section:

    [INPUT]    Name              tail    Path              /var/log/app/application.log    Tag               app.logs    Refresh_Interval  5    Mem_Buf_Limit     50MB    Skip_Long_Lines   On    storage.type      filesystem

    Here:

    • storage.type filesystem — enables writing the buffer to disk when RAM overflows.
    • Mem_Buf_Limit — sets the RAM usage limit; when the limit is exceeded, data is written to disk.
  3. Add the re-sending parameters to the [OUTPUT] section:

    [OUTPUT]    Name            s3    Match           app.logs    region          ru-msk    bucket          fluent-bit-logs    endpoint_url    https://hb.ru-msk.vkcloud-storage.ru    access_key      ${S3_ACCESS_KEY_ID}    secret_key      ${S3_SECRET_ACCESS_KEY}    use_put_object  On    s3_key_format   /logs/%Y/%m/%d/%H/%M-%S-$UUID    total_file_size 50M    upload_timeout  10m    Retry_Limit     5    storage.type    filesystem

    Here:

    • Retry_Limit — the number of re-sending attempts in case of an error.
    • storage.type filesystem — uses the file system to buffer the output stream data.
  4. Create the directory for the buffer and set the permissions:

    mkdir -p /var/log/flb-storage/chown fluent:fluent /var/log/flb-storage/

4. Set up the file naming format

The file naming format (key format) determines how objects in the VK Object Storage bucket will be named. A convenient format allows downstream systems to efficiently parse and query logs by date and time.

The s3_key_format parameter in the [OUTPUT] section specifies the file name template. Available template variables:

  • %Y — year
  • %m — month
  • %d — day
  • %H — hour
  • %M — minute
  • %S — second
  • $UUID — unique file identifier
  • $TAG — log stream tag

Format examples:

# Hierarchical format by date and times3_key_format   /logs/%Y/%m/%d/%H/%M-%S-$UUID# Format with a tag for separating by sourcess3_key_format   /logs/$TAG/%Y/%m/%d/%H-%M-$UUID# Flat format with a dates3_key_format   /logs/%Y%m%d-%H%M%S-$UUID

For example, the hierarchical format /logs/%Y/%m/%d/%H/%M-%S-$UUID is used, which creates a directory structure by date and time:

logs/  2025/    01/      15/        14-30-00-a1b2c3d4-e5f6-7890-abcd-ef1234567890        14-35-00-b2c3d4e5-f6a7-8901-bcde-f12345678901

5. Verify the operation

  1. Start or restart Fluent Bit:

    sudo systemctl restart fluent-bit
  2. Make sure the service is running without errors:

    sudo systemctl status fluent-bit
  3. Check the Fluent Bit logs for errors:

    journalctl -u fluent-bit -f --no-pager
  4. Add a test record to the log file:

    echo '{"timestamp":"2025-01-15T14:30:00.000","level":"info","message":"Test log entry"}' >> /var/log/app/application.log
  5. Wait for the log to be sent to VK Object Storage. The sending time depends on the Flush (5 seconds by default) and upload_timeout (10 minutes for this example) parameters.

  6. Check that the object has appeared in the bucket:

    aws s3 ls s3://fluent-bit-logs/logs/ --recursive --endpoint-url https://hb.ru-msk.vkcloud-storage.ru
  7. (Optional) Download the object and check its contents:

    aws s3 cp s3://fluent-bit-logs/logs/2025/01/15/14/30-00-a1b2c3d4-e5f6-7890-abcd-ef1234567890 - --endpoint-url https://hb.ru-msk.vkcloud-storage.ru

Delete unused resources

The created resources are billed. If you no longer need them:

  1. Delete the objects (log files) from the fluent-bit-logs bucket.

  2. Delete the bucket fluent-bit-logs.

  3. Stop the Fluent Bit service if it is no longer needed:

    sudo systemctl stop fluent-bitsudo systemctl disable fluent-bit
  4. Delete the files with credentials and the buffer:

    rm -f /etc/fluent-bit/s3-credentials.envrm -rf /var/log/flb-storage/
Was this article helpful?