Log replication to object storage using Fluent Bit
Fluent Bit allows you to configure aggregation and replication of logs from distributed systems to the VK Object Storage object storage. Logs are collected from machines, processed, and sent to the VK Object Storage bucket for long-term storage and subsequent analysis by downstream systems.
The following data is used as an example below:
- Bucket name:
fluent-bit-logs - Account region:
ru-msk(Moscow) - VK Object Storage endpoint:
https://hb.ru-msk.vkcloud-storage.ru - Log file path:
/var/log/app/application.log
-
Create an account for the VK Object Storage service, if not done yet:
-
Go to the VK Cloud personal dashboard.
-
Select a project.
-
Go to the Object Storage → Accounts section.
-
Click the Add account or Add button.
-
Set the account name.
-
Click the Create button.
-
In the window that opens, copy and save the access key identifier (Access Key ID) and the secret key (Secret Key).
-
-
Create a VK Object Storage bucket named
fluent-bit-logs. -
Make sure the AWS CLI is installed and configured to work with the created VK Object Storage account. This will be needed to verify the sent logs.
-
Install Fluent Bit on the machine from which the logs will be collected. Installation instructions are available in the official Fluent Bit documentation.
To access VK Object Storage, the Fluent Bit agent uses the access key identifier and the secret key. Do not specify credentials directly in the configuration file — use environment variables.
-
Create a file with environment variables, for example
/etc/fluent-bit/s3-credentials.env:export S3_ACCESS_KEY_ID=<ACCESS_KEY_ID>export S3_SECRET_ACCESS_KEY=<SECRET_KEY>Here:
<ACCESS_KEY_ID>— the access key identifier saved when creating the VK Object Storage account.<SECRET_KEY>— the secret key saved when creating the VK Object Storage account.
-
Restrict access to the file with credentials:
chmod 600 /etc/fluent-bit/s3-credentials.env -
Configure loading of environment variables before starting Fluent Bit. If Fluent Bit is started via a systemd plugin, add the following to the unit file:
[Service]EnvironmentFile=/etc/fluent-bit/s3-credentials.env
The Fluent Bit configuration consists of three main sections: Input (log source), Filter (processing), and Output (sending to VK Object Storage).
-
Create or edit the file
/etc/fluent-bit/fluent-bit.conf:[SERVICE]Flush 5Log_Level infostorage.path /var/log/flb-storage/Parsers_File /etc/fluent-bit/parsers.conf[INPUT]Name tailPath /var/log/app/application.logTag app.logsRefresh_Interval 5Mem_Buf_Limit 50MBSkip_Long_Lines On[FILTER]Name parserMatch app.logsKey_Name logParser app_log_parserPreserve_Key trueReserve_Data true[OUTPUT]Name s3Match app.logsregion ru-mskbucket fluent-bit-logsendpoint_url https://hb.ru-msk.vkcloud-storage.ruaccess_key ${S3_ACCESS_KEY_ID}secret_key ${S3_SECRET_ACCESS_KEY}use_put_object Ons3_key_format /logs/%Y/%m/%d/%H/%M-%S-$UUIDtotal_file_size 50Mupload_timeout 10mHere:
-
Name— the plugin used. Instead of thetailplugin, you can use thesystemdplugin to collect logs from journald. In this case, additional parameters must be added to the[INPUT]block. -
Path— path to the log file. For example,/var/log/app/application.logis used. -
Tag— tag for marking the log stream. -
region— region code of the VK Object Storage account. For example,ru-mskis used. -
bucket— name of the bucket to which the logs will be sent. For example,fluent-bit-logsis used. -
endpoint_url— domain of the VK Object Storage service, must match the account region:https://hb.ru-msk.vkcloud-storage.ru— for the Moscow region;https://hb.kz-ast.vkcloud-storage.ru— for the Kazakhstan region.
-
access_keyandsecret_key— values from the environment variables configured earlier. -
s3_key_format— format of the file name (object key) in the bucket.
-
-
Create the parser file
/etc/fluent-bit/parsers.conf:[PARSER]Name app_log_parserFormat jsonTime_Key timestampTime_Format %Y-%m-%dT%H:%M:%S.%L
If the network connection between Fluent Bit and VK Object Storage is lost, the logs must be stored in a local buffer and sent after the connection is restored. For this purpose, the file system is used as the buffer storage.
-
Make sure the
[SERVICE]section specifies the path to the buffer storage:[SERVICE]storage.path /var/log/flb-storage/ -
Add the
storage.typeparameter to the[INPUT]section:[INPUT]Name tailPath /var/log/app/application.logTag app.logsRefresh_Interval 5Mem_Buf_Limit 50MBSkip_Long_Lines Onstorage.type filesystemHere:
storage.type filesystem— enables writing the buffer to disk when RAM overflows.Mem_Buf_Limit— sets the RAM usage limit; when the limit is exceeded, data is written to disk.
-
Add the re-sending parameters to the
[OUTPUT]section:[OUTPUT]Name s3Match app.logsregion ru-mskbucket fluent-bit-logsendpoint_url https://hb.ru-msk.vkcloud-storage.ruaccess_key ${S3_ACCESS_KEY_ID}secret_key ${S3_SECRET_ACCESS_KEY}use_put_object Ons3_key_format /logs/%Y/%m/%d/%H/%M-%S-$UUIDtotal_file_size 50Mupload_timeout 10mRetry_Limit 5storage.type filesystemHere:
Retry_Limit— the number of re-sending attempts in case of an error.storage.type filesystem— uses the file system to buffer the output stream data.
-
Create the directory for the buffer and set the permissions:
mkdir -p /var/log/flb-storage/chown fluent:fluent /var/log/flb-storage/
The file naming format (key format) determines how objects in the VK Object Storage bucket will be named. A convenient format allows downstream systems to efficiently parse and query logs by date and time.
The s3_key_format parameter in the [OUTPUT] section specifies the file name template. Available template variables:
%Y— year%m— month%d— day%H— hour%M— minute%S— second$UUID— unique file identifier$TAG— log stream tag
Format examples:
# Hierarchical format by date and times3_key_format /logs/%Y/%m/%d/%H/%M-%S-$UUID# Format with a tag for separating by sourcess3_key_format /logs/$TAG/%Y/%m/%d/%H-%M-$UUID# Flat format with a dates3_key_format /logs/%Y%m%d-%H%M%S-$UUID
For example, the hierarchical format /logs/%Y/%m/%d/%H/%M-%S-$UUID is used, which creates a directory structure by date and time:
logs/2025/01/15/14-30-00-a1b2c3d4-e5f6-7890-abcd-ef123456789014-35-00-b2c3d4e5-f6a7-8901-bcde-f12345678901
-
Start or restart Fluent Bit:
sudo systemctl restart fluent-bit -
Make sure the service is running without errors:
sudo systemctl status fluent-bit -
Check the Fluent Bit logs for errors:
journalctl -u fluent-bit -f --no-pager -
Add a test record to the log file:
echo '{"timestamp":"2025-01-15T14:30:00.000","level":"info","message":"Test log entry"}' >> /var/log/app/application.log -
Wait for the log to be sent to VK Object Storage. The sending time depends on the
Flush(5 seconds by default) andupload_timeout(10 minutes for this example) parameters. -
Check that the object has appeared in the bucket:
aws s3 ls s3://fluent-bit-logs/logs/ --recursive --endpoint-url https://hb.ru-msk.vkcloud-storage.ru -
(Optional) Download the object and check its contents:
aws s3 cp s3://fluent-bit-logs/logs/2025/01/15/14/30-00-a1b2c3d4-e5f6-7890-abcd-ef1234567890 - --endpoint-url https://hb.ru-msk.vkcloud-storage.ru
The created resources are billed. If you no longer need them:
-
Delete the objects (log files) from the
fluent-bit-logsbucket. -
Delete the bucket
fluent-bit-logs. -
Stop the Fluent Bit service if it is no longer needed:
sudo systemctl stop fluent-bitsudo systemctl disable fluent-bit -
Delete the files with credentials and the buffer:
rm -f /etc/fluent-bit/s3-credentials.envrm -rf /var/log/flb-storage/