timestamp missing with kinesis output about amazon-kinesis-firehose-for-fluent-bit HOT 8 CLOSED

@UladzimirSemiankou @ScOut3R @arthurmauvezin @raskad Released in 2.2.0: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.2.0

from amazon-kinesis-firehose-for-fluent-bit.

@UladzimirSemiankou curious if you're using the aws-for-fluent-bit image or the fluent/fluent-bit image? Because the systemd input plugin is broken in aws-for-fluent-bit right now - aws/amazon-cloudwatch-logs-for-fluent-bit#10 - though we'll release a fix for it soon.

Can anything be done about this?

I've seen some Fluentd plugins that have a config option like include_time_key, for example, the CloudWatch Logs Fluentd plugin. We need to add something like this to the plugin.

I think this is a high priority feature request.

from amazon-kinesis-firehose-for-fluent-bit.

I second this request. I tried this plugin today with the ECS FireLens preview. A timestamp field would be really nice.
I dont know how feasable it is to add the ability to specify the timestamp format, but that would be really nice too.

from amazon-kinesis-firehose-for-fluent-bit.

@PettitWesley i've built the plugin on top of fluent/fluent-bit image, systemd works fine there

from amazon-kinesis-firehose-for-fluent-bit.

any updates on this?

from amazon-kinesis-firehose-for-fluent-bit.

Ran into this the other day. Timestamp disappears or if I manage to pass it along with Time_Keep it is treated as string. Using AWS ElasticSearch as the destination for Firehose and ECS as input.

from amazon-kinesis-firehose-for-fluent-bit.

I just tried with amazon-kinesis-for-fluent-bit plugin and amazon-kinesis-streams-for-fluent-bit and it has the same effect. Timestamp is present locally in stdout but there is no date or time received by AWS Elasticsearch.

I use both these plugins in amazon/aws-for-fluent-bit:latest Docker image.

My configuration file:

[SERVICE]
    Flush           5
    Daemon          off
    Log_Level       debug
    Parsers_File /fluent-bit/parsers/parsers.conf

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[FILTER]
    Name parser
    Match *
    Parser nginx
    Key_Name log
    Preserve_Key True
    Reserve_Data True

[OUTPUT]
    Name   stdout
    Match  *

[OUTPUT]
    Name   firehose
    Match  *
    region eu-west-1
    delivery_stream my-log-test

[OUTPUT]
    Name            kinesis
    Match           *
    region          eu-west-1
    stream          my-log-test-stream

from amazon-kinesis-firehose-for-fluent-bit.

@UladzimirSemiankou @ScOut3R @arthurmauvezin @raskad Apologies for how long it took us to get to this; see the comment here for the planned config options: #20 (comment)

from amazon-kinesis-firehose-for-fluent-bit.