ACKs with fluent-bit about fluency HOT 8 CLOSED

Thanks for reporting it. I'll try to reproduce this issue.

from fluency.

I had a look at the code:

• I would make the passed around ack token a String (in org.komamitsu.fluency.fluentd.ingester.sender.FluentdSender#sendInternal).
• In org.komamitsu.fluency.fluentd.ingester.sender.RequestOption I would change the type to String
• In org.komamitsu.fluency.fluentd.ingester.FluentdIngester#ingest I would change the chunk generation to

            UUID uuid = UUID.randomUUID();
            ByteBuffer uuidByteBuffer = ByteBuffer.wrap(new byte[16]);
            uuidByteBuffer.putLong(uuid.getMostSignificantBits());
            uuidByteBuffer.putLong(uuid.getLeastSignificantBits());
            byte[] uuidBytes = uuidByteBuffer.array();
            String uuidBase64Encoded = Base64.getEncoder().encodeToString(uuidBytes);

            ByteBuffer optionBuffer = ByteBuffer.wrap(objectMapper.writeValueAsBytes(new RequestOption(dataLength, uuidBase64Encoded)));
            List<ByteBuffer> buffers = Arrays.asList(headerBuffer, dataBuffer, optionBuffer);

            synchronized (sender) {
                sender.sendWithAck(buffers, uuidBase64Encoded);
            }

• The String always holds the base64 encoded token. And when the ack token is received from fluentd, I would just compare the base64 encoded version (I would not convert it back to a UUID, only when required for the UnmatchedAckException exception.

I am happy to put up a PR with those changes if you agree that we got an issue and with my solution proposal ;)

from fluency.

@elruwen Your plan looks good! Thanks.

Let me add a comment on the plan.

This issue happens because fluent-bit expects ACK token called chunk is MessagePack String not Binary at https://github.com/fluent/fluent-bit/blob/bb0c95b4d846dc90b9e4072f081c7fff9a2bdad6/plugins/in_forward/fw_prot.c#L162 and doesn't validate the value itself. So, I don't think Fluency needs to send Base64 format value for now.

Also strictly speaking, in fluent-bit the source of chunk is a SHA-512 of the data sent to in_forward https://github.com/fluent/fluent-bit/blob/bb0c95b4d846dc90b9e4072f081c7fff9a2bdad6/plugins/out_forward/forward_format.c#L96. The 3rd change item you suggested is a bit incomplete and I think we can skip the change since sending Base64 format chunk isn't required for now.

from fluency.

This issue is fixed by #182

from fluency.

This issue is fixed by #182

Thanks for merging it, can we get a release? :)

from fluency.

I'll release the fix in a few days.

from fluency.

Released this change as 2.5.1. It'll be visible in a few hours on Maven Central.

from fluency.

Awesome, thanks for the quick turn-around :)

from fluency.