Comments (4)
@mensfeld no sarcasm intended! We built a fair amount of application specific abstraction on top of Karafka’s base classes. I’ll make it a priority to upgrade to 1.3 as there is other functionality we’re eager to make use of. I appreciate the support, I’ll reach out if we have any issues 👍
from waterdrop.
This functionality is already implemented in Waterdrop 2.0-alpha: https://github.com/karafka/waterdrop/tree/2.0 and won't be backported to 1.3. Sorry :(
Waterdrop 2.0 supports promises:
producer = WaterDrop::Producer.new
producer.setup do |config|
config.kafka = { 'bootstrap.servers': 'localhost:9092' }
end
promise = producer.produce_async(topic: 'my-topic', payload: 'my message')
result = promise.wait
result.partition #=> 0
result.offset #=> 2065
result.error #=> nil
promise = producer.produce_async(topic: 'invalid-topic', payload: 'my message')
# On the wait you might as well get a timeout error if the delivery takes a lot of time
result = promise.wait
result.partition #=> 2
result.offset #=> 100
# This is not yet merged to rdkafka due to some internals problems with CI that we're trying to fix
result.error #=> "Some sort of error from librdkafka"
from waterdrop.
@mensfeld I appreciate the quick response! We're currently pinned to Karafka 1.2 as 1.3 resulted in breaking changes. This is yet another reason for us to resolve and upgrade. Thanks again 👍
from waterdrop.
I appreciate the quick response!
@nickoliver86 dunno if that was irony or not 🤔 ;) took me a while as I had sick leave.
Sorry for the fact, that 1.3
has breaking changes :( I do my best to keep as much of the backward compatibility and the API intact but that is not always an option :(
In case you didn't see: https://mensfeld.pl/2019/09/karafka-framework-1-3-0-release-notes-ruby-kafka/ - you can also reach out to me in case you need any help in the upgrade process.
I highly recommend upgrading to 1.3
as I plan to release Waterdrop 2.0
and make a patch to make it work with Karafka 1.3
(while Karafka 2.0
is under construction).
from waterdrop.
Related Issues (20)
- Move payload reporting into debug for logger
- Allow waterdrop to use topic as a symbol or a string
- Expose `#flush`, `#purge` and `#close!` (combination of those) once karafka-rdkafka released
- Commit transaction should not kick in when transaction is aborted
- Do not publish error messages on purged messages that were part of an aborted transaction (if possible)
- Buffered client transactions on abort store messages in the internal accu
- Make dummy and buffer clients return incremental offsets
- Expand instrumentation views in DD with scale (ms vs micro)
- Make WaterDrop Buffered `#messages_for` return arrah instead of `nil` when nothing dispatched HOT 1
- Provide ability to pass trace_id (name tba) when producing to use within the delivery handler and report HOT 1
- Support transactional offset metadata storage
- Support getting partition count data from producer for custom partitioners HOT 1
- Document possibility of using partitions count for custom partitioners
- Produce lifecycle events similar to karafka
- Trying to understand recommendation against at_exit HOT 2
- Revisit the librdkafka `delivery.timeout.ms` vs. max wait time
- Introduce `instrument_on_wait_queue_full` for ability to disable instrumentation on non-critical queue full errors
- Document `instrument_on_wait_queue_full` in the docs on retries
- Transactional producer config not aligned by default
- Middleware runs twice when using buffered produce
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from waterdrop.