Streams context forward multiple topics? about kafka-streams HOT 5 OPEN

Hi @imperio-wxm,

Sorry for the seriously delayed response. The best way for you to do this is with KStream#branch or in the ProcessorAPI where you can forward to specific child nodes by name. Either way, the output topics need to be defined in the topology ahead of time. Let me know if you need more info.

HTH
Bill

from kafka-streams.

@bbejeck What if we want to create a new record in addition to what we have and forward both to different child processors ? It's like,

 new_record = createNewRecord(old_record);
 context.forward(original_record, to(child_1));
 context.forward(new_record, to(child_2));

Is this a good practice ?

from kafka-streams.

I think that's the API(Kafka) restriction and wouldn't be a good practice.
Instead of doing the above, you could have two topologies as children, and let them obtain the output of old_record

ParentTopo -> NewRecordTopo(createNewRecord) -> sink1
           -> OldRecordTopo -> sink2

This could achieve your goal. @bbejeck maybe could give more insight about why Processor context cannot forward to two processors.

from kafka-streams.

Hi, @bbejeck
Sorry, my reply is late.

I think @psawmora is right.

I do like this, and it works, for a long time I didn't find any problems:

switch(type) {
     case type1:
          context.forward(newKey1, newValue1, TOPIC_1);
          break;
     case type2:
          context.forward(newKey2, newValue2, TOPIC_2);
          break:
     .......
}

I wrote a generic Processor to specifically split the topic. Topological image of a tree.
But source topic can only be one.
Is this a good practice ?

from kafka-streams.

@imperio-wxm

But have you tried forwarding the same flow to two different processors at the same time ?

from kafka-streams.