[Feature]: Support RocketMQ Option After Streaming Service Finished about milvus HOT 7 OPEN

we need implement the following interface:

• message.MessageID
• walimpls.OpenerBuilderImpls
• walimpls.OpenerImpls
• walimpls.ScannerImpls
• walimpls.WALImpls

from milvus.

@CocaineCong In Milvus 2.4.5, we will introduce wal service instead of original mqwrapper interface. new design will be given at #33285.

And we want a more simple interface of wal instead of mqwrapper. Read + Append instead of Produce and Consume.

Recently, the interface definition PR #33745 will be merged. You can see and review the definition in README, #33745 (comment). And then I will give a new PR as an example to reimplement pulsar mq into wal.

Could you help to modify rocketMQ proposal after those PR merged? rocketMQ can be implement as the wal interface directly, and to be published at v2.4.5.

ok, I got it. anything need my help just let me know.

from milvus.

related PR is ready at master branch: #34046

You can design and implement wal services based on RocketMQ now. pulsar implementation is here, pkg/streaming/walimpls/impls/pulsar/.

If the PR pass the walimpls.NewWALImplsTestFramework unit test, we can start to review it. And please link your PR with issue #33285.

Moreover, ScannerImpls interface should be implemented as a scanner without any server-side persistent state, such as Pulsar's Reader interface to avoid resource leak. If RocketMQ does not support this, please let me know. I will design scanner's garbage collection plan in future.

/assign @CocaineCong

got it.

from milvus.

If this proposal passes, please assign it to me.🫡

from milvus.

@chyezh
could you helo on reviewing it?

from milvus.

@CocaineCong
In Milvus 2.4.5, we will introduce wal service instead of original mqwrapper interface.
new design will be given at #33285.

And we want a more simple interface of wal instead of mqwrapper.
Read + Append instead of Produce and Consume.

Recently, the interface definition PR #33745 will be merged.
You can see and review the definition in README, #33745 (comment).
And then I will give a new PR as an example to reimplement pulsar mq into wal.

Could you help to modify rocketMQ proposal after those PR merged?
rocketMQ can be implement as the wal interface directly, and to be published at v2.4.5.

from milvus.

related PR is ready at master branch: #34046

You can design and implement wal services based on RocketMQ now.
pulsar implementation is here, pkg/streaming/walimpls/impls/pulsar/.

If the PR pass the walimpls.NewWALImplsTestFramework unit test, we can start to review it. And please link your PR with issue #33285.

Moreover, ScannerImpls interface should be implemented as a scanner without any server-side persistent state, such as Pulsar's Reader interface to avoid resource leak. If RocketMQ does not support this, please let me know. I will design scanner's garbage collection plan in future.

/assign @CocaineCong

from milvus.