Comments (3)
I have few issues :
- Topic metadata does not replicate to destination when brooklin datastream is setup. And neither it replicate modifications in same. What I can configure to do same ?
- How can I configure change in topic name in destination? Fo example, in source , I have a topic name first-topic and I have a datastream setup to mirror it in destination cluster with topic name as first-topic.global ?
As @configOps mention bove in 1st point, I found/have some more issues/doubt with KafkaMirror :
- topic partitions are not creating on destination. I used REST endpoint to create a datastream with metadata "system.destination.identityPartitioningEnabled": "true".
- after creating datastream, new messages only replicating to destination.
- if partition increased on source side then destination should also get this change.
- auto topic creation is disabled in our datacenter. so on destination need to create topic first before pushing data.
- Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old verssion?
This project is really nice, and lot of things we are learning from it.... Thanks a lot.
from brooklin.
MirrorMaker 1 doesn't replicate topic metadata either (configs or partitions). It requires destination topic manually be made to match (which is easily done programmatically on your own).
I'm not aware of any rename feature in Brooklin, but I would like to think it does exist in some manner. What research have you done for the same?
Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old version?
Can you please clarify your concerns here? What happens when you try it?
from brooklin.
Hi,
First - thank you for this project, it is really great.
As a follow up to this question i wanted to ask if there are plans to add the ability of changing topic names (to mirror topics to different topic names in destination cluster).
I tried lookin at the documentation and the code, but did not manage to find such ability.
We currently need to use additional clusters to overcome this issue, and it would be great if we can do this with configuration.
from brooklin.
Related Issues (20)
- NPE at AbstractKafkaConnector.stopTask HOT 1
- InstanceAlreadyExistsException: kafka.producer:type=app-info,id=brooklin-producer-1 HOT 2
- Brooklin takes a long time to recover from errors in the destination kafka cluster, up to a complete halt of replication
- Brooklin KAFKA mirroring task produces duplicated records on re-balance HOT 3
- Getting connection refused when trying to run https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka HOT 13
- Inclusive language
- Start Offsets not taking effect in KafkaConnectorTask
- Brooklin does not alllow multiple datastreams replicating to the same destination Kafka topic
- Messages duplicated after container restart
- Error streaming topic with cleanup policy: compact,delete
- build fails on AArch64, Fedora 33 HOT 1
- Producer is closed forcefully loop
- Log4j Critical Vulnerability - Remote Code Injection HOT 1
- Invalid negative sequence number used
- git script uses deprecated system python
- understanding the deletion of datastream and recreating HOT 1
- Question regarding the 1.1.0 BMM version
- Brooklin having issue connecting with zk server with java 21 runtime HOT 3
- brooklin can't connect to zk when starting up HOT 6
- TaskPrefix assignment does not follow 'system.reuseExistingDestination' flag.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from brooklin.