Comments (4)
Don't know how to add it, in this place :
Datastream.pdsc
{
"name": "metadata",
"doc": "Generic metadata for Datastream (e.g. owner, expiration, etc). Metadata is stored as user defined name/value pair.",
"type": {
"type": "map",
"values": "string"
},
"optional": true
}
from brooklin.
You can set this option in datastream metadata, e.g.
bin/brooklin-rest-client.sh -o CREATE -m '{"system.destination.identityPartitioningEnabled":"true","owner":"test-user","system.reuseExistingDestination":"false"}' -u http://localhost:32311/ -n first-mirroring-stream -s "kafka://localhost:9093/^(first|second)-topic$" -c kafkaMirroringConnector -t kafkaTransportProvider 2>/dev/null
from brooklin.
My test steps are as follows: first, create a topic with three partitions in the source kafka and execute it as you do:"./brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream-nine -s "kafka://10.124.201.51:9092,10.124.201.52:9092,10.124.201.53:9092/bigdata_test_topic_nine" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"system.destination.identityPartitioningEnabled":"true","owner":"test-user","system.reuseExistingDestination":"false"}' 2>/dev/null
",
but the topic of the target kafka has only one partition.
Should three partitions be created in advance in the destination kafka?
from brooklin.
Yes, topic needs to be created with correct partition count in advance.
from brooklin.
Related Issues (20)
- NPE at AbstractKafkaConnector.stopTask HOT 1
- InstanceAlreadyExistsException: kafka.producer:type=app-info,id=brooklin-producer-1 HOT 2
- Brooklin takes a long time to recover from errors in the destination kafka cluster, up to a complete halt of replication
- Brooklin KAFKA mirroring task produces duplicated records on re-balance HOT 3
- Getting connection refused when trying to run https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka HOT 13
- Inclusive language
- Start Offsets not taking effect in KafkaConnectorTask
- Brooklin does not alllow multiple datastreams replicating to the same destination Kafka topic
- Messages duplicated after container restart
- Error streaming topic with cleanup policy: compact,delete
- build fails on AArch64, Fedora 33 HOT 1
- Producer is closed forcefully loop
- Log4j Critical Vulnerability - Remote Code Injection HOT 1
- Invalid negative sequence number used
- git script uses deprecated system python
- understanding the deletion of datastream and recreating HOT 1
- Question regarding the 1.1.0 BMM version
- Brooklin having issue connecting with zk server with java 21 runtime HOT 3
- brooklin can't connect to zk when starting up HOT 6
- TaskPrefix assignment does not follow 'system.reuseExistingDestination' flag.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from brooklin.