Comments (1)
For anyone else having a similar issue for kafka mirroring using ssl:
kafkaTransportProvider - this is the destination server
There is no domain config name required, Kafka Producer property names come directly after brooklin.server.transportProvider.kafkaTransporProvider.
It will look like this in brooklin's server.properties:
brooklin.server.transportProvider.kafkaTransportProvider.factoryClassName=com.linkedin.datastream.kafka.KafkaTransportProviderAdminFactory
brooklin.server.transportProvider.kafkaTransportProvider.bootstrap.servers=kafkaserver:port
brooklin.server.transportProvider.kafkaTransportProvider.zookeeper.connect=zookeeper:port/dest
brooklin.server.transportProvider.kafkaTransportProvider.client.id=datastream-producer
brooklin.server.transportProvider.kafkaTransportProvider.security.protocol=ssl
brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.location=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.key.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.location=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.endpoint.identification.algorithm=
brooklin.server.transportProvider.kafkaTransportProvider.ssl.enabled.protocols=TLSv1.2
brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.type=JKS
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.type=JKS
The Kafka Mirroring Connector requires "consumer" domain property after brooklin.server.connector.kafkaMirroringConnector.
It will look like this in brooklin's server.properties:
brooklin.server.connector.kafkaMirroringConnector.factoryClassName=com.linkedin.datastream.connectors.kafka.mirrormaker.KafkaMirrorMakerConnectorFactory
brooklin.server.connector.kafkaMirroringConnector.assignmentStrategyFactory=com.linkedin.datastream.server.assignment.BroadcastStrategyFactory
brooklin.server.connector.kafkaMirroringConnector.consumer.security.protocol=ssl
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.location=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.key.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.location=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.endpoint.identification.algorithm=
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.enabled.protocols=TLSv1.2
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.type=JKS
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.type=JKS
Stream creation looks like this:
brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream -s "kafkassl://localhost:9093/first-topic" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"owner":"test-user","system.reuseExistingDestination":"false"}'
This is now an ssl to ssl configured Kafka Mirror.
HOWEVER, when you query for the description of the stream
brooklin-rest-client.sh -o READALL -u http://localhost:32311/
You will see a regular kakfa connection string, not kafkassl:
{
"name" : "first-mirroring-stream",
"connectorName" : "kafkaMirroringConnector",
"transportProviderName" : "kafkaTransportProvider",
"source" : {
"connectionString" : "kafkassl://localhost:9093/first-topic"
},
"Status" : "READY",
"destination" : {
"connectionString" : "kafka://localhost:9095/*"
},
"metadata" : {
"datastreamUUID" : "aa891768-abbe-44d9-8c4e-18615ef31e91",
"group.id" : "first-mirroring-stream",
"owner" : "test-user",
"system.IsConnectorManagedDestination" : "true",
"system.creation.ms" : "1580139099105",
"system.destination.KafkaBrokers" : "localhost:9095",
"system.reuseExistingDestination" : "false",
"system.taskPrefix" : "first-mirroring-stream"
}
}
This is just a visual bug as mirroring works just fine.
from brooklin.
Related Issues (20)
- NPE at AbstractKafkaConnector.stopTask HOT 1
- InstanceAlreadyExistsException: kafka.producer:type=app-info,id=brooklin-producer-1 HOT 2
- Brooklin takes a long time to recover from errors in the destination kafka cluster, up to a complete halt of replication
- Brooklin KAFKA mirroring task produces duplicated records on re-balance HOT 3
- Getting connection refused when trying to run https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka HOT 13
- Inclusive language
- Start Offsets not taking effect in KafkaConnectorTask
- Brooklin does not alllow multiple datastreams replicating to the same destination Kafka topic
- Messages duplicated after container restart
- Error streaming topic with cleanup policy: compact,delete
- build fails on AArch64, Fedora 33 HOT 1
- Producer is closed forcefully loop
- Log4j Critical Vulnerability - Remote Code Injection HOT 1
- Invalid negative sequence number used
- git script uses deprecated system python
- understanding the deletion of datastream and recreating HOT 1
- Question regarding the 1.1.0 BMM version
- Brooklin having issue connecting with zk server with java 21 runtime HOT 3
- brooklin can't connect to zk when starting up HOT 6
- TaskPrefix assignment does not follow 'system.reuseExistingDestination' flag.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from brooklin.