Code Monkey home page Code Monkey logo

Comments (1)

TheKnowles avatar TheKnowles commented on August 18, 2024

For anyone else having a similar issue for kafka mirroring using ssl:

kafkaTransportProvider - this is the destination server

There is no domain config name required, Kafka Producer property names come directly after brooklin.server.transportProvider.kafkaTransporProvider.

It will look like this in brooklin's server.properties:

brooklin.server.transportProvider.kafkaTransportProvider.factoryClassName=com.linkedin.datastream.kafka.KafkaTransportProviderAdminFactory
brooklin.server.transportProvider.kafkaTransportProvider.bootstrap.servers=kafkaserver:port
brooklin.server.transportProvider.kafkaTransportProvider.zookeeper.connect=zookeeper:port/dest
brooklin.server.transportProvider.kafkaTransportProvider.client.id=datastream-producer

brooklin.server.transportProvider.kafkaTransportProvider.security.protocol=ssl

brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.location=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.key.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.location=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.password=omitted
brooklin.server.transportProvider.kafkaTransportProvider.ssl.endpoint.identification.algorithm=

brooklin.server.transportProvider.kafkaTransportProvider.ssl.enabled.protocols=TLSv1.2
brooklin.server.transportProvider.kafkaTransportProvider.ssl.keystore.type=JKS
brooklin.server.transportProvider.kafkaTransportProvider.ssl.truststore.type=JKS

The Kafka Mirroring Connector requires "consumer" domain property after brooklin.server.connector.kafkaMirroringConnector.

It will look like this in brooklin's server.properties:

brooklin.server.connector.kafkaMirroringConnector.factoryClassName=com.linkedin.datastream.connectors.kafka.mirrormaker.KafkaMirrorMakerConnectorFactory
brooklin.server.connector.kafkaMirroringConnector.assignmentStrategyFactory=com.linkedin.datastream.server.assignment.BroadcastStrategyFactory
brooklin.server.connector.kafkaMirroringConnector.consumer.security.protocol=ssl

brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.location=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.key.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.location=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.password=omitted
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.endpoint.identification.algorithm=

brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.enabled.protocols=TLSv1.2
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.keystore.type=JKS
brooklin.server.connector.kafkaMirroringConnector.consumer.ssl.truststore.type=JKS

Stream creation looks like this:

brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream -s "kafkassl://localhost:9093/first-topic" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"owner":"test-user","system.reuseExistingDestination":"false"}'

This is now an ssl to ssl configured Kafka Mirror.

HOWEVER, when you query for the description of the stream

brooklin-rest-client.sh -o READALL -u http://localhost:32311/

You will see a regular kakfa connection string, not kafkassl:

{
  "name" : "first-mirroring-stream",
  "connectorName" : "kafkaMirroringConnector",
  "transportProviderName" : "kafkaTransportProvider",
  "source" : {
    "connectionString" : "kafkassl://localhost:9093/first-topic"
  },
  "Status" : "READY",
  "destination" : {
    "connectionString" : "kafka://localhost:9095/*"
  },
  "metadata" : {
    "datastreamUUID" : "aa891768-abbe-44d9-8c4e-18615ef31e91",
    "group.id" : "first-mirroring-stream",
    "owner" : "test-user",
    "system.IsConnectorManagedDestination" : "true",
    "system.creation.ms" : "1580139099105",
    "system.destination.KafkaBrokers" : "localhost:9095",
    "system.reuseExistingDestination" : "false",
    "system.taskPrefix" : "first-mirroring-stream"
  }
}

This is just a visual bug as mirroring works just fine.

from brooklin.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.