Code Monkey home page Code Monkey logo

mirrormaker_topic_rename's Introduction

Message Handler For MirrorMaker to change topic names

MirrorMaker is the de-facto standard tool for mirroring Kafka topics between different clusters. By default it will always keep the topic name the same between clusters, but there can be cases in which it is necessary to change a topic name during the mirroring process. This repository shows how this can easily be achieved by implementing a custom MessageHandler and using this with MirrorMaker.

This work is based on code from Gwen Shapiras example repository, so most of the credit goes to her.

Setup

To use this code with MirrorMaker simply build the jar from this repository and put it on the machine that you will be running MirrorMaker from.

Usage

Before starting MirrorMaker you need to include the jar file in your classpath:

export CLASSPATH=/home/sliebau/mmchangetopic-1.0-SNAPSHOT.jar

Having done this, you can start MirrorMaker with your usual configuration and just add the handler classname as well as a configuration string to tell the handler, which topics to rename and which to leave as is.

Implemented Handlers

This repository contains two MessageHandler implementations, which are explained below.

RenameTopicMessageHandler

This handler allows changing the name of the topic in the target cluster that data is written to. It needs to be configured via parameters passed to the command line of Mirror Maker.

The two parameters you will need to add are:

--message.handler:

This takes the classname of the handler class to use, if you have not changed anything in the code the value from the example below should work.

--message.handler.args:

This is used to configure which topics to change, it should have the following format: sourcetopic1,targettopic1;sourcetopic2,targettopic2;...

It is a semicolon separated list of string pairs, which are itself separated by a comma. In the above example any message from the topic sourcetopic1 would be mirrored to targettopic1 (and the same with 2) on the target cluster. Any other topics that MirrorMaker is following will not be changed and written to a topic of the same name on the target cluster.

kafka-mirror-maker --consumer.config consumer.properties --producer.config producer.properties --whitelist test_.* --message.handler com.opencore.RenameTopicHandler --message.handler.args `test_source,test_target;test_source2,test_target2`

ExactMessageHandler

This message handler tries to create as close a copy of the original message during mirroring as possible. The most notable change over the default message handler is, that partitioning based on the key is bypassed completely. Instead the partititon number from the original message is taken and used on the target topic as well, thus ensuring that the mirror topic has the exact same partitioning as the source topic.

This will create Exceptions, if the target topic has a different partition count than the source topic, as MirrorMaker will either try to write to non-existing partitions (target < source) or leave partitions empty (target > source).

This message handler takes no configuration.

mirrormaker_topic_rename's People

Contributors

fernandoney avatar jbresciani avatar nightscape avatar soenkeliebau avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

mirrormaker_topic_rename's Issues

You don't copy message header

Just started using mirrormaker_topic_rename and our application immediately became dead. During investigation we've recognized that the message headers lost during replication.

Messages are not being replicated on to consumer topic

MirrorMakerHandler.zip
We are trying to use the same from Gwen Shapiras , but consumer topic is not receiving any messages on target side.
export CLASSPATH=/Users/sxe518/Documents/test/MirrorMakerHandler/target/TopicSwitchingHandler-1.0-SNAPSHOT.jar
bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist test_.* —message.handler com.shapira.examples.TopicSwitchingHandler --message.handler.args ‘test,test123’

No license

I'd like to use this project internally at work. I've already tested it and it does precisely what I need. It would help greatly in getting official approval from the higher-ups if you could add a license to this repo.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.