Code Monkey home page Code Monkey logo

echo-sink's Introduction

Echo Sink Kafka Connect plugin

Echo Sink is a plugin for Kafka Connect. It implements only a sink connector. The Sink connector receives messages from selected topic(s) and logs them using the Kafka Connect logger. The main purpose of this plugin is to test a Kafka Connect installation. I created it to help me during the development of Strimzi Kafka operator for Kubernetes and OpenShift](http://strimzi.io).

Configuration options

Option Description Default
level Defines the log level on which the received messages will be logged. INFO
fail.task.after.records The tasks created by this connector will fail after receiving the specified number of records with an error. This is useful to test things such as status updated at task failures or automatic task restarts. If set to 0 or not set at all, this feature will be disabled and the connector will never fail intentionally. 0
fail.connector.startup The connector will fail at startup. When set to true, the connector instance will never get running. false

Examples

Using REST API

  1. Copy the echo-sink-1.2.0.jar jar file to your Kafka Connect plugin directory
  2. Create a connector instance Kafka Connect REST API: 
    curl -X POST -H "Content-Type: application/json" --data '{ "name": "echo-sink-test", "config": { "connector.class": "EchoSink", "tasks.max": "3", "topics": "my-topic", "level": "INFO" } }' http://localhost:8083/connectors

Using with Strimzi

  1. Deploy Strimzi
  2. Deploy Kafka Connect using the KafkaConnect CR: 
    apiVersion: v1
kind: Secret
metadata:
  name: kafkaconnectbuild-pull-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: XXXXXXXXXX

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
  build:
    output:
      type: docker
      image: my-registry/my-org/kafka-connect-build:latest
      pushSecret: kafkaconnectbuild-pull-secret
    plugins:
      - name: echo-sink-connector
        artifacts:
          - type: jar
            url: https://github.com/scholzj/echo-sink/releases/download/1.4.0/echo-sink-1.4.0.jar
            sha512sum: 6d40b6334b9e6ff81a11176f23d082688ace58ff000570867dfabbd69a301c579c21f571985612c36cddd9d03454cea8cf0ace0d1f9335b5ef2e039f9781074a
  3. Deploy the connector using the KafkaConnector CR: 
    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: echo-sink-connector
  labels:
    strimzi.io/cluster: my-connect
spec:
  class: EchoSink
  tasksMax: 1
  config:
    level: "INFO"
    topics: "my-topic"

echo-sink's People

Contributors

scholzj avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

bygui86

echo-sink's Issues

java.lang.NoClassDefFoundError: io/opentracing/util/GlobalTracer in v.1.2.0

Hi!

I'm using your sink, v1.2.0 in a custom Kafka Connect Docker Image, but when I try to start a connector, I receive the following message:

[2021-07-08 15:32:28,797] ERROR WorkerSinkTask{id=EchoSinkConnector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:184)
java.lang.NoClassDefFoundError: io/opentracing/util/GlobalTracer
	at cz.scholz.kafka.connect.echosink.EchoSinkTask.start(EchoSinkTask.java:44)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:308)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: io.opentracing.util.GlobalTracer
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)

The dockerfile that I'm using is:

FROM bitnami/kafka:2.8.0
USER root:root

RUN mkdir -p /opt/bitnami/kafka/plugins

RUN wget -q https://github.com/scholzj/echo-sink/releases/download/1.2.0/echo-sink-1.2.0.jar -O /opt/bitnami/kafka/plugins/echo-sink-1.2.0.jar

USER 1001
CMD ["/opt/bitnami/kafka/bin/connect-distributed.sh", "/opt/bitnami/kafka/config/connect-distributed.properties"]

and in connect-distributed.properties I have a line with:
plugin.path=/opt/bitnami/kafka/plugins

The configuration of connector is:

connector.class=cz.scholz.kafka.connect.echosink.EchoSinkConnector
key.converter.schemas.enable=false
topics=topic-test
tasks.max=2
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

Other connectors (i.e FileStreamSinkConnector) are working fine.

The error is correct, in echo-sink 1.2.0 jar there is no opentracing jar. Actually there are no extra jars inside provided jar.

How can I skip this error?

Thank you

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.