simplesteph / kafka-connect-github-source Goto Github PK

View Code? Open in Web Editor NEW

439.0 7.0 186.0 51 KB

Get a stream of issues and pull requests for your chosen GitHub repository

Home Page: https://links.datacumulus.com/kafka-connect-coupon

License: MIT License

Shell 1.63% Java 97.90% Dockerfile 0.48%

kafka kafka-connect github github-api source

kafka-connect-github-source's Introduction

Learning

This project is a companion repository to the Apache Kafka Connect course on Udemy.

https://links.datacumulus.com/kafka-connect-coupon

Kafka Connect Source GitHub

This connector allows you to get a stream of issues and pull requests from your GitHub repository, using the GitHub Api: https://developer.github.com/v3/issues/#list-issues-for-a-repository

Issues are pulled based on updated_at field, meaning any update to an issue or pull request will appear in the stream.

The connector writes to topic that is great candidate to demonstrate log compaction. It's also a fun way to automate your GitHub workflow.

It's finally aimed to be an educative example to demonstrate how to write a Source Connector a little less trivial than the FileStreamSourceConnector provided in Kafka.

Contributing

This connector is not perfect and can be improved, please feel free to submit any PR you deem useful.

Configuration

name=GitHubSourceConnectorDemo
tasks.max=1
connector.class=com.simplesteph.kafka.GitHubSourceConnector
topic=github-issues
github.owner=kubernetes
github.repo=kubernetes
since.timestamp=2017-01-01T00:00:00Z
# I heavily recommend you set those two fields:
auth.username=your_username
auth.password=your_password

Running in development

Note: Java 8 is required for this connector. Make sure config/worker.properties is configured to wherever your kafka cluster is

./build.sh
./run.sh

The simplest way to run run.sh is to have docker installed. It will pull a Dockerfile and run the connector in standalone mode above it.

Deploying

Note: Java 8 is required for this connector.

TODO

kafka-connect-github-source's People

Contributors

Stargazers

Watchers

Forkers

rpan93 tejashtarun14 cristinanegrean kvasagiri iterrator venkareddy37 dongjunqiang rbalusup yinrich fumengyao5544 nazender arunprasadbh maggie-lagos pobrienjhu abdul99 themightylaz nvichare ktpm489 giampaolotrapasso sffej skashan-ali yefan000 andrew-gurney asantos2000 louth mehul9891 dwtcourses adityaautomator badri03iter muarshad01 t1seungy janaidu01 vinodkandula azurecloudmonk sharebuilder tristindavis dbolshak oopsaleem manojponugupati jagadeeshtechmind tayfourius philippewanner prashantn85 andreizhoraven sheikh-qzaman chuqiaoshen zhuohuwu0603 adolfoeliazat impidimenta martindupond rayanegouda saroj-mandal kjang1750 vkrivcov chrisqiqiu tianhao-au prasanthcircus sergion vinod-unknown ogfunkycold jcims123 zacharybell-ref muralik1989 guizmaii walter-li onurtokat antoinegoze abelash arnabkar aftabvirtual mohitrajranu fiston benluteijn vpez maciej-rzeszutko diegolucasilva ldubych neeraj97 silvrwolfboy iamvinayakj mcallist1 sumitya gomesnayagam akanshgoswami mangojuicy kieronedwards el27egs abhinavece csaltos elaineon109 khansaleemahmed rs111 chen17981 biocar2001 nandakishoremutyala theprogrammersbook alfeuduran craftcube prabhakar747 ramakrishna580

kafka-connect-github-source's Issues

GitHubAPIHttpClient http response headers have wrong case

Problem

Currently GitHubSourceTaskTest fails when attempting to assert headers in the response from GitHub. This prevents the connector from being built. This seems due to change of the github api from v3 to v4 which now uses graphql and has a different strategy for rate limiting

Solution

Need to change "RateLimit" to "Ratelimit" in the headers. The test needs to be updated to take account of these ideally using constants. Further more the "Etag" property in the test is also now lower case and perhaps can be removed as is not used in the code (?)

Longer Term Solution

Upgrade to use the graphql version of the github api (ratelimits are specified in an object).

Get rid of nasty error on first startup when in standalone mode

Error below:

[2017-04-27 22:55:18,729] ERROR CRITICAL: Failed to deserialize offset data when getting offsets for task with namespace GitHubSourceConnectorDemo-2. No value for this data will be returned, which may break the task or cause it to skip some data. This could either be due to an error in the connector implementation or incompatible schema. (org.apache.kafka.connect.storage.OffsetStorageReaderImpl:102)
org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires "schema" and "payload" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.
        at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:309)
        at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:96)
        at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:54)
        at com.simplesteph.kafka.GitHubSourceTask.initializeLastVariables(GitHubSourceTask.java:57)
        at com.simplesteph.kafka.GitHubSourceTask.start(GitHubSourceTask.java:52)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:141)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Connector still works, and the error disappears after some offsets have been first committed.
PR welcome

Externalize schemas as .avsc files

Dockerfile for standalone run

Better Readme

test case failing in GitHubSourceTaskTest.java

how i create a custom group.id for connector workers

hi,
i am running kafka connector and tried to sink topic data to file, as my topic is correct but kafka connect is prefixing connect- to name of group.id .

eg name: simplesteph
in consume group.id = connect-simplesteph.

can i change the prefix.

thank you.

Add License

Dependency org.apache.httpcomponents:httpclient, leading to CVE problem

Hi, In kafka-connect-github-source，there is a dependency org.apache.httpcomponents:httpclient:4.5.2 that calls the risk method.

CVE-2020-13956

The scope of this CVE affected version is [,4.5.13)

After further analysis, in this project, the main Api called is <org.apache.http.client.utils.URIUtils: org.apache.http.HttpHost extractHost(java.net.URI)>

Risk method repair link : GitHub

CVE Bug Invocation Path--

Path Length : 6

<org.apache.http.client.utils.URIUtils: org.apache.http.HttpHost extractHost(java.net.URI)>
at <org.apache.http.impl.client.DecompressingHttpClient: org.apache.http.HttpHost getHttpHost(org.apache.http.client.methods.HttpUriRequest)> (org.apache.http.impl.client.DecompressingHttpClient.java:[137]) in /.m2/repository/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
at <org.apache.http.impl.client.DecompressingHttpClient: org.apache.http.HttpResponse execute(org.apache.http.client.methods.HttpUriRequest)> (org.apache.http.impl.client.DecompressingHttpClient.java:[123]) in /.m2/repository/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
at <com.mashape.unirest.http.HttpClientHelper: com.mashape.unirest.http.HttpResponse request(com.mashape.unirest.request.HttpRequest,java.lang.Class)> (com.mashape.unirest.http.HttpClientHelper.java:[138]) in /.m2/repository/com/mashape/unirest/unirest-java/1.4.9/unirest-java-1.4.9.jar
at <com.mashape.unirest.request.BaseRequest: com.mashape.unirest.http.HttpResponse asJson()> (com.mashape.unirest.request.BaseRequest.java:[68]) in /.m2/repository/com/mashape/unirest/unirest-java/1.4.9/unirest-java-1.4.9.jar
at <com.simplesteph.kafka.GitHubAPIHttpClient: com.mashape.unirest.http.HttpResponse getNextIssuesAPI(java.lang.Integer,java.time.Instant)> (com.simplesteph.kafka.GitHubAPIHttpClient.java:[84]) in /detect/unzip/kafka-connect-github-source-1.1/target/classes

Dependency tree--

[INFO] com.simplesteph.kafka:kafka-connect-github-source:jar:1.1
[INFO] +- org.apache.kafka:connect-api:jar:0.10.2.0-cp1:provided
[INFO] |  +- org.apache.kafka:kafka-clients:jar:0.10.2.0-cp1:provided
[INFO] |  |  +- net.jpountz.lz4:lz4:jar:1.3.0:provided
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.2.6:provided
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile
[INFO] |  \- log4j:log4j:jar:1.2.17:compile
[INFO] \- com.mashape.unirest:unirest-java:jar:1.4.9:compile
[INFO]    +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO]    |  +- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO]    |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO]    |  \- commons-codec:commons-codec:jar:1.9:compile
[INFO]    +- org.apache.httpcomponents:httpasyncclient:jar:4.1.1:compile
[INFO]    |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.4:compile
[INFO]    +- org.apache.httpcomponents:httpmime:jar:4.5.2:compile
[INFO]    \- org.json:json:jar:20160212:compile

Suggested solutions:

Update dependency version to 4.5.13 or higher

Thank you very much.

Multiple repository support

Make Schema complete and in line with GitHub API

See https://developer.github.com/v3/issues/#list-issues

does anyone need repository?