zensum / leia Goto Github PK
View Code? Open in Web Editor NEWLeia writes HTTP requests to kafka topics, redis pub/sub or gcloud pub/sub.
License: MIT License
Leia writes HTTP requests to kafka topics, redis pub/sub or gcloud pub/sub.
License: MIT License
Dependabot couldn't find a build.gradle for this project.
Dependabot requires a build.gradle to evaluate your project's current Java dependencies. It had expected to find one at the path: /build.gradle
.
If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.
You can mention @dependabot in the comments below to contact the Dependabot team.
Currently Leia
supports only supports a trivial authentication scheme. If the JWK_URL
env-var is set routes with verify = true
require there to be a JWT token present in the http authentication header. While JWT
is a good interoperable and federated authentication scheme the current method is lacking in a number of ways. The system does not verify audience or subject claims as specified in the JWT standard nor does it offer any mechanisms for verifying other parts of the claim.
In addition to the limitations in the implementation of JWT, there is also a need for other authentication schemes such as HTTP-basic auth. Finally, if configurable authentication and authorisation mechanisms were introduced it would enable us to expose certain internals to potential admin and monitoring tools if proper authentication was provided.
For these reasons I propose that we introduce a new concept to Leia
which for purposes of discussion we will call an Auth-provider
. An auth-provider is a named filter which can be applied to a route so that only authenticated requests match the route. In addition to the filter the Auth-provider may expose additional fields to the data-sinks and formats such as username
or isAuthenticated
. A single route is configurable to use multiple Auth-providers and only a single auth-provider needs to match for the route to match.
Auth-providers themselves, will have type e.g. JWK
or BasicAuth
or HashJWT
. Depending on the type different configuration variables will be available e.g URL for JWK or users
for BasicAuth. Another idea for an auth-provider may be webhook which authenticates the request against some other provider before letting it through.
We should log all incoming calls in order to more easily follow what happens in Leia. Should also map which topic the incoming request is written to.
08:44:59.625 [nettyCallPool-4-1] ERROR ktor.application - Unhandled: POST - /xxxxxxx
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at franz.producer.kafka_one.FutureCallback.onCompletion(Async.kt:19)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:201)
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:185)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:599)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:575)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:539)
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:474)
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:75)
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:660)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:454)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:446)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
This happened in production, resulting in an event not being recordded to Kafka.
Dependabot couldn't find a build.gradle for this project.
Dependabot requires a build.gradle to evaluate your project's current Java dependencies. It had expected to find one at the path: /build.gradle
.
If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.
You can mention @dependabot in the comments below to contact the Dependabot team.
Some different possible causes (can be responsible alone for this behavior or in combination)
Content-Length
(or rather the absence of it) ❗️transfer-encoding: chunked
- should we really use this? ❗️Add a config provider that reads a folder of Kubernetes style objects, reloading them on some interval
---
apiVersion: leia/v1
kind: LeiaRoute
name: the_name_of_my_route_for_dedeupe
somevar: somevalue
This might be a prerequisite for #62
We need to implement either a CustomResourceDefinition or a ConfigMap for each route
JSON schema is a commonly used format validating JSON documents. Checking incoming request against a specified JSON schema would allow consumer of Leia produced data to skip certain validation steps, and would allow producers to get proper error messages when their data-format is invalid.
This test should run everything with docker-compose and Kafka and try a number of requests ensuring that they all return 200
Leia should be able to read from directory of configuration files, but when second file is added it does not affect the app.
Looks like the issue is with computeCurrentState() in TomlRegistry which does not join configurations, but returns last one.
This is directly related to #13, but since there are some questions directly concerning the basic auth implementation, I created an issue of its own.
1. Should we use a simpler hash algorithm such as SHA-512
or a more complex one like PBKDF2WithHmacSHA1
? Or should it be configurable?
2. Should we (allow/enforce) the use of salt for hashed basic auth credentials? Or can we expect the inputs to be so unique that salting will not be necessary?
3. If we will use PBKDF2, should the number of hash iterations be possible to configure?
WWW-Authenticate: Basic realm="User Visible Realm", charset="UTF-8"
as a response to any 401 request.According to HTTP RFC:
Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.
Some service providers will expect a specific response code and regard every other response as a failure (even when it is a 2** code). For example, Twilio expects 204/No-Content and Mandrill expects 200/OK. This will make it hard to make both of these services happy. This can be solved by providing an optional response code on success for each entry/route, leaving no entry defaulting to 204.
When making an HTTP endpoint it is useful to be able to reject responses that do not contain valid JSON. To implement this issue add a configuration field allowing a route to restrict input to valid JSON. If the JSON is invalid 400 should be returned.
Be careful with what happens if the JSON payload is too big, so that it doesn't crash everything
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.