zensum / leia Goto Github PK

Dependabot couldn't find a build.gradle for this project.

Dependabot requires a build.gradle to evaluate your project's current Java dependencies. It had expected to find one at the path: /build.gradle.

If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

Currently Leia supports only supports a trivial authentication scheme. If the JWK_URL env-var is set routes with verify = true require there to be a JWT token present in the http authentication header. While JWT is a good interoperable and federated authentication scheme the current method is lacking in a number of ways. The system does not verify audience or subject claims as specified in the JWT standard nor does it offer any mechanisms for verifying other parts of the claim.

In addition to the limitations in the implementation of JWT, there is also a need for other authentication schemes such as HTTP-basic auth. Finally, if configurable authentication and authorisation mechanisms were introduced it would enable us to expose certain internals to potential admin and monitoring tools if proper authentication was provided.

For these reasons I propose that we introduce a new concept to Leia which for purposes of discussion we will call an Auth-provider. An auth-provider is a named filter which can be applied to a route so that only authenticated requests match the route. In addition to the filter the Auth-provider may expose additional fields to the data-sinks and formats such as username or isAuthenticated. A single route is configurable to use multiple Auth-providers and only a single auth-provider needs to match for the route to match.

Auth-providers themselves, will have type e.g. JWK or BasicAuth or HashJWT. Depending on the type different configuration variables will be available e.g URL for JWK or users for BasicAuth. Another idea for an auth-provider may be webhook which authenticates the request against some other provider before letting it through.

We should log all incoming calls in order to more easily follow what happens in Leia. Should also map which topic the incoming request is written to.

08:44:59.625 [nettyCallPool-4-1] ERROR ktor.application - Unhandled: POST - /xxxxxxx
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at franz.producer.kafka_one.FutureCallback.onCompletion(Async.kt:19)
	at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:201)
	at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:185)
	at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:599)
	at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:575)
	at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:539)
	at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:474)
	at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:75)
	at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:660)
	at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
	at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:454)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:446)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.

This happened in production, resulting in an event not being recordded to Kafka.

Dependabot couldn't find a build.gradle for this project.

Dependabot requires a build.gradle to evaluate your project's current Java dependencies. It had expected to find one at the path: /build.gradle.

If this isn't a Java project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

Some different possible causes (can be responsible alone for this behavior or in combination)

• Status codes, especially No-Content (204) vs OK (200)
• Header Content-Length (or rather the absence of it) ❗️
• Header transfer-encoding: chunked - should we really use this? ❗️
• Google load balancer (error cannot be reproduced without it, but this may still not be the actual problem) ❗️

Add a config provider that reads a folder of Kubernetes style objects, reloading them on some interval

---
      apiVersion: leia/v1
      kind: LeiaRoute 
      name: the_name_of_my_route_for_dedeupe
      somevar: somevalue 

This might be a prerequisite for #62

We need to implement either a CustomResourceDefinition or a ConfigMap for each route

JSON schema is a commonly used format validating JSON documents. Checking incoming request against a specified JSON schema would allow consumer of Leia produced data to skip certain validation steps, and would allow producers to get proper error messages when their data-format is invalid.

This test should run everything with docker-compose and Kafka and try a number of requests ensuring that they all return 200

Leia should be able to read from directory of configuration files, but when second file is added it does not affect the app.

Looks like the issue is with computeCurrentState() in TomlRegistry which does not join configurations, but returns last one.

This is directly related to #13, but since there are some questions directly concerning the basic auth implementation, I created an issue of its own.

1. Should we use a simpler hash algorithm such as SHA-512 or a more complex one like PBKDF2WithHmacSHA1? Or should it be configurable?
2. Should we (allow/enforce) the use of salt for hashed basic auth credentials? Or can we expect the inputs to be so unique that salting will not be necessary?
3. If we will use PBKDF2, should the number of hash iterations be possible to configure?

• Hashing
  • Uses SHA-256
  • No Salt
  • Only hash password and not username
• Return WWW-Authenticate: Basic realm="User Visible Realm", charset="UTF-8" as a response to any 401 request.
• Allow configuration of credentials - Example config provided here

According to HTTP RFC:

Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.

Some service providers will expect a specific response code and regard every other response as a failure (even when it is a 2** code). For example, Twilio expects 204/No-Content and Mandrill expects 200/OK. This will make it hard to make both of these services happy. This can be solved by providing an optional response code on success for each entry/route, leaving no entry defaulting to 204.

When making an HTTP endpoint it is useful to be able to reject responses that do not contain valid JSON. To implement this issue add a configuration field allowing a route to restrict input to valid JSON. If the JSON is invalid 400 should be returned.

Be careful with what happens if the JSON payload is too big, so that it doesn't crash everything