Treating data/variants.tsv.zip
as source data:
- Write a
Kafka
producer which publishes records onto aKafka
topic - Come with a strategy to populate published records into an
Elasticsearch
index - Come with a strategy to populate published records into a
postgres
db table [Optional]
We're trying to mimic a 1 source -> 2 sink scenario, but you can choose to treat #3 as optional. Please craft solutions prioritizing maintainability and best practices.
- A
docker-compose
setup is provided as a starting point, which sets upKafka
,Zookeeper
,Elasticsearch
,Kibana
&Postgres
- Feel free to use a more compact setup which reduces # of individual pieces (eg:
lenses.io
)
Python
or a JVM
language (Kotlin
, Scala
, Java
) is preferred, but feel free to employ any programming language