yahoo / bandar-log Goto Github PK

See https://docs.aws.amazon.com/en_us/streams/latest/dev/monitoring-with-cloudwatch.html

Adding should be pretty straightforward as it is done via JDBC

https://github.com/OneByAol/bandar-log/blob/020db46494e3970b1333ee907514a48f72da6fb4/bandarlog/src/main/resources/application.conf#L16

According to RFC 5737:

The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), and 203.0.113.0/24 (TEST-NET-3) are provided for use in documentation.

Right now Bandar-log supports only Datadog as target metrics consumer, but it will definitely be good to support cloud built-in solutions like AWS Cloudwatch (AWS), Stackdriver monitoring (GCP) and Azure Monitor (Azure).

We need to monitor metrics on Athena, so Bandar-log needs Athena/Glue SQL data source support.
Keep in mind the method of obtaining max(batch_id) specific to AWS Glue to not do full scan of target table.

We are trying to add bandarlog for MSK and facing issues. On first run we got

ERROR connectors.KafkaConnector: Cannot obtain leaders offsets for topic:[Set(input_test)], cause java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available.

After restarting bandarlog we started to get

ERROR connectors.KafkaConnector: Cannot obtain leaders offsets for topic:[Set(input_test)], cause java.util.concurrent.TimeoutException: Futures timed out after [1 second]

MSK security group has an open port for bandarlog machine. Any tips in debugging would be helpful.

Thanks

@yurir @zsaltys Hi, could you please clarify, if bandar-log could be made fully OSS project and be put under control of its current contributors?

At the moment, Bandarlog relies on semantics of Kafka, however it could be re-done with more generalized idea of observation of queue-like services.
Should be done to enable support of different providers of queue/log nature, eg Kinesis, SQS, etc.
Also, could possibly enable cross-queue monitoring.
Needs to be done before #18 and #19.

It's hard to import project in IDE because of sbt project depends on TRAVIS_BUILD_NUMBER's value.
In case it's not set, import procedure fails.

Add bypass for this variable for development environment, like setting it to constant if environment variables is not set.

Similar to #31, it would be good to have Clickhouse support out of the box

SQS could be monitored via CloudWatch API.
See
https://docs.aws.amazon.com/en_us/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-access-metrics.html
https://docs.aws.amazon.com/en_us/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html

Per brettwooldridge/HikariCP#314, can be fixed pretty quickly by adding networkaddress.cache.ttl property with a relatively small default value, like 10 [seconds].

Pretty much self-descriptive. Some projects do not have Datadog license, but Bandarlog is a great tool which solves their problems.

LAG and REALTIME_LAG metrics could share value from remote data source within short period of time, especially in case of Glue data source.

We could reuse PagerDuty's metrics library and avoid reporter re-implementation.
This would provide more flexibility with metrics creation and reporting.

We use Kafka client and auxiliary classes from Spark, while it could be replaced with pure Kafka client. This would reduce dependency list a lot.
We need to use client that is able to connect to brokers no older that 0.10.0.2 version.