ipni / index-observer Goto Github PK

To reduce noise of pod eviction in dev cluster specify some resource limit and request for the index observer. Ideally, it should gracefully be restarted using a con job instead of letting it run until it runs out of memory and evicted.

currently has a memory leak of some form

probably should recycle the libp2p host every few hours to make sure there aren't any other long-lived resource uses

The local storage of index observer exceeds the maximum allowed ephemeral storage on K8S. To have a more predictable runtime behaviour, define a dedicated PVC and mount that as the data directory for the instance.

Currently index-observer in dev monitors both dev and prod environments. We need to separate them up.

Much like storetheindex, the index-observer should have a dedicated grafana dashboard that tracks:

• Operational metrics, such as health checks, uptime, machine stats.
• Business metrics, such as the overall index deal coverage, number of SP index providers, etc.

This will help monitor and maintain the service, as well as ensure stability of the metrics reported for SitReps.

it looks like nodes don't like answering statemarketdeals because it's expensive. we should calculate instead off the snapshot at https://marketdeals.s3.amazonaws.com/StateMarketDeals.json

currently the hourly tasks samples entries from providers. It should then feed into a downstream pipeline that checks those hashes against the monitored network indexers and confirms availability there

Implement a mechanism that allows the maximum disk usage by index-observer to be configurable. This means there is a need for some data removal strategy.

This is so that the hardware requirements can be configured and predictable at runtime.