[Bug]: Replica cluster Status is healthy while connection to the source cluster is broken. Metric is also missing to report that about cloudnative-pg HOT 3 OPEN

This is missing. Consider that at the moment our "replica cluster" story is not yet complete. In this specific case, we should monitor if the WAL receiver is up. Our recommendation is to always use the WAL archive as a fallback mechanism.

"Cluster is healthy" is referred to the local replica cluster. It means that all replicas are up and so the designated primary.

from cloudnative-pg.

I checked and we already monitor this: https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW

Look at the pg_replication metric in the Prometheus exporter.

See: https://github.com/cloudnative-pg/cloudnative-pg/blob/main/config/manager/default-monitoring.yaml#L109

from cloudnative-pg.

Hi @gbartolini

Thanks for your feedback

I looked at the content of the "pg_stat_wal_receiver" view and I have the impression that the data in pg_stat_wal_receiver only concerns the primary pod and the replica pods within the same cluster.

I've checked the primary and replica clusters, both of which only contain data for their replica pods. The hosts correspond only to their replica pods.

So I'm afraid the pg_replication metric is only valid within the same cluster.
cnpg_pg_replication_streaming_replicas value (2) matchs only the clusters replica pods
And only replica pods name and addr are present in metrics labels .... no information to Replica cluster and its pods

I'll also have a look at wal archiving

from cloudnative-pg.