The memphis-k8s from memphisdev

REST Gateway Pods missing tolerations

The REST Gateway pods do include the Helm chart's pod (anti-)affinity settings, but ignore the tolerations.

This is problematic because I have a k8s node group that is tainted to specifically only allow memphis pods to land on them. So I need nodeSelector, podAntiAffinity and tolerations to work on all pods to schedule my memphis pods on a specific node group, and also spread them over my AZs.

Create k8s operator

test
test
test
test

busy box not working well

the control plane can not establish connection with the DB

Add plain K8S yamls, without HELM

Motivation

As discussed in this discord thread https://discord.com/channels/963333392844328961/1088042559969951875
Required by a non-managed kubernetes distribution.

Cant install two memphis deployments with the same name on the same cluster

helm install my-memphis memphis/memphis --create-namespace --namespace memphis2

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "my-memphis-memphis-busybox-role" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "memphis2": current value is "memphis"

Set pullPolicy for busybox top IfNotPresent

Since the busybox image is set to a fixed tag there is no reason to set the pullPolicy to Always. This can lead to be rate limited by docker in case a deployment fails.

An ability to provide custom secret for memphis creds.

An ability to provide a custom secret name with memphis-creds during helm deployment that will ensure that the values are not exposed in the install/upgrade process.
Asked by: @madestreel

Memphis broker should run on separate nodes

test

Add support for automatic protocol detection for most common service meshes

Add support for automatic protocol detection for the ports provided by the service resource for service meshes like istio or kong. This can normally be done by adding the appProtocol field to the ports in the service or prefixing the service ports name with the protocol.

I created a PR to add the field see #185

External mongodb

Is it possible to connect Memphis with externally hosted MongoDB? I think the chart does not have any setting to change the connection string and does not self-host the MongoDB in the k8s cluster.

Why does the memphis statefulset use host ports?

As the title suggests. Currently there is no way to disable this.

Bug: Helm deployment fails on K3S Cluster

Describe the bug

The helm deployment fails on a k3s cluster. k8s-busybox-waits4broker runs but the memphis-http-proxy is always stuck waiting and that is as far as the deployment gets.

The logs from the busybox container show:

waiting for broker
nslookup: can't resolve 'memphis-cluster.memphis.svc.cluster.local'
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

DNS resolution in the cluster is working. The cluster is using the default k3s core-dns container.

I do get the successful deployment message printed out with the deployment details.

Let me know if there is any other output that i can provide to help debug

Steps to reproduce

Run the helm deploy commands on a k3s based Kubernetes cluster.

Affected services

Broker

Platforms

Kubernetes

If UI - Browsers

No response

Environment

Production, Development, Testing

Additional context

No response

Code of Conduct

I agree to follow this project's Code of Conduct

Contributing Docs

I agree to follow this project's Contribution Docs

Add global value for metadata pvc configuration

Metadata persistence.existingClaim value should be configurable with global values.

Refreshing creds every deployment

memphis-creds refreshed the JWT_SECRET and REFRESH_JWT_SECRET every deployment (Terraform / Pulumi). Would be good to have this also included in the values.yaml.
https://github.com/memphisdev/memphis-k8s/blob/v0.4.4/memphis/templates/memphis-creds.yaml#L10-L11

Check Memphis deployment on OCP

Test Memphis deployment on top of the Red Hat Openshift platform. Need to be tested as a Standalone installation and as a cluster as well.

ServiceMonitor resource endpoint spec is using port instead of targetPort

According to the endpoint spec of a ServiceMonitor resource either of port: or targetPort: need to be used. In the case of port: it should reference the name of a port defined in the Service. In the case of targetPort: it should reference a port of a pod behind the service.

Since the metrics port is not defined in the Service it leads to a prometheus operator not finding what to monitor. In order not to expose the exporter port in the service I suggest to use targetPort: instead.

Create k8s operator

Memphis K8S operator will be based on Golang.
The operator needs to cover all day 1 and day 2 automation phases:

test2

Move user configuration to a secret

Hey,

Thanks for adding the user provisioning through a configuration file! The only issue here is that passwords need to be in plain text in our configuration file (we could also auto generate them in a template file but this would change their value at each apply) our we need to save them to a secret, read from it and write that value to the configuration. This is not very practical...

Furthermore, I think passwords should be in secrets anyway...

Would it be possible to change this so that the user configuration would be fetched from a custom / pre-defined secret instead of being part of the default configuration ConfigMap.

I suppose this would require some changes on the server side to be able to read the configuration from another directory since it is not possible to mount two ConfigMap / Secret at the same place.

and I would then have a volume that would be mounted like this

volumes:
    - name: initial-user-configuration
      secret:
        secretName: {{ .Values.auth.existingSecret }}

And in the container definition

volumeMounts:
  - name: initial-user-configuration
    readOnly: true
    mountPath: "/etc/nats-config/conf.d/user-configuration"

Finally, in the memphis.conf I would include everything that is in /etc/nats-config/conf.d/**/*.conf and on the memphis server read the user configuration directly from the main configuration file.

That way configuration is make more dynamic and gives the possibility to user to mount extra configuration volumes to /etc/nats-config/conf.d like they which.

Of course this is just a proposition, if you prefer this can also be a ConfigMap.

Helm deployment fails on K8S Cluster

Please let me know what went wrong.

command: helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && helm install memphis memphis/memphis --create-namespace --namespace memphis --wait

kubectl get all -n memphis

NAME                                                READY   STATUS             RESTARTS          AGE
pod/memphis-0                                       0/2     Pending            0                 12h
pod/memphis-metadata-0                              0/1     Pending            0                 12h
pod/memphis-metadata-coordinator-6f5b7c9b59-qhwd7   0/1     CrashLoopBackOff   197 (3m34s ago)   12h
pod/memphis-rest-gateway-649774875b-bfmdl           0/1     Init:0/1           0                 12h

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
service/memphis                        ClusterIP   None             <none>        9000/TCP,7770/TCP,6666/TCP,8222/TCP   12h
service/memphis-metadata               ClusterIP   10.107.188.205   <none>        5432/TCP                              12h
service/memphis-metadata-coordinator   ClusterIP   10.108.3.182     <none>        5432/TCP                              12h
service/memphis-metadata-headless      ClusterIP   None             <none>        5432/TCP                              12h
service/memphis-rest-gateway           ClusterIP   10.106.37.203    <none>        4444/TCP                              12h

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/memphis-metadata-coordinator   0/1     1            0           12h
deployment.apps/memphis-rest-gateway           0/1     1            0           12h

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/memphis-metadata-coordinator-6f5b7c9b59   1         1         0       12h
replicaset.apps/memphis-rest-gateway-649774875b           1         1         0       12h

NAME                                READY   AGE
statefulset.apps/memphis            0/1     12h
statefulset.apps/memphis-metadata   0/1     12h

kubectl -n memphis logs deployment.apps/memphis-metadata-coordinator

pgpool 23:00:59.79
pgpool 23:00:59.79 Welcome to the Bitnami pgpool container
pgpool 23:00:59.80 Subscribe to project updates by watching https://github.com/bitnami/containers
pgpool 23:00:59.80 Submit issues and feature requests at https://github.com/bitnami/containers/issues
pgpool 23:00:59.80
pgpool 23:00:59.80 INFO  ==> ** Starting Pgpool-II setup **
pgpool 23:00:59.83 INFO  ==> Validating settings in PGPOOL_* env vars...
pgpool 23:00:59.85 INFO  ==> Initializing Pgpool-II...
pgpool 23:00:59.86 INFO  ==> Generating pg_hba.conf file...
pgpool 23:00:59.86 INFO  ==> Generating pgpool.conf file...
pgpool 23:01:00.03 INFO  ==> Generating password file for local authentication...
pgpool 23:01:00.04 INFO  ==> Generating password file for pgpool admin user...

pgpool 23:01:00.05 INFO  ==> ** Pgpool-II setup finished! **
pgpool 23:01:00.08 INFO  ==> ** Starting Pgpool-II **
2024-01-23 23:01:00.094: main pid 1: LOG:  Backend status file /opt/bitnami/pgpool/logs/pgpool_status does not exist
2024-01-23 23:01:00.094: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.094: main pid 1: LOG:  memory cache initialized
2024-01-23 23:01:00.094: main pid 1: DETAIL:  memcache blocks :64
2024-01-23 23:01:00.094: main pid 1: LOG:  allocating (144190784) bytes of shared memory segment
2024-01-23 23:01:00.094: main pid 1: LOG:  allocating shared memory segment of size: 144190784
2024-01-23 23:01:00.225: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.225: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.226: main pid 1: LOG:  memory cache initialized
2024-01-23 23:01:00.226: main pid 1: DETAIL:  memcache blocks :64
2024-01-23 23:01:00.228: main pid 1: LOG:  pool_discard_oid_maps: discarded memqcache oid maps
2024-01-23 23:01:00.234: main pid 1: LOG:  unix_socket_directories[0]: /opt/bitnami/pgpool/tmp/.s.PGSQL.5432
2024-01-23 23:01:00.234: main pid 1: LOG:  listen address[0]: *
2024-01-23 23:01:00.234: main pid 1: LOG:  Setting up socket for 0.0.0.0:5432
2024-01-23 23:01:00.235: main pid 1: LOG:  Setting up socket for :::5432
2024-01-23 23:01:00.241: main pid 1: LOG:  find_primary_node_repeatedly: waiting for finding a primary node
2024-01-23 23:01:00.244: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.244: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:00.246: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.246: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:00.248: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.248: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:01.251: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.251: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:01.253: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.253: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:01.258: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.258: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:02.261: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.261: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:02.263: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.263: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:02.264: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.264: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:03.267: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:03.268: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:06.821: child pid 151: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:06.821: child pid 151: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:06.821: child pid 151: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:08.271: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:08.271: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:08.273: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:08.273: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:08.273: main pid 1: LOG:  reaper handler
2024-01-23 23:01:08.274: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:09.276: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.276: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:09.278: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.278: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:09.279: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.280: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:10.282: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.282: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:10.283: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.283: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:10.285: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.285: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:11.288: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.288: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:11.290: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.290: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:11.291: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.291: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:11.814: child pid 141: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.814: child pid 141: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:11.814: child pid 141: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:11.815: main pid 1: LOG:  reaper handler
2024-01-23 23:01:11.816: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:12.294: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.294: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:12.296: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.296: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:12.297: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.297: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:13.300: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.300: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:13.301: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.301: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:13.307: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.307: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:14.310: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.310: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:14.311: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.311: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:14.313: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.313: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:15.316: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.316: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:15.317: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.317: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:15.319: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.319: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:16.322: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.322: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:16.324: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.324: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:16.325: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.325: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:17.328: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.328: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:17.329: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.329: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:17.331: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.331: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:18.334: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.334: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:18.335: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.335: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:18.337: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.337: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:19.339: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.339: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:19.341: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.341: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:19.342: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.342: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:20.345: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.345: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:20.346: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.346: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:20.348: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.348: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:21.351: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.351: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:21.353: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.353: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:21.355: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.355: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:21.820: child pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.820: child pid 148: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:21.820: child pid 148: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:21.821: main pid 1: LOG:  reaper handler
2024-01-23 23:01:21.828: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:21.878: child pid 164: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.878: child pid 164: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:21.878: child pid 164: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:21.886: main pid 1: LOG:  reaper handler
2024-01-23 23:01:21.887: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:22.357: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.357: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:22.358: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.358: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:22.360: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.360: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:23.362: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.362: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:23.363: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.363: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:23.365: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.365: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:24.367: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.367: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:24.369: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.369: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:24.371: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.371: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:25.373: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.374: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:25.375: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.375: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:25.377: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.377: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:26.380: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.380: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:26.381: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.381: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:26.383: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.383: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:26.824: child pid 197: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.824: child pid 197: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:26.824: child pid 197: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:26.825: main pid 1: LOG:  reaper handler
2024-01-23 23:01:26.825: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:27.385: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.385: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:27.386: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.386: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:27.387: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.387: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:28.391: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.391: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:28.394: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.394: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:28.395: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.395: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:29.398: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.398: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:29.400: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.400: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:29.403: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.403: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:30.408: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.408: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:30.410: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.410: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:30.415: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.415: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:31.419: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.419: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:31.421: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.421: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:31.423: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.423: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:31.821: child pid 153: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.821: child pid 153: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:31.821: child pid 153: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:31.822: main pid 1: LOG:  reaper handler
2024-01-23 23:01:31.823: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:32.426: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.426: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:32.427: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.428: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:32.429: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.429: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:33.433: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.433: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:33.434: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.434: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:33.436: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.436: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:34.440: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.440: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:34.442: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.442: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:34.444: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.444: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:35.447: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.447: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:35.448: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.448: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:35.450: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.450: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:36.452: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.452: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:36.454: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.454: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:36.455: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.455: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:36.844: child pid 160: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.845: child pid 160: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:36.845: child pid 160: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:36.846: child pid 158: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.846: child pid 158: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:36.846: child pid 158: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:36.846: main pid 1: LOG:  reaper handler
2024-01-23 23:01:36.847: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:36.849: main pid 1: LOG:  reaper handler
2024-01-23 23:01:36.849: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:37.457: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.458: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:37.459: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.459: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:37.460: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.460: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:38.463: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.463: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:38.464: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.464: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:38.466: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.466: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:39.469: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.469: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:39.470: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.470: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:39.472: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.472: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:40.475: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.475: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:40.476: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.476: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:40.478: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.478: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:41.481: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.481: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:41.483: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.483: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:41.485: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.485: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:41.821: child pid 140: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.821: child pid 140: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:41.821: child pid 140: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:41.822: main pid 1: LOG:  reaper handler
2024-01-23 23:01:41.823: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:42.487: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.487: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:42.488: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.488: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:42.490: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.490: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:43.493: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.493: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:43.495: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.495: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:43.498: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.498: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:44.501: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.501: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:44.503: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.503: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:44.504: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.504: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:45.507: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.507: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:45.509: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.509: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:45.511: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.511: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:46.514: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.514: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:46.515: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.515: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:46.516: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.517: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:46.820: child pid 154: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.820: child pid 154: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:46.820: child pid 154: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:46.824: main pid 1: LOG:  reaper handler
2024-01-23 23:01:46.824: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:46.865: child pid 240: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.865: child pid 240: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:46.865: child pid 240: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:46.866: main pid 1: LOG:  reaper handler
2024-01-23 23:01:46.867: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:47.519: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.519: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:47.521: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.521: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:47.522: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.522: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:48.526: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.526: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:48.527: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.527: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:48.529: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.529: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:49.532: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.532: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:49.533: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.533: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:49.535: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.535: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:50.538: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.538: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:50.539: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.539: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:50.540: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.540: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:51.543: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.543: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:51.545: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.545: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:51.546: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.546: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:51.815: child pid 273: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.815: child pid 273: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:51.815: child pid 273: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:51.816: main pid 1: LOG:  reaper handler
2024-01-23 23:01:51.817: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:52.549: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.549: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:52.551: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.551: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:52.553: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.554: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:53.556: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.556: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:53.557: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.557: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:53.559: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.559: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2

kubectl -n memphis logs deployment.apps/memphis-rest-gateway

Defaulted container "memphis-rest-gateway" out of: memphis-rest-gateway, k8s-busybox-waits4broker (init)
Error from server (BadRequest): container "memphis-rest-gateway" in pod "memphis-rest-gateway-649774875b-bfmdl" is waiting to start: PodInitializing

kubectl -n memphis logs statefulset.apps/memphis

Defaulted container "memphis" out of: memphis, reloader, memphis-metadata-readiness (init)

kubectl -n memphis logs statefulset.apps/memphis-metadata

NONE

Unable to specify a secret as external PostgreSQL credential source

The chart in its current form is unable to use a secret as source for external PostgreSQL configuration.

memphis-k8s/memphis/templates/memphis_statefulset.yaml

Lines 405 to 418 in 9a91a11

    
           {{- if .Values.metadata.external.enabled }} 
        
           - name: METADATA_DB_USER 
        
             value: {{ .Values.metadata.external.dbUser | quote }} 
        
           - name: METADATA_DB_PASS 
        
             value: {{ .Values.metadata.external.dbPass | quote }} 
        
           - name: METADATA_DB_DBNAME 
        
             value: {{ .Values.metadata.external.dbName | quote }} 
        
           - name: METADATA_DB_HOST 
        
             value: {{ .Values.metadata.external.dbHost | quote }} 
        
           - name: METADATA_DB_PORT 
        
             value: {{ .Values.metadata.external.dbPort | quote }} 
        
           - name: METADATA_DB_TLS_MUTUAL 
        
             value: {{ .Values.metadata.external.dbTlsMutual | quote }} 
        
           {{- else }}

Instead of configuring credentials like so:

    metadata:
      external:
        enabled: true
        dbUser: "foo"
        dbPass: "bar"

I'd like to set:

    metadata:
      external:
        enabled: true
        dbUser:
          value: "foo"
        dbPass:
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password

The change can be made backwards compatible by checking whether the field is set to a string instead of a dictionary.

I'd also suggest you deprecate hard-coding the credentials like that, immediately after implementing the new syntax. It's simply a very bad practice.

P.S. Is a PostgreSQL connection URL (postgesql://...) theoretically supported by Memphis?

Can't install using flux due to duplicate key

Hi,
I'm trying to install this using flux but ran into the following error:

Helm install failed: error while running post render on files: map[string]interface {}(nil): yaml: unmarshal errors: line 18: mapping key "JWT_SECRET_REST_GW" already defined at line 16

This seems to have occurred with other charts as well and I think the cause is this duplication of the env vars. Is it possible to remove it?

Thanks

Edit: After looking at this open PR, I think the refactoring there will already solve this issue 🥳

	{{- if .Values.metadata.external.enabled }}
	- name: METADATA_DB_USER
	value: {{ .Values.metadata.external.dbUser \| quote }}
	- name: METADATA_DB_PASS
	value: {{ .Values.metadata.external.dbPass \| quote }}
	- name: METADATA_DB_DBNAME
	value: {{ .Values.metadata.external.dbName \| quote }}
	- name: METADATA_DB_HOST
	value: {{ .Values.metadata.external.dbHost \| quote }}
	- name: METADATA_DB_PORT
	value: {{ .Values.metadata.external.dbPort \| quote }}
	- name: METADATA_DB_TLS_MUTUAL
	value: {{ .Values.metadata.external.dbTlsMutual \| quote }}
	{{- else }}

memphisdev / memphis-k8s Goto Github PK

memphis-k8s's People

Contributors

Stargazers

Watchers

Forkers

memphis-k8s's Issues