Namespace-Scoped pgAdmin 4 is not accessible via Service about postgres-operator HOT 14 CLOSED

I got it working using the suggestion from @benjaminjb

PG Admin code

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGAdmin
  metadata:
  name: rhino
namespace: postgres
spec:
  dataVolumeClaimSpec: # 1
    accessModes:
      - "ReadWriteOnce"
    resources:
      requests:
        storage: 1Gi
  serverGroups: # 2
    - name: local
      postgresClusterSelector: {}
  config:
    settings:
      DEFAULT_SERVER: "0.0.0.0"

After that, I created a Service on top and in my case a Traefik IngressRoute point to the service. Without those three lines, I got a Bad Gateway error from Traefik.

from postgres-operator.

upd.

to ensure that Istio isn't causing the problem, I have deployed another pod without a sidecar.
results are similar:

using service

/ # curl http://pg-admin:5050 --verbose
* Host pg-admin:5050 was resolved.
* IPv6: (none)
* IPv4: 10.0.1.169
*   Trying 10.0.1.169:5050...
* connect to 10.0.1.169 port 5050 from 10.244.2.17 port 53174 failed: Connection refused
* Failed to connect to pg-admin port 5050 after 4 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to pg-admin port 5050 after 4 ms: Couldn't connect to server

using pod IP

/ # curl 10.244.5.49:5050 --verbose
*   Trying 10.244.5.49:5050...
* connect to 10.244.5.49 port 5050 from 10.244.2.17 port 57800 failed: Connection refused
* Failed to connect to 10.244.5.49 port 5050 after 1 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to 10.244.5.49 port 5050 after 1 ms: Couldn't connect to server

IP addresses here are different because I had to recreate the service & pgadmin statefulset.

from postgres-operator.

Hi @FNSdev, sorry to hear you're running into this. Can you share the spec for the rhino-pgadmin svc that is created by kubectl expose?

from postgres-operator.

hello @benjaminjb

sure, here it is.

please note that I created it for my pgadmin resource, not the one from example, but I believe it shouldn't make a difference.

pgadmin

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGAdmin
metadata:
  name: fdp
  namespace: default
spec:
  metadata:
    labels:
      sidecar.istio.io/inject: "false"
  dataVolumeClaimSpec:
    accessModes:
      - "ReadWriteOnce"
    resources:
      requests:
        storage: 1Gi
  serverGroups:
    - name: default
      postgresClusterSelector: {} # automatically detect all clusters in the namespace

service

-> % kubectl expose pod pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0 --name test-pgadmin
service/test-pgadmin exposed

-> % kubectl get service test-pgadmin -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-12-20T15:41:57Z"
  labels:
    controller-revision-hash: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-6f9dffbcbc
    postgres-operator.crunchydata.com/data: pgadmin
    postgres-operator.crunchydata.com/pgadmin: fdp
    postgres-operator.crunchydata.com/role: pgadmin
    sidecar.istio.io/inject: "false"
    statefulset.kubernetes.io/pod-name: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0
  name: test-pgadmin
  namespace: default
  resourceVersion: "106674245"
  uid: ea0abf32-c636-42d0-ac7a-5bb276afef42
spec:
  clusterIP: 10.0.112.172
  clusterIPs:
  - 10.0.112.172
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    controller-revision-hash: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-6f9dffbcbc
    postgres-operator.crunchydata.com/data: pgadmin
    postgres-operator.crunchydata.com/pgadmin: fdp
    postgres-operator.crunchydata.com/role: pgadmin
    sidecar.istio.io/inject: "false"
    statefulset.kubernetes.io/pod-name: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

from postgres-operator.

Thanks! Hmmm, that service spec looks good (or at least, it looks like mine). Just to double-check, are you running kubectl port-forward on the service after creating it?

from postgres-operator.

Port forwarding works fine, that is not the issue.

I can't access pgadmin using service name as host from a different pod, for example.

That leads to not being able to make pgadmin part of our service mesh. If I add istio-proxy sidecar and create a Virtual Service, it doesn't work, because istio can't reach pgadmin via service.

And that issue is specific to pgadmin, we have no issues with other applications in the cluster. I was able to modify the selector to point to one of our applications (instead of pgadmin) - and it works fine.

from postgres-operator.

Ah, OK, I think I'm following and now I can reproduce this issue.

To restate the issue:

(a) inside the pgadmin pod, I can curl localhost:5050 and get a result
(b) inside the pgadmin pod or another pod, I cannot curl the IP of the pod (or the service)
(c) for comparison, inside a PG pod, I can curl localhost:5432; and I get the same result if I curl the IP (inside the same pod or a different pod).

So I think the issue isn't with the service (since I get the same bad result when I try to access the pod directly). I wonder if the issue is with how the image is serving pgadmin4. (Though why does it work fine for kubectl expose/kubectl port-forward?) I'm going to run some tests with other images.

from postgres-operator.

Yeah, that's a nice summary.

I also got to the same conclusion about pgadmin image being the problem, but unfortunately I am not sure if there is something to be done on my side.

The only setting that caught my attention was PGADMIN_LISTEN_ADDRESS, but setting it to 0.0.0.0 didn't solve the issue.

from postgres-operator.

OK, @FNSdev, can you try putting this in your pgadmin spec:

  config:
    settings:
      DEFAULT_SERVER: "0.0.0.0"

It seems to work for me (i.e., I can curl localhost, curl the pod's IP, curl the service; and port-forward the service and access the website), but I feel uncomfortable with exposing a server like that. Really depends on the security of the network. Well, first, let's see if this even works for you.

My other thoughts on this issue are

(a) We have a ticket in our backlog re: the server on the pgadmin image; let me attach my notes from this conversation;
(b) I have minimal hands-on experience with Istio, but I wonder if we could work this problem from the other side: leave the pgadmin server serving on 127.0.0.1, and set Istio to route to localhost. (

(I'm not sure that's possible with Istio, though this article seems promising. If you want to explore that option, please share your Istio sidecar/virtual service specs.)

So, to recap:

• I think DEFAULT_SERVER: "0.0.0.0" might work in the short-term
• I wonder if changing the Istio configurations might offer a more secure solution
• I'm making notes for changing the image for a different solution

from postgres-operator.

Thanks a lot @benjaminjb, I will try that out after returning from holidays. Merry Christmas 🙂

I will also check the other approach with Istio, didn't know that it is possible.

maybe you could also explain to me what is PGADMIN_LISTEN_ADDRESS (set as env variable) and how is it different from DEFAULT_SERVER that we set in the config?

from postgres-operator.

Hmmm, I think PGADMIN_LISTEN_ADDRESS would be the right var to change but only if you were running the pgadmin Docker image -- that's part of their entrypoint script.

Have a good break!

from postgres-operator.

Hi,
I have the same issue and tested successfully the proposed solution (this article).
Here is my setup:

---
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGAdmin
metadata:
  labels:
    app: my-pgadmin
  name: my-pgadmin
spec:
  dataVolumeClaimSpec:
    accessModes:
      - 'ReadWriteOnce'
    resources:
      requests:
        storage: 1Gi
  serverGroups:
    - name: supply
      postgresClusterSelector: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-pgadmin
  name: my-pgadmin
  namespace: onap
spec:
  ports:
    - name: pgadmin
      port: 5050
      protocol: TCP
      targetPort: pgadmin
  selector:
    postgres-operator.crunchydata.com/pgadmin: my-pgadmin
  type: ClusterIP
---
# fix for pgadmin (https://github.com/CrunchyData/postgres-operator/issues/3809)
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: my-pgadmin
spec:
  workloadSelector:
    labels:
      postgres-operator.crunchydata.com/pgadmin: my-pgadmin
  ingress:
  - port:
      number: 5050
      protocol: TCP
      name: pgadmin
    defaultEndpoint: 127.0.0.1:5050

from postgres-operator.

@andreasgeisslerdt Thanks for providing that information. Glad to hear that the solution is working for you!

@FNSdev were you able to test the proposed solution in your environment?

from postgres-operator.

@tjmoore4 my apologies, I didn't have a chance to get back to this yet because of change in priorities.

good to see that it works for @srkoster 🙂

from postgres-operator.