Failed to connect D-Tale process in hosted notebook about dtale HOT 7 CLOSED

@felihong I think the problem here is that you can't map both port 8888 (notebook) and 40000 (d-tale) to port 80. If you pick a different port it should work

e.g.

ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  - name: http-notebook-dtale
    port: 40000
    protocol: TCP
    targetPort: 40000

Also how are you exposing notebook? I see from the snippet that the k8s service it is a ClusterIP service. Do you also have an Ingress? If so, you also need to update that. Posting your Ingress config if you need more help.

from dtale.

Oooh, kubernetes is a tough one. Let me talk to our core team and get back to you. We were able to get it running but it did take some work

from dtale.

Hi @felihong , you can map d-tale (port 4000) to a different path (e.g. /dtale) of the same ingress, given there is not to be an conflict with jupyterhub. Let me know how it works for you :)

from dtale.

@felihong I just added this documentation on how we have D-Tale interacting with Kubernetes when using JupyterHub

Hope this helps. Let me know if you're still encountering errors. You'll probably want to get whomever manages jupyterhub or kubernetes involved as well.

from dtale.

Hi @aschonfeld , thanks for the detailed explanation!

Below a snippet of the original configuration of my notebook pod's service:

spec:
  clusterIP: <cluster-ip>
  ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  selector:
    statefulset: notebook
  sessionAffinity: None
  type: ClusterIP

My intuition was modifying the service with multiple target ports, which should be supported by K8S as multi-port services

ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  - name: http-notebook-dtale
    port: 80
    protocol: TCP
    targetPort: 40000

However this is immediately reconciled by my notebook controller and restored to the previous value (Has something to do with the default config). So I created a new service targeting to the notebook pod, with port value 80 and targetPort value 40000. Thus my pod has two services being exposed. But nothing has changed.

Interestingly, I can access to the DTale page (by visiting the ip directly) if I changed the service's type to LoadBalancer, which allocates an external ip endpoint to the dashboard page. Something is still going wrong with the internal traffic...

from dtale.

Hi @javefang , sorry for the late reply.

You are right about the mapping, I specified the port as 40000 and the service yaml can be now updated properly. Thanks for pointing out.

My cluster is configured with an Ingress, which serves the main endpoint using istio-ingressgateway service:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-cert: <SSL_CERT>
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.global-static-ip-name: <IP_NAME>
    networking.gke.io/managed-certificates: gke-certificate
  name: envoy-ingress
  namespace: istio-system
  selfLink: /apis/extensions/v1beta1/namespaces/istio-system/ingresses/envoy-ingress
spec:
  rules:
  - host: <HOST_NAME>
    http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
status:
  loadBalancer:
    ingress:
    - ip: <IP_NAME>

Should I add another backend to pointing to servicePort 40000? Thanks for your help!

from dtale.

@javefang if you're still having issues I just recently released v.1.8.13 with support for a jupyterhub extension jupyter-server-proxy. Here's the documentation

Hopefully this might help with using D-Tale in jupyterhub on top of kubernetes

from dtale.