opendatahub-io-contrib / jupyterhub-singleuser-profiles Goto Github PK

If the JupyterHub pod is unable to be started, user is presented with a generic

This is really not helpful at all.

We figured that in this particular case it was caused by the PVC being full. We figured that our from the Pod's logs before it was killed. In this case restarting the pod doesn't help at all, because the PVC remains full.

Failed to write server-info to /opt/app-root/src/.local/share/jupyter/runtime/nbserver-144.json: [Errno 122] Disk quota exceeded

Would it be possible to show the pod's log in the UI or any indication to the problem's cause?

Do we need to have the ui/templates/page.html?

Currently, the main profile menu at the top of the spawner is driven by container images it finds out on the cluster. This works reasonably well but it has a couple disadvantages.

1. it isn't intuitive for users to conceptualize the profiles by image
2. in order to provide users with multiple options, I had to create several container images that were all identical except for the name, so that I could "hang" different spark size profiles (and env vars, etc) from them.

I believe a cleaner organization would be to simply declare the profiles in the configmap by a name (for example, "Jupyter with Large Spark Cluster"), and then the actual jupyter image is just another property of the profile, it isn't driving the menu choices per se. For example in this scheme, I would generate a single cusomized jupyter image, and it would be the same value under all the profiles. The profiles would differ only in terms of the configuration of spark cluster size (or other service configurations, etc).

Hey, Kebechet!

Create a new minor release, please.

I forgot to mention that in the PR (https://github.com/vpavlin/jupyterhub-singleuser-profiles/pull/21), but we should add documentation for this to README - especially since the code got moved here from the other project - can you come up with a section on this in README @mroman-redhat ?

Right now the GPU field is shown all the time. We should make this configurable in profiles - i.e. add a feature which would only show the GPU field for a list of users and/or images (so that a random user selecting minimal image does not block a GPU by mistake or intentionally)

This "watcher" should watch pods related to a particular user and record major events. These events could then be visualized in JH or ODH dashboard for status/error reporting.

• Design the "watcher" process
• Design where/how the events are stored (e.g. in a special section in user ConfigMap)
• Propose how the events could be surfaced for the user (e.g. in JH - what is running for the user, what failed and why)

e.g.

• notebook pod got OOM killed
• pods deployed via services integration fail/are killed/are unschedullable
• notebook pod and all the services are running successfully - show some green lights in JH/ODH dashboard

CC @mroman-redhat @erikerlandson

An admin should be able to configure the limit for GPU allocation - somewhere in the main ConfigMap set

gpu:
  limit: NUMBER
  style: input/checkbox

UI should then respect the style and enforce the limit

WDYT? @mroman-redhat @altruistcoder

Hey, Kebechet!

Create a new minor release, please.

Hey, Kebechet!

Create a new patch release, please.

Hey, AICoE-CI!

Please deliver the package module for the following tag, as it missing on PyPI:

Tag: v0.1.1

Hey, Kebechet!

Create a new patch release, please.

Currently all env vars defined in the Spawner UI are in plain text and stored in ConfigMap.

We'd like to be able to switch the input type to password on demand and store such values in secret rather than CM.

• Add a checkbox to env var form with label "Secret"
• When the checkbox is checked make the value type password instead of text
• Env vars with the checkbox checked need to be stored/loaded via secret rather than CM

Dropdown value check is implemented twite - in ImageForm and SizeForm, it should be a function of Dropdown

Hey, Kebechet!

Create a new patch release, please.

We should provide an ability to set PVC size per user/profile similarly to what we do with memory and CPU resources.

Hey, Kebechet!

Create a new patch release, please.

eg. https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/df2f274261d7112e74e956df73cfa958d8b9a3ac/ui/src/EnvVarForm/components/EnvVarForm.js#L36

eg. https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/2ae009c1158d276c868bba37a7c83e458910a6b2/jupyterhub_singleuser_profiles_ui/src/ImageForm/components/ImageForm.js#L37

There are better ways to check item existence - https://www.tutorialrepublic.com/faq/how-to-check-if-a-value-exists-in-an-array-in-javascript.php

Concept would be to modify the spawner to handle more advanced env-var structures, like:

    - env:
      - name: MYAPP_SECRET_TOKEN
        valueFrom:
          secretKeyRef:
            key: SECRET_TOKEN
            name: secret-token-for-my-app

see for example:
https://major.io/2018/12/06/use-secret-as-environment-variable-in-openshift-deployments/

Hey, Kebechet!

Create a new patch release, please.

@vpavlin Deployed JupyterHub as per the readme, using 'master' branch, on OpenShift Dedicated 3.11. Selected s2i-minimal-notebook:3.6, and clicked 'Spawn'. Get "Error: name 'config_map_name' is not defined". Logs from JupyterHub pod:

[D 2019-02-11 18:13:07.155 JupyterHub base:603] Initiating spawn for afeiszli

  | [D 2019-02-11 18:13:07.155 JupyterHub base:609] 0/100 concurrent spawns
  | [D 2019-02-11 18:13:07.155 JupyterHub base:612] 0 active servers
  | [E 2019-02-11 18:13:07.245 JupyterHub user:477] Unhandled error starting afeiszli's server: name 'config_map_name' is not defined
  | [D 2019-02-11 18:13:07.253 JupyterHub user:578] Deleting oauth client jupyterhub-user-afeiszli
  | [E 2019-02-11 18:13:07.264 JupyterHub pages:148] Failed to spawn single-user server with form
  | Traceback (most recent call last):
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/pages.py", line 146, in post
  | await self.spawn_single_user(user, options=options)
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user
  | timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
  | await spawn_future
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn
  | raise e
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 405, in spawn
  | await maybe_future(spawner.run_pre_spawn_hook())
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/spawner.py", line 716, in run_pre_spawn_hook
  | return self.pre_spawn_hook(self)
  | File "/opt/app-root/src/.jupyter/jupyterhub_config.py", line 244, in setup_environment
  | spawner.single_user_profiles.load_profiles(username=spawner.user.name)
  | File "/opt/app-root/lib/python3.6/site-packages/jupyterhub_singleuser_profiles/profiles.py", line 76, in load_profiles
  | print("Could not find config map %s" % config_map_name)
  | NameError: name 'config_map_name' is not defined

PR https://github.com/vpavlin/jupyterhub-singleuser-profiles/pull/22 allows us to use Kubernetes native format, but we should also allow "not setting" one of the requests/limits.

The actual issue seems to be our form dropdown for sizes - https://github.com/vpavlin/jupyterhub-singleuser-profiles/blob/master/jupyterhub_singleuser_profiles/sizes.py#L42-L46 - it would be good to use .get() to access the values in dict to make sure it does not crash if some value is not set - which is ok and a default value would be used.

I merged the PR #39 and then realized there is still the development URL. Can you please fix it?

Based on document: https://docs.google.com/document/d/1BR9iVY9IkOEdgl4JA5Hg-C_9wSVGKEAZ1FMik6znQD8

• #30 API Definition
• #40 UI Definition
• Replace spawner

cc @mroman-redhat

See: https://github.com/AICoE/jupyterhub-ocp-oauth/blob/master/.jupyter/jupyterhub_config.py#L177-L196

This is used to get a list of images (ImageStreams actually) from OpenShift API and display it on Spawner UI.

Refactor this so that the code lives in singleuser profiles and we only need to call the function from the config.py

I had a discussion with @guimou about potential feature for the profiles - adding a set of volumes mounted to the singleuser pod.

This would be useful for

• a set of users want to have access to a shared volume (requires RWX PV(C)s)
• a workshop requires some dataset/examples and instead of pulling it from S3, we want to have it mounted in the container
• we want to provide additional storage to a set of users
• specialized binaries/libraries mounted into the container

The above examples would require a JupyterHub admin to create the PVCs, add data and specify the references in profiles config

Another use case with slightly different implementation (kind of vice versa to above) might be to provide a feature to label a set of PVCs which would then be loaded and presented in the spawner UI (similarly to Images and Sizes at the moment) to make example datasets, workshop materials or generally shared storage available to users. e.g.:

jupyterhub.opendatahub.io/volume: public
jupyterhub.opendatahub.io/volume-access: ro / rw
jupyterhub.opendatahub.io/volume-group: opendatahub

The volume: public means it would be offered to all the users, volume-group: opendatahub means it would be offered to all the users in group opendatahub etc.

ETA: 3 weeks

Is your feature request related to a problem? Please describe.
Users don't understand the "meaning" of images, its content and which is the best to spawn for them

Describe the solution you'd like
A tooltip similar to the sizes selection

Describe alternatives you've considered
n/a

Additional context

@pacospace I think we can turn that into an RFE for JH spawner. It would be nice if you can see some (rich? markdown with links?) description to the images (possibly listing the components and so on) here:

Similarly to the size tooltip:

I know this would have to be OpenShift specific, because ImageStreams are OpenShift only. @pacospace @vpavlin do you know where would be the best place to file an RFE issue for it?

I love this idea @tumido. Let's do it!!

Originally posted by @pacospace in operate-first/support#143 (comment)

cc @mroman-redhat

Do this after #15

Currently the list of images is generated by getting all ImageStreams from the namespaces and filtering by their name - looking for suffix -notebook.

To follow the best practices, we should use labels (metadata.labels) to find the ImageStreams which we want to list.

If the ImageStream has a label opendatahub.io/notebook-image: true, it should be added to the list, ignored otherwise

FYI @erikerlandson

Hey, Kebechet!

Create a new patch release, please.

There are multiple Api clients instances being used across the project. It would be good to unify the initialization and use a single instance of client - probably the OpenShift Dynamic Client since we work with both Kubernetes and OpenShift resources

Currently a service is deployed by processing a template (https://github.com/vpavlin/jupyterhub-singleuser-profiles/blob/master/jupyterhub_singleuser_profiles/service.py#L57). Since templates are going away we need to figure out a different way.

We could probably include the service resource directly in the singleuser configMap - for Spark that would mean:

Current:

          spark:
            template: 'jupyterhub-spark-operator-configmap'
            parameters:
              WORKER_NODES: '2'
              MASTER_NODES: '1'
              MASTER_MEMORY_LIMIT: '2Gi'
              MASTER_CPU_LIMIT: '1'
              MASTER_MEMORY_REQUEST: '2Gi'
              MASTER_CPU_REQUEST: '1'
              WORKER_MEMORY_LIMIT: '2Gi'
              WORKER_CPU_LIMIT: '1'
              WORKER_MEMORY_REQUEST: '2Gi'
              WORKER_CPU_REQUEST: '1'
              SPARK_IMAGE: 'quay.io/opendatahub/spark-cluster-image:spark22python36'
            return:
              SPARK_CLUSTER: 'metadata.name'

Future:

spark:
  resources:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        radanalytics.io/kind: SparkCluster
      name: spark-cluster-${USERNAME}
    data:
      config: |-
        worker:
          instances: 2
          memoryLimit: 2Gi
          cpuLimit: 1
          memoryRequest: 2Gi
          cpuRequest: 1
        master:
          instances: 1
          memoryLimit: 2Gi
          cpuLimit: 1
          memoryRequest: 2Gi
          cpuRequest: 1
        customImage: 'quay.io/opendatahub/spark-cluster-image:spark22python36'
        env:
        - name: SPARK_METRICS_ON
          value: prometheus
  return:
    SPARK_CLUSTER: 'metadata.name'

We could also link a configMap/secret containing the resource

Thoughts, @LaVLaS @erikerlandson

Currently, we have resources limited to setting limit for memory and cpu. We'd like to be able to set requests and limits, so it would be best to standardize on the structure used by Kubernetes (https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)

We would like to keep the current structure working for backwards compatibility

• Change how resources are defined, processed and set in profiles.py
• Change how resources used in sizes.py
• Change README to show new structure of resources in sizes and profiles

CC @erikerlandson

We currently only set nvidia/gpu flag in resources of the Pod when GPU count is higher than 0. This might cause issues when a pod lands on GPU node by chance - IIUC if the nvidia/gpu: 0 is not specified, it will get access to all GPUs on the node.

The fix is simple - we just need to move the condition (https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/master/jupyterhub_singleuser_profiles/profiles.py#L284) from the top of the function to the mode check.

Before a dynamic UI can be made for the Spawner, an API needs to be created. Swagger API will be used for this.

A basic React UI will also be created for development purposes, which will be replaced by a proper UI later on

This will be divided into tasks and more will be added as necessary:

• Get sizes

• Get Images

• Get user profile

• Modify env vars

• Modify selections

Project information:
https://docs.google.com/document/d/1BR9iVY9IkOEdgl4JA5Hg-C_9wSVGKEAZ1FMik6znQD8/edit?usp=sharing

Happens randomly after refreshing the page.

See datashim-io/datashim#40

@Guimo this could be interesting for you as well:-)

https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/72282dc75cd985f49ff81e19c30c17686b761aef/ui/src/SizesForm/components/SizesForm.js#L47

This https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/72282dc75cd985f49ff81e19c30c17686b761aef/ui/src/EnvVarForm/components/EnvVarForm.js#L40-L41 could be merged into as single if

To be able to configure the UI to provide only the neccessary information and in a correct manner, a configuration structure needs to be applied such as:

images:
  visible: False
gpu:
  limit: 4
  style: "input"
---------------- or
gpu:
  style: "dropdown"
  values:
    - 0
    - 2
    - 4
---------------- or
gpu:
  style: "checkbox"
envvars:
  style: "limited"/"unlimited"
  variables(if limited):
    - "example1"
    - "example2"

That is to say, the admin should be able to set whether a component is visible, what style it has (if applicable). Environment variables could be limited for example and only pre-written, GPU could be set by either a dropdown, checkbox or input. And any component could be set to not appear in the UI if neccessary.

the :latest image of the operator may break your api

IDs in HTML tags need to be unique. Review the UI react code generating the HTML to either use unique IDs everywhere, or switch some of those to class - e.g. droptxt

Example profile:

- name: globals
    env:
    - name: THIS_IS_GLOBAL
      value: "This will appear in all singleuser pods"
    resources:
      requests:
        memory: "500Mi"
        cpu: "250m"
      limits:
        memory: "1Gi"
        cpu: "500m"
    volumes:
    - name: dataset
      persistentVolumeClaim:
        claimName: example-dataset-pvc
      mountPath(1): /opt/app-root/src/example-dataset
      mountPaht(2): ./example-dataset (add prefix)
      (3) no-mountPath: /opt/app-root/src/$(name)

• Replace hardcoded path in https://github.com/opendatahub-io/jupyterhub-odh/blob/master/.jupyter/jupyterhub_config.py with variable
• Add argument to SingleuserProfiles init to pass default mount path
• Enable 3 alternatives - absolute path (use the exact value), relative path (prefix the value with default mount path), no path (use name of the volume as a relative path)
• Add volume and volumeMounts items to pod in apply_pod_profile

References:

• https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md
• https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1VolumeMount.md

Implements first use case of #38

https://github.com/NVIDIA/nvidia-container-runtime#environment-variables-oci-spec

Possibly not strictly required if the base image already sets these.

      - name: NVIDIA_VISIBLE_DEVICES
        value: all
      - name: NVIDIA_DRIVER_CAPABILITIES
        value: compute,utility
      - name: NVIDIA_REQUIRE_CUDA
        value: cuda>=9.0

This library currently offers some customizability of the pod manifest (env vars, resource limits, taints/tolerations..). There have been feature requests for more flexibility and ability to be able to more flexibly customize the pod manifest (e.g. ports, volumes...)

cc @erikerlandson

In the logs for v1.0.0 of ODH I've noticed that jsp's flask app is claiming to use the built-in WSGI server, which is not meant for production use. Can we swap this out for something like gunicorn or gevent?

Happy to work on a PR for this.