opendatahub-io-contrib / jupyterhub-odh Goto Github PK

Hey, Kebechet!

Create a new minor release, please.

Describe the bug
I see this error: /tmp/scripts/assemble: line 16: cd: /opt/app-root/src/jupyterhub-singleuser-profiles/jupyterhub_singleuser_profiles/ui/: No such file or directory when checking the logs of the JupyterHub image buildConfig.

I also see this error error: build error: error building at STEP "RUN /tmp/scripts/assemble": error while running runtime: exit status 1 when checking the logs of the JupyterHub pod created by oc new-app --template jupyterhub-ocp-oauth

To Reproduce
Since the README in this repo points to imagestreams that don't exist, I am using the imagestreams in the jupyterhub-quickstart repo

Steps to reproduce the behavior:
1. oc apply -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/image-streams/s2i-minimal-notebook.json
2. oc apply -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/image-streams/jupyterhub.json
3. oc create -f https://raw.githubusercontent.com/aicoe/jupyterhub-ocp-oauth/master/templates.json
4. Change jupyterhub image tag in template from latest to 3.4.0
5. oc new-app --template jupyterhub-ocp-oauth
6. See errors: oc logs bc/my-jupyterhub-img and oc logs my-jupyterhub-img-1-build

Expected behavior
The JupyterHub pod should be running with no errors

Additional context
Are these the correct imagestreams I should be using or are there updated ones I can use?

Automated version release cannot be performed.
Related: #118

So, while trying to run tensorflow, I had to expose tensorboard.
I was able to achieve that by adding a label to my jupyterhub pod, creating a service to use that label as a selector and then exposing the route.

It would be nice to have that as a default whenever I use the tensorflow image to spin up jupyterhub.

As proposed, the easier thing to do would be to add the tensorboard jupyter extension to the base image.

Describe the bug
Source https://issues.redhat.com/browse/ODH-610

We have observed that a *User spawned Jupyterhub pod terminates right after the User's access token expires after 24 hours of creation of it. Is it an expected behavior (because I thought that the JupyterHub server continues running even after the token expires)? If not can you please guide us what can be the cause of this issue as it doesn't allow any of the users to *train their model continuously more than 24 hours which causes a huge problem.

ODH Version: 1.1.0
Openshift Version: 4.8.24
Platform: OVH Cloud

To Reproduce
Steps to reproduce the behavior:

1. Start a user notebook
2. Start a long running notebook that last > 24hrs
3. Notebook is terminated once the user token expires

Expected behavior
User can be automatically logged out but the notebook pod is not terminated. The user should be able to login a with the notebook stil running

Screenshots
N/A

Additional context
N/A

The building required images section of the README refers to image manifests that no longer exist: oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/images.json oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/images.json
These appear to have been moved in the jupyter-quickstart project, but it's not clear which images are needed to consume this project.

Describe the bug
If OAuth exchange fails, JupyterHub returns a bare 500 instead of some helpful error explanation.

[E 2021-05-10 11:01:46.555 JupyterHub web:1793] Uncaught exception GET /hub/oauth_callback?code=sha256~dbN9Os_LL5zXqJtIDDiVgDoKMkjVGtL86bKpMPTEAmI&state=eyJzdGF0ZV9pZCI6ICI0YWZhMWMzNTk0YTQ0ZTBjYjU3ODc3ZDQwYzgzM2RhMCIsICJuZXh0X3VybCI6ICIvaHViL2hvbWUifQ%3D%3D (::ffff:10.131.0.1)
    HTTPServerRequest(protocol='http', host='jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud', method='GET', uri='/hub/oauth_callback?code=sha256~dbN9Os_LL5zXqJtIDDiVgDoKMkjVGtL86bKpMPTEAmI&state=eyJzdGF0ZV9pZCI6ICI0YWZhMWMzNTk0YTQ0ZTBjYjU3ODc3ZDQwYzgzM2RhMCIsICJuZXh0X3VybCI6ICIvaHViL2hvbWUifQ%3D%3D', version='HTTP/1.1', remote_ip='::ffff:10.131.0.1')
    Traceback (most recent call last):
      File "/opt/app-root/lib/python3.6/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/opt/app-root/src/oauthenticator/oauthenticator/oauth2.py", line 224, in get
        user = await self.login_user()
      File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/opt/app-root/src/oauthenticator/oauthenticator/openshift.py", line 130, in authenticate
        resp = await http_client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: NSS: client certificate not found (nickname not specified)
    
[E 2021-05-10 11:01:46.562 JupyterHub log:181] {
      "X-Forwarded-For": "87.219.202.227,::ffff:10.131.0.1",
      "Forwarded": "for=87.219.202.227;host=jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud;proto=https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud",
      "Host": "jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Language": "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6",
      "Accept-Encoding": "gzip, deflate, br",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close"
    }
[E 2021-05-10 11:01:46.563 JupyterHub log:189] 500 GET /hub/oauth_callback?code=[secret]&state=[secret] (@::ffff:10.131.0.1) 20012.70ms

Expected behavior
Better error handling and guidance for the user to what they can do.

Screenshots

Additional context
This error can be fixed by user flushing JupyterHub browser cache + page refresh.

Is your feature request related to a problem? Please describe.

To get new notebooks to be available in the "Spawner options" one needs to restart the jupyterhub.

That's inconvenient at least and given JH user's in general need not have the permission to restart the JH pod it means that people might not be able to add their images at all.

Reference to relevant code:

https://github.com/opendatahub-io/jupyterhub-odh/blob/master/.jupyter/jupyterhub_config.py#L145,

https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/master/jupyterhub_singleuser_profiles/images.py

Describe the solution you'd like

I'd like to see new images in the drop-down automatically w/o restarting anything.

Either watch for changes on imagestreams and check them for the needed label or check for them when building the "Spawn Server" screen.

Additional context

I have a feeling that @vpavlin mentioned that this is being resolved somewhere, but I'd love to track this.

FYI @tumido

Describe the bug
One a jupyterlab instance has been created (with a previously selected image), the use is always redirected to that running instance, no chance to re-access jupyterhub to create additional instances, change the instance configuration (e.g. resources or containerimage

To Reproduce
Steps to reproduce the behavior:

1. in ODH launch "JupyterHub"
2. create a jupyterlab instance by selecting image, resources, ... in the jupyterhub interface
3. close everything
4. go to step 1 again => jupyterhub interface won't show up, you're directly redirected to the JL instance

Expected behavior
Show the JL interface to stop existing instances, allow for changing configs and re-launch..

Environment

• openshift:4.8.31_1546 on IBM Cloud
• ODH operator 1.1.2 stable
• Kubeflow version: build version dev_local (version number can be found at the bottom left corner of the Kubeflow dashboard):
• kfctl version: (use kfctl version):
• Kubernetes platform: openshift
• Kubernetes version: openshift:4.8.31_1546

Currently there is a simple service which polls proxy API and updates proxy for published notebooks route. This could be done without polling by using pre_spawn and post_stop hooks in the spawner which would only configure the new proxy route once

i.e. remove https://github.com/AICoE/jupyterhub-ocp-oauth/blob/master/.jupyter/jupyterhub_config.py#L24

add the public route setup/teardown to https://github.com/AICoE/jupyterhub-ocp-oauth/blob/master/.jupyter/jupyterhub_config.py#L303

Describe the bug
Creating a blank imagestream without any tags in it makes Jupyterhun spawner

To Reproduce
Steps to reproduce the behavior:

1. Submit an imagestream manifest
2. Open JH spawner
3. See that no images are found

Expected behavior
Imagestream is skipped

Screenshots

kind: ImageStream
apiVersion: image.openshift.io/v1
metadata:
  annotations:
    opendatahub.io/notebook-image-name: test
    opendatahub.io/notebook-image-url: test
  name: test
  labels:
    opendatahub.io/notebook-image: 'true'
spec:
  lookupPolicy:
    local: true

Produces:

Deleting the imagestream restores correct behavior.

Additional context
n/a

If a user selects spark notebook image we could add a route to JH proxy to expose spark-cluster-ui service

Implementation would be similar to #37, just need to check how to make sure this is done based on services section in singleusre-profiles

Description

This is an automated issue generated by Kebechet. The version manager threw an exception (TypeError) at
runtime. If you think this exception is a bug please open an issue upstream at https://github.com/thoth-station/kebechet
otherwise use the traceback below to help you fix whatever issues were encountered with your repository.

Traceback

Traceback (most recent call last):
File "/home/user/kebechet/kebechet_runners.py", line 193, in run
instance.run(**manager_configuration)
File "/home/user/kebechet/managers/version/version.py", line 446, in run
version_identifier, old_version = self._adjust_version_in_sources( # type: ignore
TypeError: cannot unpack non-iterable NoneType object

The repository got moved to https://github.com/opendatahub-io/jupyterhub-singleuser-profiles, so we need to update the requirements.txt

The repository is also still available as my personal fork, so untill we do fix this issue, things should work as expected

Hey, Kebechet!

Create a new patch release, please.

In attempt to standardize the image upgrade process, we need to document and implement a process and steps for tagging images.

I'm proposing that all JupyterHub images stick to a tagging format of
{jupyterhub image base}-{unique number}
where

• jupyterhub image base is the base builder image. Currently we are using the 3.0.7 image from jupyterhub-quickstart
• unique-number is any unique indentifier for this image build. It can be just a simple integer or the short hash of the commit this image was built from.

Based on the current base image 3.0.7 and repo commit of 952741b the new tag would be
jupyterhub:3.0.7-952741b

Is your feature request related to a problem? Please describe.
ODH JH comes with a pre-configured 20GB PVC which can't be increased from any ODH UI

Describe the solution you'd like
An ODH UI function to increase the JH PVC size

Describe alternatives you've considered
Login to open shift admin console, find PVC, resize and restart pods

Is your feature request related to a problem? Please describe.
Currently, JH only has access to basic information about the user - i.e. username. We'd like to be able to at least access groups information, but ideally also user's OpenShift token to be able to pass it into their Jupyter environment

Describe the solution you'd like

1. We need to create an OAuthClient resource on deployment
2. The JH config then needs to use OAuthClient name as client_id and the same secret specified in the OAuthClient
3. We need load the user information when we need it by calling

   auth_state = yield spawner.user.get_auth_state() #this has to be run in @gen.coroutine annotated method
   

4. Implement groups and token passage into JupyterHub SIngleuser Profiles

Describe alternatives you've considered
No alternative methods

Additional context

Is your feature request related to a problem? Please describe.
We have a public instance of Jupyterhub and sometimes users forget about their running servers which blocks our hardware resources even when they are not being utilized.
Related https://github.com/operate-first/SRE/issues/229

Describe the solution you'd like
We would like the jupyterhub-idle-culler module be included in the Jupyterhub image, so that the pod culling service can be set up from the jupyterhub-config.py configuration.

Describe alternatives you've considered
Current Alternative is to deploy the culling service as a separate deployment.

Describe the bug
JH server is throwing an error that the jupyterhub-hub serviceAccount is unable to list nodes at the cluster scope. This error is occuring on a long running JH serer that was previously working without any issues.

To Reproduce
Possible reproducer step:

1. Add notebook imagestreamTag and confirm it spawns successfully then shutdown the notebook
2. Modify the imagestreamTag to point to a new remote image
3. Attempt to spawn the new notebook and wait for the error

Expected behavior
JH Spawner does not throw permissions error when the jupyterhub-hub serviceAccount has the appropriate permissions

Screenshots
N/A

Additional context
N/A
jh-erorr.log

Describe the bug
Hey team, I am trying to install the fbprophet python package while using the s2i-minimal-notebook:3.6 image, but its installation fails.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://jupyterhub.datahub.redhat.com/
2. Click on s2i-minimal-notebook:3.6 in notebook image dropdown and then click Spawn.
3. Click on New -> Terminal to open a terminal.
4. Run pip3 install fbprophet
5. See error

Expected behavior
fbprophet installs without any errors.

Screenshots

Additional context
Not entirely sure, but the error seems related to the gcc (version 4.8.5) installed on the image (see screenshot). This package installs fine on my local machine (fedora 31), where the gcc version is 9.3.

Describe the bug
When the jh images are pulled in an env where the PVC is almost full, the pip install command that installs the image's dependencies leads to no space left on device and causes a crash loop error. There should be a better way to deal with this situation.

To Reproduce
Steps to reproduce the behavior:

1. Go to your jh instance and fill up your pvc, shut it down
2. Spawn the pvc again with an image that requires pip installs
3. See error

Expected behavior
There should either be a warning before I shut my instance that says that I should delete my pvc contents or a minimal env should be spawned when I have no space left so that I can delete contents.

Screenshots

Hey, AICoE-CI!

Please build and deliver the following git tag:

Tag: v0.3.2-test

Hey, AICoE-CI!

Please build and deliver the following git tag:

Tag: v0.1.1

Hey, AICoE-CI!

Please build and deliver the following git tag:

Tag: v0.2.2

With the kubespawner update to 0.11.1 in #44, many variables available in the Kubespawner class have been updated to use new names.

List of Kubespawner deprecated traits

Is your feature request related to a problem? Please describe.
Currently, when companies use repositories, regardless of whether for Docker or for machine learning packages or for pipeline extensions (python, R), connecting to such servers is not possible when the use SSL certificates that were generated by a custom, internal PKI. This is due to those SSL certificates not being publicly trusted. There are ways in openshift to define trusted CAs at a cluster-level and to auto-inject them into a configmap at the namespace-level.

https://docs.openshift.com/container-platform/4.8/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

Describe the solution you'd like
Auto-inject configmap containing trusted-bundle CAs should be added to namespace. Map / pem certs should then be mounted to a spawned jupyter lab container, be it Elyra or others. proposed location /opt/app-root/etc/jupyter/custom/certs

configmap example from another project

https://github.com/trevorbox/reloader-operator/blob/f07d1858825cc8515f45c2cf03b84c23e994aa7e/helm/app/templates/configmap-trusted-cabundle.yaml

and mounting it into a location in the spawned container that e.g. python then uses for request.get trust later

https://github.com/trevorbox/reloader-operator/blob/f07d1858825cc8515f45c2cf03b84c23e994aa7e/helm/app/templates/app-nginx-echo-headers.yaml#L50

Describe alternatives you've considered
PKI internal CA trust support is needed, regardless which clients (python, curl, others) are accessing urls of servers containing resources.

Additional context
See e.g. Elyra Ticket conceptual notes

elyra-ai/elyra#2797

there, the question came up in the context of Airflow pipeline components

and also air-gapped elyra elyra-ai/elyra#2812

The non-publicly-trusted CA-bundle-issue is also touched on here https://github.com/opendatahub-io/odh-manifests/issues/575#issuecomment-1167486544

Describe the bug
I get a 500 error about certificate validation after authenticating on the oauth endpoint. I have tried to inject the CA bundle using [1] but the system CA seems to be ignored.

I suspect this is related to this: jupyterhub/oauthenticator#411 but I would be happy to get feedback on my investigation.

[1] https://docs.openshift.com/container-platform/4.6/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

Describe the bug
Source https://issues.redhat.com/browse/ODH-610

When we log out and login back again either on the Spawner page or from the running instance of a user, it doesn't get us back to the Login Page of KeyCloak (integrated into our cluster for Login). Actually, after logging out we click on the JupyterHub button in the top left corner of the navigation bar and it takes us back to the last page where we were before logging out.

We can see in the logs of the JupyterHub pod that the User logged out and logged into the instance but cannot infer it from the UI of the instance and we are not able to clearly understand if it logs in with a new token or the old one.

ODH Version: 1.1.0
Openshift Version: 4.8.24
Platform: OVH Cloud

To Reproduce
Steps to reproduce the behavior:

1. Start a user notebook
2. Logout of the user notebook
3. User is not redirected back to the OpenShift authentication

Expected behavior
When a user manually logs out they should be redirected by the the JH page allowing user login

Screenshots
N/A

Additional context
N/A

Jupyterhub Singleuser Profiles currently only has information about the username of the current user. To enable advanced functionality, such as sharing volumes between users of the same group, it is neccessary to gather data about the user group from the openshift auth. This issue will be connected to an Issue/PR in the jupyterhub-odh repo.