Advice needed for long running jobs on hubs about docs HOT 2 CLOSED

For QCL, I think they merit from having all culling logic disabled to avoid issues - but warn them that they need to shut down their own servers if they aren't using them.

If they have very expensive machines running long duration, and they incorrectly fail along the way due to culling, that is the far bigger cost I expect.

Related

• #168

Action points

• Investigate basehub's jupyterhub-idle-culling configuration in jupyterhub
  jupyterhub-idle-culler is not configured in basehub, but it is by default in z2jh to cull servers with no activity reported in the last hour
• Investigate basehub's kernel culling configuration in user servers
  Kernels are not culled if they are busy by the kernel culling, but idle kernels are after one hour of idling
• Investigate if kernel culling is something you opt-in or opt-out of, so that we understand the consequences of removing config via basehub
  It seems that cull_idle_timeout is defaulting to 0, with culling of idle kernels disabled.
  We are setting it to 3600, which means that a server with a long running job will loose its state after 3600 seconds.
• Read up on what Min helped me understand once in jupyterhub/jupyterhub-idle-culler#55
• Make a decision on how to best help QCL avoid possible disruption of long running jobs
  • Idea 1: disable kernel culling to avoid loosing state after a long computation completes
  • Idea 2: disable jupyterhub-idle-culler to avoid loosing server after a period of inactivity
• Consider if and how we want to update our docs and default config for basehub
  Yes. But I'll open a separate issue about it.

from docs.

Advice provided, I'll probably reconfigure something for QCL as a followup so I re-assigned myself to the support ticket.

from docs.