Add a configurable to toggle between AMQP based queues and SQLAlchemy jobs options.

Update the development setup instructions to clarify the steps to install the packages required for the development environment and how to install the package in editable mode.

Replace tests based on Selenium with tests that use nbgrader's RESTful API endpoints

Overview

IllumiDesk needs to re-design the auto-grading services to improve scalability and decouple the service from the rest of the LMS system.

Desires and Goals

• The auto-grading service can be called from any client using standard APIs (RESTful, etc)
• Like the rest of the system, requests should be authenticated
• Leverage native Kubernetes services to the extent possible to manage jobs
• Maintain compatibility (at least for the first iteration) with the nbgrader database schema and nbgrader ipynb schema
• Allow system administrators to adjust CPU / Mem settings for auto-grading containers
• Enable notifications based on job status (completed, failed, in-progress, etc)
• Log auto-grading jobs to a central location via stdout/stderr (thus allowing administrators to integrate the logging tool(s) of their choice.

Notes

• The nbgrader auto-grading feature is tightly coupled with the Jupyter Notebook. The async-nbgrader project helps but need to take this a step further and siphon off the auto-grading service into a more traditional queue/task-based job architecture.
• The queue/job architecture setup naturally has many options. We are more inclined to stick with a Kubernetes-native approach. This of course has pros/cons. If we use something with native Kubernetes (Argo, K8s jobs, etc) then it's more complex to setup and maintain, however it would allow us to provision this with any cloud vendor if we chose to do so and/or offer this with the IllumiDesk Enterprise version. Perhaps when managing this with our SaaS solution we could combine K8s with a managed service, such as Fargate via plugins/configs/middleware but that is wait-and-see.
• The queue/job architecture should be generic enough so that we can receive notifications based on the status of the desired state and alert the end-user and/or internal IllumiDesk collaborators based on certain rules.
• We could use the gofer_service and gofer_submit projects as sources of inspiration.

Nice to haves

• Once the auto-grading has been completed, allow users to configure whether or not they need to auto-submit grades to their LMS either with LTI 1.3 or third-party integration service (Zapier, generic Webhooks, etc).

Diagrams and Architecture

This diagram has a basic first draft of how the re-architecture could look.

Resources

• For the background workers which can be configured by the end-user, we can use Horizontal Pod Autoscaler dependent on external metrics: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects (this will need change in our helm-chart)
• Background worker with kubernetes - https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/

Complications identified

• Worker containers would need shared volume access of the jupyter, in order to read the notebook. This is on basis of the current solution where we are using file storage.
• Acquiring locks on basis of Notebook so that multiple workers don't write to the same notebook

• Add automation for GitHub Actions

• Add flake8
• Add black
• Add isort
• Integrate codecov
• Add pre-commit hook
• Run unit tests with GitHub Actions