This repository includes a fully functional continuous integration and continuous deployment system for Jupyter Notebooks.

The goal of this repository is to showcase what can be built using notebook-centric development practices, and we aim for it to be a reference implementation for those who want to develop similar systems on Google Cloud. For the purposes of this example, we have created the sample notebook demo.ipynb, which takes in a bigquery dataset to generate a usage dashboard, and have set up automated testing and deployment for this notebook.

Each individual file has been documented with an explanation of its use in the CI+CD system, and we will cover the architecture below.

In our setup, we have leveraged the following services/products of Google Cloud Platform:

Core services:

• AI Platform Notebooks
• Cloud Deep Learning VMs (DLVM)
• Cloud Build
• Cloud Storage (GCS)
• GitHub (or Cloud Source Repository)
• Cloud Functions (CD only)
• Cloud Pub/Sub (CD only)

Extra glue (may vary depending on your setup):

• Secrets Management
• Cloud Scheduler (CD only)

Overview of CI component:

In this flow, a user is developing on an AI Platform Notebook instance. He pushes his notebook with changes to the Github repository, which triggers a git hook and in turn a Cloud Build job (cloudbuild.yaml). This Cloud Build job creates a testing instance on which the notebook will be executed, and results of the execution will be reported to Cloud Storage for the user to examine.

Let's follow what happens when a user submits broken code in the notebook.

• User pushes commit that breaks notebook.
• GitHub (or Cloud Source Repository) has a special pre-configured hook that triggers Cloud Build each time when a new commit is pushed to master. Cloud Build has provided a guide to set up this git hook here.
• Cloud Build looks for the file cloudbuild.yaml that describes the testing steps. In our example, this file:
  • clones the repository
  • checkout required commit under test
  • executes testing logic in run_notebook_instance.sh which does the following steps:
    • upload notebook (you can upload other files if needed) to GCS
    • use gcp-notebook-executer to start executing the notebook
    • wait until notebook execution is finished
    • check output directory in cloud storage
  • upload tested execution code to GCS and Cloud Functions (used later for CD)
• mark build as green or red

1. Make sure that your notebook self-contained and can be executed on top of one of available Cloud Deep Learning VMs image families
2. Add Cloud Build hook to your repository
3. Create testing script run_notebook_instance.sh based on our script. In the script you need to customize:
   • image family that need to be used
   • VM configuration (GPU type, VM shape, etc)
   • name of the notebook
   • you might need update logic that uploads dependencies to cloud storage
4. Create cloudbuild.yaml similar to the one from this repository.

Overview of CI component (updated):

Overview of CD component:

Now that we have a tested, reproducible notebook, let's automate the execution of the notebook on a set schedule. As the last step of the CI workflow, let's deploy a Cloud Function which runs the same notebook execution code with modifiable parameters (function code). For orchestration we'll use Cloud Scheduler, but this could be any code/utility/trigger that can publish a message to Pub/Sub (for example, new data arrives in GCS, or a manual request to execute the notebook). Once the schedule is set up, new notebook results will automatically appear in the configured output directory.

• Execute an iteration of the CI workflow, if successful this will deploy the Cloud Function and create a Pub/Sub topic demo-notebook which the function subscribes to.
• Create a Cloud Scheduler job: frequency can be any cron style schedule (e.g. 0 0 * * 0 = every sunday midnight), target should be Pub/Sub with the topic demo-notebook, and the payload should be {}, which will use the current date (behavior defined in the function).

If you have further questions, or encounter issues using AI Platform Notebooks or Deep Learning VMs, reach out at our team's forum: https://groups.google.com/forum/#!forum/google-dl-platform