This repository houses Watson Assistant Improve notebooks and the underlying assistant improve toolkit library.

To help improving your Watson Assistant after you have deployed it to production, we prepared the following Jupyter notebooks. These notebooks include practical steps for measuring, analyzing, and actively improving your assistant in a continuous manner. Check out IBM Watson Assistant Continuous Improvement Best Practices for more details.

• Measure notebook contains a set of automated metrics that help you monitor and understand the behavior of your system. The goal is to understand where your assistant is doing well vs where it isn’t, and to focus your improvement effort to one of the problem areas identified. This notebook generates an assessment spreadsheet for you to use to label problematic conversations, and then feed to the Effectiveness notebook.

• Effectiveness notebook helps you understand the relative performance of each intent and entity as well as the confusion between your intents. This information helps you prioritize your improvement effort. The input to this notebook is an assessment spreadsheet generated from the Measure notebook. Update the marked columns in the spreadsheet with your labels and load it into the Effectiveness notebook for analysis.

• Logs notebook helps you fetch logs using Watson Assistant API. You can fetch logs with various filters, and save them as a JSON file, or export the utterances in the logs into a CSV file. The JSON file can be loaded into the Measure notebook. The CSV file can be used for intent recommendation service. Alternatively, you can run python scripts fetch_logs and export_csv_for_intent_recommendation to fetch logs and export them to intent recommendation CSV, respectively. Run python get_logs -h and python export_csv_for_intent_recommendation.py -h for usage.

• Dialog Flow Analysis notebook help you assess and analyze user journeys and issues related to the dialog flow of ineffective (low quality) conversations based on production logs. Check out Dialog Flow Analysis for more details.

• Dialog Skill Analysis notebook help you analyze characteristics of your data such as the number of training examples for each intent or the terms which seem to be characteristic of a specific intent. Check out Dialog Skill Analysis for more details.

You can either run the notebooks locally or in IBM Watson Studio.

• Run locally

  1. Install Jupyter Notebook, see Jupyter/IPython Notebook Quick Start Guide for more details.
  2. Download the Jupyter notebooks available in this repository's notebook directory. Note: These notebook files are not designed for Watson Studio environment
  3. Start jupyter server jupyter notebook
  4. Follow the instructions in each of the notebooks. Be sure to add your Watson Assistant credentials if necessary.

• Run in Watson Studio

  1. Create a Watson Studio account.
     Sign up in Watson Studio, or use an existing account. Lite plan is free to use.

  2. Create a new project and add a Cloud Object Storage (COS) account.
     For more information regarding COS plans, see Pricing.

  3. Copy Measure or Effectiveness notebook from Watson Studio community into your project.

  4. Follow the instructions in each notebook to add project tokens and Watson Assistant credentials if necessary.

• Learn more about our measure and effectiveness notebook on Medium: Continuously Improve Your Watson Assistant with Jupyter Notebooks

See CONTRIBUTING.md for more details on how to contribute

This library is licensed under the Apache 2.0 license.