Please drop me a note on Slack or by mail [email protected] if you find glitches or problems.

Those are the steps that you have to execute to install a complete demo environment:

1. TLDR - Fast Track
2. AI Manager Installation
3. AI Manager Configuration
4. Event Manager Installation
5. Event Manager Configuration
6. Slack integration
7. Demo the Solution

❗You can find a PDF version of this guide here: PDF.

🚨🚨🚨🚨 📺 Here is a video that walks you through the complete installation process.

These are the options you have to install the demo environment.

1. In the the OCP Web UI click on the + sign in the right upper corner
2. Copy and paste the content from this file
3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1 (the Entitlement key from https://myibm.ibm.com)
4. Click Save

1. In the the OCP Web UI click on the + sign in the right upper corner
2. Copy and paste the content from this file
3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1 (the Entitlement key from https://myibm.ibm.com)
4. Click Save

1. In the the OCP Web UI click on the + sign in the right upper corner
2. Copy and paste the content from this file
3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1 (the Entitlement key from https://myibm.ibm.com)
4. Click Save

1. In the the OCP Web UI click on the + sign in the right upper corner
2. Select the content file from this directory
3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1 (the Entitlement key from https://myibm.ibm.com)
4. Click Save

ℹ️ If you get a ClusterRoleBinding already exists, just delete it at the beginning of the YAML

This is provided as-is:

• I'm sure there are errors
• I'm sure it's not complete
• It clearly can be improved

❗This has been tested for the new CP4WAIOPS v3.5.0 release on OpenShift 4.10 on ROKS

So please if you have any feedback contact me

• on Slack: @niklaushirt or
• by Mail: [email protected]

Clone the GitHub Repository

git clone https://github.com/niklaushirt/cp4waiops-deployer.git

I installed the demo in a ROKS environment.

You'll need:

• ROKS 4.10
• 5x worker nodes Flavor b3c.16x64 (so 16 CPU / 64 GB)

You might get away with less if you don't install some components (Event Manager, ELK, Turbonomic,...) but no guarantee:

• Typically 4x worker nodes Flavor b3c.16x64 for only AI Manager

IBMers can get a temporary one from Techzone (ususally valid for 2 weeks)

1. Create a cluster for Practice/Self Education if you don't have an Opportunity Number

   

2. Select the maximum end date that fits your needs (you can extend the duration once after creation)

   

3. Fill-in the remaining fields

   1. Geograpy: whatever is closest to you
   2. Worker node count: 5
   3. Flavour: b3c.16x64
   4. OpenShift Version: 4.10

   

4. Click Submit

❗ Only needed if you decide to install from your PC

You need the following tools installed in order to follow through this guide:

• ansible
• oc (4.8 or greater)
• jq
• kafkacat (only for training and debugging)
• elasticdump (only for training and debugging)
• IBM cloudctl (only for LDAP)

Only needed if you decide to install from your PC

Just run:

./10_install_prerequisites_mac.sh

Only needed if you decide to install from your PC

Just run:

./11_install_prerequisites_ubuntu.sh

You can get the installation (registry pull token) token from https://myibm.ibm.com/products-services/containerlibrary.

This allows the CP4WAIOPS images to be pulled from the IBM Container Registry.

You have different options:

1. Install directly from the OCP Web UI (no need to install anything on your PC)

   1. In the the OCP Web UI click on the + sign in the right upper corner
   2. Copy and paste the content from this file
   3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1 (the Entitlement key from https://myibm.ibm.com)
   4. Click Save

2. Install from your PC with the token from 1.3.1

   ansible-playbook ./ansible/01_cp4waiops-aimanager-all.yaml -e CP_ENTITLEMENT_KEY=<REGISTRY_TOKEN> 

3. Install with the Easy Installer with the token from 1.3.1

   1. Just run:

      ./01_easy-install.sh -t <REGISTRY_TOKEN>

   2. Select option 🐥01 to install the complete AI Manager environment with Demo Content.

This takes about 1.5 to 2 hours. After completion Easy Installer will open the documentation and the AI Manager webpage (on Mac) and you'll have to to perform the last manual steps.

You now have a full, basic installtion of AI Manager with:

• AI Manager
• Open LDAP & Register with AI Manager
• RobotShop demo application
• Trained Models based on pre-canned data (Log- and Metric Anomalies, Similar Incidents, Change Risk)
• Topologies for demo scenarios
• AWX (OpenSource Ansible Tower) with runbooks for the demo scenarios
• Demo UI
• Demo Service Account
• Valid certificate for Ingress (Slack)
• External Routes (Flink, Topology, ...)
• Policies for Stories and Runbooks

Those are the manual configurations you'll need to demo the system and that are covered by the flow above.

Configure Topology

1. Re-Run Kubernetes Observer

Configure Slack

1. Setup Slack

After successful installation, the Playbook creates a file ./LOGINS.txt in your installation directory (only if you installed from your PC).

ℹ️ You can also run ./tools/20_get_logins.sh at any moment. This will print out all the relevant passwords and credentials.

• Run ./tools/20_get_logins.sh to get all the logins and URLs

or

• Run:

export WAIOPS_NAMESPACE=$(oc get po -A|grep aiops-orchestrator-controller |awk '{print$1}')
	
echo "🌏 AI Manager:           https://$(oc get route -n $WAIOPS_NAMESPACE cpd -o jsonpath={.spec.host})"
echo "🌏 Demo UI:              https://$(oc get route -n $WAIOPS_NAMESPACE waiops-demo-ui-python -o jsonpath={.spec.host})"

• Open the URL from the above
• Click on Enterprise LDAP
• Login as demo with the password P4ssw0rd!

• Open the URL from the above

• Click on IBM provided credentials (admin only)

  

• Login as admin with the password from the LOGINS.txt file

  

In the AI Manager (CP4WAIOPS)

1. In the AI Manager "Hamburger" Menu select Define/Data and tool integrations
2. Click Kubernetes
3. Under robot-shop, click on Run (with the small play button)

Continue here for Slack integration

You have different options:

1. Install directly from the OCP Web UI (no need to install anything on your PC)

   1. In the the OCP Web UI click on the + sign in the right upper corner
   2. Copy and paste the content from this file
   3. Replace <REGISTRY_TOKEN> at the end of the file with your pull token from step 1.3.1
   4. Click Save

2. Install from your PC with the token from 1.3.1

   ansible-playbook ./ansible/04_cp4waiops-eventmanager-all.yaml -e CP_ENTITLEMENT_KEY=<REGISTRY_TOKEN> 

3. Install with the Easy Installer with the token from 1.3.1

   1. Just run:

      ./01_easy-install.sh -t <REGISTRY_TOKEN>

   2. Select option 🐥02 to install the complete Event Manager environment with Demo Content.

This takes about 1 hour.

After successful installation, the Playbook creates a file ./LOGINS.txt in your installation directory (only if you installed from your PC).

ℹ️ You can also run ./tools/20_get_logins.sh at any moment. This will print out all the relevant passwords and credentials.

• Open the LOGINS.txt file that has been created by the Installer in your root directory
• Open the URL from the LOGINS.txt file
• Login as smadmin with the password from the LOGINS.txt file

• To get the connection parameters, run:

./tools/97_addons/prepareNetcoolIntegration.sh

Execute the listed commads at the Objectserver prompt.

This gives you all the parameters needed for creating the connection.

• In the AI Manager "Hamburger" Menu select Operate/Data and tool integrations
• Click Add connection
• Under Netcool, click on Add connection
• Click Connect
• Name it Netcool
• Fill-in the information from the script above
• Click Test Connection
• Click Next
• Toggle Enable Data Collection to the ON position
• Click Save

Create Webhooks in EventManager for Event injection and incident simulation for the Demo.

The demo scripts (in the demo folder) give you the possibility to simulate an outage without relying on the integrations with other systems.

At this time it simulates:

• Git push event
• Log Events (ELK)
• Security Events (Falco)
• Instana Events
• Metric Manager Events (Predictive)
• Turbonomic Events
• CP4MCM Synthetic Selenium Test Events

You have to define the following Webhook in EventManager (NOI):

• Administration / Integration with other Systems
• Incoming / New Integration
• Webhook
• Name it Demo Generic
• Jot down the WebHook URL and copy it to the NETCOOL_WEBHOOK_GENERIC in the ./tools/01_demo/incident_robotshop-noi.shfile
• Click on Optional event attributes
• Scroll down and click on the + sign for URL
• Click Confirm Selections

Use this json:

{
  "timestamp": "1619706828000",
  "severity": "Critical",
  "summary": "Test Event",
  "nodename": "productpage-v1",
  "alertgroup": "robotshop",
  "url": "https://pirsoscom.github.io/grafana-robotshop.html"
}

Fill out the following fields and save:

• Severity: severity
• Summary: summary
• Resource name: nodename
• Event type: alertgroup
• Url: url
• Description: "URL"

Optionnally you can also add Expiry Time from Optional event attributes and set it to a convenient number of seconds (just make sure that you have time to run the demo before they expire.

• In the Event Manager "Hamburger" Menu select Netcool WebGui
• Click Administration
• Click Filters
• Select Global Filters from the DropDown menu
• Select Default
• Click Copy Filter (the two papers on the top left)
• Set to global
• Click Ok
• Name: AIOPS
• Logic: Any ❗ (the right hand option)
• Filter:
  • AlertGroup = 'CEACorrelationKeyParent'
  • AlertGroup = 'robot-shop'

• In the Event Manager "Hamburger" Menu select Netcool WebGui
• Click Administration
• Click Views
• Select System Views from the DropDown menu
• Select Example_IBM_CloudAnalytics
• Click Copy View (the two papers on the top left)
• Set to global
• Click Ok
• Name: AIOPS
• Configure to your likings.

• In the Event Manager "Hamburger" Menu select Netcool WebGui
• Click Insights
• Click Scope Based Grouping
• Click Create Policy
• Action select fielt Alert Group
• Toggle Enabled to On
• Save

In the Netcool WebGUI

• Go to Administration / Tool Configuration

• Click on LaunchRunbook

• Copy it (the middle button with the two sheets)

• Name it Launch URL

• Replace the Script Command with the following code

   var urlId = '{$selected_rows.URL}';
   
   if (urlId == '') {
       alert('This event is not linked to an URL');
   } else {
       var wnd = window.open(urlId, '_blank');
   }

• Save

Then

• Go to Administration / Menu Configuration
• Select alerts
• Click on Modify
• Move Launch URL to the right column
• Save

For the system to work you need to follow those steps:

1. Create Slack Workspace
2. Create Slack App
3. Create Slack Channels
4. Create Slack Integration
5. Get the Integration URL
6. Create Slack App Communications
7. Slack Reset

1. Create a Slack workspace by going to https://slack.com/get-started#/createnew and logging in with an email which is not your IBM email. Your IBM email is part of the IBM Slack enterprise account and you will not be able to create an independent Slack workspace outside if the IBM slack service.

2. After authentication, you will see the following screen:

3. Click Create a Workspace ->

4. Name your Slack workspace

Give your workspace a unique name such as aiops-<yourname>.

5. Describe the workspace current purpose

This is free text, you may simply write “demo for Watson AIOps” or whatever you like.

You may add team members to your new Slack workspace or skip this step.

At this point you have created your own Slack workspace where you are the administrator and can perform all the necessary steps to integrate with CP4WAOps.

Note : This Slack workspace is outside the control of IBM and must be treated as a completely public environment. Do not place any confidential material in this Slack workspace.

1. Create a Slack app, by going to https://api.slack.com/apps and clicking Create New App.

   

2. Select From an app manifest

3. Select the appropriate workspace that you have created before and click Next

4. Copy and paste the content of this file ./doc/slack/slack-app-manifest.yaml.

   Don't bother with the URLs just yet, we will adapt them as needed.

5. Click Next

6. Click Create

7. Scroll down to Display Information and name your CP4WAIOPS app.

8. You can add an icon to the app (there are some sample icons in the ./tools/4_integrations/slack/icons folder.

9. Click save changes

10. In the Basic Information menu click on Install to Workspace then click Allow

1. In Slack add a two new channels:

   • aiops-demo-reactive
   • aiops-demo-proactive

   

2. Right click on each channel and select Copy Link

   This should get you something like this https://xxxx.slack.com/archives/C021QOY16BW The last part of the URL is the channel ID (i.e. C021QOY16BW) Jot them down for both channels

3. Under Apps click Browse Apps

   

4. Select the App you just have created

5. Invite the Application to each of the two channels by typing

   @<MyAppname>

6. Select Add to channel

   You shoud get a message from saying was added to #<your-channel> by ...

In the Slack App:

1. In the Basic Information menu get the Signing Secret (not the Client Secret!) and jot it down

   

2. In the OAuth & Permissions get the Bot User OAuth Token (not the User OAuth Token!) and jot it down

   

In the AI Manager (CP4WAIOPS)

1. In the AI Manager "Hamburger" Menu select Define/Data and tool integrations

2. Click Add connection

   

3. Under Slack, click on Add Connection

4. Name it "Slack"

5. Paste the Signing Secret from above

6. Paste the Bot User OAuth Token from above

   

7. Paste the channel IDs from the channel creation step in the respective fields

   

   

8. Test the connection and click save

In the AI Manager (CP4WAIOPS)

1. Go to Data and tool integrations

2. Under Slack click on 1 integration

3. Copy out the URL

   

This is the URL you will be using for step 6.

Return to the browser tab for the Slack app.

1. Select Event Subscriptions.

2. In the Enable Events section, click the slider to enable events.

3. For the Request URL field use the Request URL from step 5.

   e.g: https://<my-url>/aiops/aimanager/instances/xxxxx/api/slack/events

4. After pasting the value in the field, a Verified message should display.

   

   If you get an error please check 5.7

5. Verify that on the Subscribe to bot events section you got:

   • app_mention and
   • member_joined_channel events.

   

6. Click Save Changes button.

7. Select Interactivity & Shortcuts.

8. In the Interactivity section, click the slider to enable interactivity. For the Request URL field, use use the URL from above.

There is no automatic verification for this form

9. Click Save Changes button.

Now, configure the welcome slash command. With this command, you can trigger the welcome message again if you closed it.

1. Select Slash Commands

2. Click Create New Command to create a new slash command.

   Use the following values:

   Field	Value
   Command	/welcome
   Request URL	the URL from above
   Short Description	Welcome to Watson AIOps

3. Click Save.

The Slack app must be reinstalled, as several permissions have changed.

1. Select Install App
2. Click Reinstall to Workspace

Once the workspace request is approved, the Slack integration is complete.

If you run into problems validating the Event Subscription in the Slack Application, see 5.2

This is needed for the reset scripts in order to empty/reset the Slack channels.

This is based on Slack Cleaner2. You might have to install this:

pip3 install slack-cleaner2

In your Slack app

1. In the OAuth & Permissions get the User OAuth Token (not the Bot User OAuth Token this time!) and jot it down

In file ./tools/98_reset/13_reset-slack.sh

2. Replace not_configured for the SLACK_TOKEN parameter with the token
3. Adapt the channel name for the SLACK_REACTIVE parameter

In your Slack app

1. In the OAuth & Permissions get the User OAuth Token (not the Bot User OAuth Token this time!) and jot it down (same token as above)

In file ./tools/98_reset/14_reset-slack-changerisk.sh

2. Replace not_configured for the SLACK_TOKEN parameter with the token
3. Adapt the channel name for the SLACK_PROACTIVE parameter

Call either of the scripts above to reset the channel:

./tools/98_reset/13_reset-slack.sh

or

./tools/98_reset/14_reset-slack-changerisk.sh

• Run:

   export WAIOPS_NAMESPACE=$(oc get po -A|grep aiops-orchestrator-controller |awk '{print$1}')
   	
   echo "🌏 AI Manager:           https://$(oc get route -n $WAIOPS_NAMESPACE cpd -o jsonpath={.spec.host})"
   echo "🌏 Demo UI:              https://$(oc get route -n $WAIOPS_NAMESPACE waiops-demo-ui-python -o jsonpath={.spec.host})"
   

  

• Open the Demo UI URL from the above
• Login with the password P4ssw0rd!

Click on the red Create Incident Memory Leak button

This will create alerts and a story in AI Manager.

• Open the AI Manager URL from the above
• Click on Enterprise LDAP
• Login as demo with the password P4ssw0rd!

ℹ️ Give it a minute or two for all events and anomalies to arrive in AI Manager and Slack.

Make sure you are logged-in to the Kubernetes Cluster first

In the terminal type

./22_simulate_incident_robotshop.sh

This will delete all existing Alerts/Stories and inject pre-canned event, metrics and logs to create a story.

ℹ️ Give it a minute or two for all events and anomalies to arrive in AI Manager and Slack. ℹ️ You might have to run the script 3-4 times for the log anomalies to start appearing.

• Info
  • Changelog
  • Demo Architecture
  • Detailed Prerequisites
  • Troubleshooting
• Installation
  • Event Manager Install
  • Event Manager Configuration
  • Uninstall CP4WAIOPS
• Configuration
  • Manual Runbook Configuration
  • Additional Configuration
  • Service Now integration
• Install additional components
  • Installing Turbonomic
  • Installing ELK