Upgrade plan for entire mozdef stack for:

• nginx
• uswgi
• python
• bottle
• pyes
• ES
• mongo
• rabbitmq
• etc.

Add support for JunOS log handling including details fields for junos-specific items.

The drop zone for veris tags is buggy. It's hard be drop.
Also, tags are not saved when we click to the save button (or without clicking on it).

Process only described for yum based systems.
apt-get command should be:
sudo apt-get install make zlib1g-dev libbz2-dev libssl-dev libncurses5-dev libsqlite3-dev libreadline-dev tk-dev libpcre3-dev libpcre++-dev build-essential g++

Command for unpacking python is incorrect:
Is currently tar "xvzf http://python.org/ftp/python/2.7.6/Python-2.7.6.tgz" but should be
"tar xvzf Python-2.7.6.tgz"

File paths inconcistent throughout description: Configured as: " ./configure --prefix=/home/mozdef/python2.7 --enable-shared"
But LD_LIBRARYPATH set to: "export LD_LIBRARY_PATH=/home/netantho/python2.7/lib/"
And later start virtualenv is started by "~/python2.7/bin/virtualenv mozdef" (which would only be true if you are user mozdef in this case)

• Could have setup an ldap->mongo cron job to count password changes left/% left and done a D3 visualization

Need to be a easy python framework or webui configuration wizards, lot of snippets

Be able to reference external web documents used for an incident:

• google docs (writer, spreadsheet, slides)
• etherpad
• dropbox
• pastebin
• kibana
• bugzilla
• other links

The current docker config is tested against docker 0.9 and not the new 0.10. It would be nice to test. Boot2docker is already updated for 0.10 (docker-compatible distro).

Use the esworker plugin system to pull out ipv4 and ipv6 from events into distinct fields

MozDef bot requires KitnIRC, but the dependency is not described or included in the install process.

Seems like MozDef boz libraries should be installed along with the other python libraries?

Use the esworker plugin system to add some info in the event if the attacker is in a watch list (managed by mongodb for storage)

On the webUI, we need to be able to list the content of the watch list and manage it (add, edit, remove attackers)

We want to have Health/status of the whole stack.

Examples:

• node cpu/jvm stats
• load average
• disk free

We should also be able to configure links to external monitoring webpages (Kibana health dashboard and marvel for instance).

Tossing in ipv6 data into a field templated as 'ip' results in this error:

Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: failed to parse ip [2a02:d28:666::69], not full ip address (4 dots)

Related work on elastic search and lucene:
elastic/elasticsearch#5758
https://issues.apache.org/jira/browse/LUCENE-5596

Not clear whether elastic search will support both ip types in one field, or require differing fields. For now to support searching, mozdef should separate ipv4 from ipv6 data and store in differing fields.

Implement auditd alerts

Create screencast(s) to show and explain MozDef features.

Be careful not to disclose private information.

Write a release plan with:

• Issues required to be closed
• releasing process

Develop a mechanism to self-regulate the number of esworker.py instances running at one time compared to the incoming log volume

We should do unit testing on the project both for python and javascript.
It would be cool to integrate travisCI to run tests.

The license should be MPL2.0 with header and contributors in all source files.

Send alerts on MozDef errors (backup problems, ES problems, etc.).

Email and IRC are fine.

Create a Heka config to parse CEF logs.
I should be able to handle custom labels as well.

I can provide some sample logs if needed.

We should have a unique URL per alert so that we can share link to have more info about an alert.

• Add links to scripted Kibana alert dashboard from the alerts page
• GET /alerts/:alert_id should show the relevant alert

There should be instructions for apt based systems on how to install the different components.
(RabbitMQ, Meteor, Node, Nginx, UWSGI, Kibana)

I will comment or send a pull request as soon as i have time to test and document the proper steps.

We should be able to configure alerts in the webUI.

Plan

The goal is to make it super easy and super fast to create new alerts using whenever possible the web UI.
if not possible, it should be possible to create a very short python module.

Scheduling and task management architecture

• alert tasks (also do the correlation) are python modules describing Celery tasks (try have a modular/plugin system rather than single file scripts).
• celery project for celery beats with celerybeat-mongo scheduler
• mongodb
• meteor webUI

Alert module

Events in stored ES (use pyes) -> Alert module -> Alert sent to the alert exchange using kombu, add alerttimestamp to the events, update events index with alert for cross reference e['_source']['alerts'].append(dict(index=alertResult['_index'], type=alertResult['_type'], id=alertResult['_id']))

Types:

• generic modules: correlation/alerting logic that can be used for several alerts, once such a module is coded, alerts on different data can be created using the Web UI.
  • Xevents_in_Ytime: alert when more than X events in last Y mins/hours/days, optional unique field myfield (use method used for AMO for aggregation field)
  • topK: alert for top K events based on an unique field myfield (ES terms), optional n number that is the min number of hits
  • exact_match: alert whenever there's a doc matching the filters in the last Y mins
• custom modules: not generic, specific to an alert.

Attributes:

• Id (short name, without space)
• Name
• Type = custom|generic
• Instructions (provides also scripted Kibana dashboard if available)
• Specific fields (list)

Alerts properties:

• severity
• category
• utctimestamp
• eventtimestamp
• summary (should usually include how many docs matched)
• events
• tags

Template: TODO

WebUI

• New alert setup
  1. Select alert setup type (which alert task module to use, list served by the python REST API)
  2. Define parameters and read the alert type instructions
     • Name
     • Description
     • Enabled?
     • Frequency
     • How far back to look
     • Kibana dashboard to use for filters (if generic module), stored in mongo in Kibana dashboard format
     • Custom args fields (if generic module, served by the python REST API)
     • Summary of the alert (using mustache syntax)
  3. Save
  4. Enjoy
• Edit alert setup
  • Start ^ at step 2
• Remove alert setups
• List alert setups

Python REST API

• GET /alerts/setup/modules
• GET /alerts/setup/:alertmodule_id

Steps

1. Test Kibana filters for complex alerts we have to see if it's doable to use them
2. Have stupid (custom) Celery tasks created and scheduled with the mongodb data
3. Implement generic alert modules, try with by inject data in the mongodb data
4. Create the web interface, use mock of json for dynamic forms
5. Implement rest api part
6. testing and deployment

Limitations

• Jeff has started using facets for it but they don't return. It works for FxA but not for Auditd for example. As a result I'm not doing facets in the generic modules (I come with a custom python solution), we plan to try aggregations when they become available in pyes.

Ideas for later

• Alert preview: being able to manually launch an alert setup and see the result a few secs after without injecting the alert in the prod system

It would be nice to have the same kind of thing for data collection (cloudTrail, mig2mozdef) but it's low priority

Document how to write a plugin and activate it

Create a python script that would be standalone web server listening for logs the same way as loginput does it.
It would display debugging info of logs received and warning if they are not compliant to the JSON standard.

Display metrics what are present in all incidents. Can by displayed per incident or per time interval.

Use the esworker plugin system to set a lower ttl for events on matching a whitelist of known good events (no value in having them).
It should save some disk space.

Create a script to create sample events, correlations and alerts and send them into the system using loginput.
Because of ES 24h-based indices, we should generate data with date between now - 24h.

Launch the script in the docker configuration and update installation doc.

Use the esworker plugin system to associate events in a single event when it makes sense.

Examples:

• switch port
• bgp up/down

If you are working on this, please document in this bug the events you associate.

Use the esworker plugin system to add reference to last event (or last 10 or last day) for a user/server/endpoint

eshealth should by proxied the python rest api, just like kibana URLs. It allows to do the ES backend work in python and to have better security.

Have a realtime attacker dashboard using three.js.
It should display the top K attackers and separated them according to tags (bruteforcing, web attacker, etc.).

Currently the kibana urls are hard coded in the settings.js. Retrieve them instead from the rest api (querying elastic search) so they can be dynamically added as kibana's are added/removed, etc.

We should be able to configure which host to use for d3 figures.

-bash-4.1$ pwd
/home/netantho/MozDef-netantho/meteor/app

-bash-4.1$ grep "mozdef1.private.scl3.mozilla.com" ./ -R
./client/mozdef.js:    d3.json("https://mozdef1.private.scl3.mozilla.com:8444/ldapLogins/", function(error, jsondata) {
./client/mozdef.js:    d3.json("https://mozdef1.private.scl3.mozilla.com:8444/alerts/", function(error, jsondata) {

Use the esworker plugin system to pull out metadata from events into distinct fields:

• success/failure
• event categories (auth, etc.)
• etc.

If you are working on this please list in this bug which metadata you consider.

@trink found out that brointel and bronotice lua files are identical. They shouldn't be. I think it's a bad copy/paste on my side.

Should describe:

• Bug reports
  • Feature requests
• Documentation changes
• Code changes
  • Fork and pull requests
  • Building (link to the doc)
  • Testing

Inspiration: https://github.com/elasticsearch/elasticsearch/blob/master/CONTRIBUTING.md

It's always confusing for me to have this empty screen when I'm not logged in.
I usually think there's bug somewhere whereas it's because I need to authenticate with Persona.

Use the esworker plugin system to add some info in the event (higher severity for e.g.) if the server attacked is in a sensitive server list (managed by mongodb for storage)

On the webUI, we need to be able to list the content of the sensitive server list and manage it (add, edit, remove servers)

Be able to restore a snapshot or to backup one in one click in the web interface.
Bonus: display a progress bar

Theories panel with timestamped history of root cause guesses and evidence. Should also have authors associated to theories.

With docker 0.10 it has become harder to debug MozDef.
It would be nice to have a SSH service running and to document how to SSH in the container.

Should have a plugin system to integrate external tools.

Tools examples:

• banhammer
• banhammer-ng
• mig
• inventory

Example of usage:

• Add a banhammer-ng link with MozDef alerts
• Button to launch mig investigation

Create a documentation using Sphinx (in RST format) and build it automatically on readthedocs.

Similar to #72
Add support for snmptt (trap receiver tool) log handling including details fields for traps specific items.

Publish screenshot(s) of MozDef in action.

Be careful not to disclose private information.

On rabbitmq timeout, kombu/pyamqp goes crazy:

    Traceback (most recent call last):
    File "/home/mozdef/envs/mozdef/lib/python2.7/site-packages/bottle.py", line 764, in _handle
    return route.call(**args)
    File "/home/mozdef/envs/mozdef/lib/python2.7/site-packages/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
    File "/home/mozdef/envs/mozdef/loginput/index.py", line 94, in cefindex
    ensurePublish(cefDict,exchange=eventTaskExchange,routing_key=options.taskexchange)
    File "/home/mozdef/envs/mozdef/lib/python2.7/site-packages/kombu/connection.py", line 462, in _ensured
    new_channel = self.channel()
    File "/home/mozdef/envs/mozdef/lib/python2.7/site-packages/kombu/connection.py", line 246, in channel
    chan = self.transport.create_channel(self.connection)
    File "/home/mozdef/envs/mozdef/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 87, in create_channel
    return connection.channel()
    AttributeError: 'NoneType' object has no attribute 'channel'

Some people had similar issues with celery:

• https://bugzilla.mozilla.org/show_bug.cgi?id=663555
• https://bugzilla.mozilla.org/show_bug.cgi?id=648126

Mitigations panel with timestamped history of actions (permanent/temporary)

We should run jshint, pyflakes, pep8 and pylint to improve the code quality.

Add instruction on installation and usage of the cloudtrail integration.