anssi-fr / secuml Goto Github PK

Machine Learning for Computer Security

Home Page: https://anssi-fr.github.io/SecuML

License: GNU General Public License v2.0

Python 77.27% JavaScript 20.36% HTML 2.16% CSS 0.12% Shell 0.10%

machine-learning intrusion-detection interactive-machine-learning active-learning rare-category-detection malware-detection gui

secuml's Introduction

SecuML https://anssi-fr.github.io/SecuML/

SecuML is a Python tool that aims to foster the use of Machine Learning in Computer Security. It is distributed under the GPL2+ license.

It allows security experts to train detection models easily and comes with a web user interface to visualize the results and interact with the models. SecuML can be applied to any detection problem. It requires as input numerical features representing each instance. It supports binary labels (malicious vs. benign) and categorical labels which represent families of malicious or benign behaviours.

Benefits of SecuML

SecuML relies on scikit-learn to train the Machine Learning models and offers the additionnal features:

Web user interface
diagnosis and interaction with Machine Learning models (active learning, rare category detection)
Hide some of the Machine Learning machinery
automation of data loading, feature standardization, and search of the best hyperparameters

What you can do with SecuML

Training and diagnosing a detection model before deployment with DIADEM
Annotating a dataset with a reduced workload with ILAB
Exploring a dataset interactively with rare category detection
Clustering
Projection
Computing descriptive statistics of each feature

See the sphinx documentation for more detail.

Papers

Beaugnon, Anaël, and Pierre Chifflier. "Machine Learning for Computer Security Detection Systems: Practical Feedback and Solutions" Computer & Electronics Security Applications Rendez-vous (C&ESAR 2018)
Beaugnon, Anaël, Pierre Chifflier, and Francis Bach. "End-to-End Active Learning for Computer Security Experts."
KDD Workshop on Interactive Data Exploration and Analytics (IDEA 2018). Extended version of AICS 2018.
Beaugnon, Anaël, Pierre Chifflier, and Francis Bach. "End-to-End Active Learning for Computer Security Experts."
AAAI Workshop on Artificial Intelligence for Computer Security (AICS 2018).
Beaugnon, Anaël, Pierre Chifflier, and Francis Bach. "ILAB: An Interactive Labelling Strategy for Intrusion Detection."
International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2017).
[FRENCH] Bonneton, Anaël, and Antoine Husson. "Le Machine Learning confronté aux contraintes opérationnelles des systèmes de détection."
Symposium sur la sécurité des technologies de l'information et des communications (SSTIC 2017).

PhD Dissertation

Beaugnon, Anaël. "Expert-in-the-Loop Supervised Learning for Computer Security Detection Systems."
Ph.D. thesis, École Normale Superieure (2018)

Presentations

[FRENCH] Beaugnon, Anaël. "Appliquer le Machine Learning de manière pertinente à la détection d’intrusion."
Forum annuel du CERT-IST (CERT-IST 2017).
Bonneton, Anaël. "Machine Learning for Computer Security Experts using Python & scikit-learn."
PyParis 2017.

Authors

Anaël Beaugnon ([email protected])
Pierre Collet ([email protected])
Antoine Husson ([email protected])

secuml's People

Contributors

Stargazers

Watchers

Forkers

ah-anssi hubertloiseau cacalote piuliss xiaoerlaigeid clustersdata william-vu t2kien dip16k ruizheliuoa locussam mayri firebitsbr citlalingalvan nguyenhoaihuy wtbsw oksbsb rcoscali pierre-collet-anssi salamsajid y0un35 guillain ab-anssi kkneomis ryanbekabe tuantmb biyanisuraj flavio58it will3b theedoms manjubsavanth suleisl2000 rishi-wqd190004 olivierh59500 modulexcite zzszmyf cmnatic bcp-infosec-repo brenicolas qiyeboy incende amarjitghuman baekdavid ass77 hadidoun aoe-khkhan ouzdeville dwtcourses ismailqau oluagunloye rajeevyadav hecg119 sweetpastabox agranger13 ctc-kernel habibmrad lsotou kareypyer gjohnrao harel-coffee techthiyanes darksidesfear

secuml's Issues

Installation errors with python3.6

The installation of the dependency libraries (via pip) fails with python3.6.
For now, you should use python3.5.

Delete experiments that have not completed successfully

Currently the failed experiments are not deleted. So, they can be accessed from the SecuML web interface, but their display fails.
They should be deleted to avoid these display errors.

"Getting started"

Hi guys, I really like your project,
could you prepare a "getting started" to quick testing?
thank you so much!
nice tool!

Installation issues

Hello
I'm following the installation guide, so setup with

$ python3 -m venv venv
$ source venv_SecuML/bin/activate
(venv)$ python setup.py install
// ...installed...

and set conf file as:

input_data_dir: 'input_data'
output_data_dir: 'output_data'
db_uri: 'mysql+mysqldb://root@localhost/secuml_test'

and define path with:

$ export SECUMLCONF=conf/SecuML_travis_conf.yml

But when I try with:

(venv)$ python SecuML/web/SecuML_server
 * Serving Flask app "SecuML.web" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 226-019-981
127.0.0.1 - - [24/Jul/2018 15:16:12] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [24/Jul/2018 15:16:12] "GET /favicon.ico HTTP/1.1" 404 -

it shows like above, but shows nothing when I access to localhost:5000 on browser.
Is there something more I have to do?

ModuleNotFoundError when launching the web server

Hello,

I tried installing the module in a virtualenv, as mentionned in the docs. Everything works fine until I launch the web server with the following command :
SecuML_server --secuml-conf conf.yml
My conf.yml file looks like this :

input_data_dir: '/var/dataset.csv'
output_data_dir: '/var/results/'
db_uri: 'postgresql://user:password@localhost/secuml'

The error I get is the following :

Traceback (most recent call last):
  File "/var/SecuML/venv/bin/SecuML_server", line 4, in <module>
    __import__('pkg_resources').run_script('SecuML==0.12', 'SecuML_server')
  File "/var/SecuML/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/var/SecuML/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1438, in run_script
    exec(code, namespace, namespace)
  File "/var/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/EGG-INFO/scripts/SecuML_server", line 53, in <module>
    import secuml.web.views
  File "/var/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/secuml/web/views/__init__.py", line 18, in <module>
    from .active_learning import active_learning  # NOQA
  File "/var/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/secuml/web/views/active_learning/active_learning.py", line 30, in <module>
    from secuml.exp.active_learning import ActiveLearningExp  # NOQA
  File "/var/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/secuml/exp/active_learning/__init__.py", line 30, in <module>
    from .iteration import Iteration
  File "/var/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/secuml/exp/active_learning/iteration.py", line 26, in <module>
    from .monitoring.exec_times import ExecutionTimesMonitoring
ModuleNotFoundError: No module named 'secuml.exp.active_learning.monitoring'

Thank you

Simple confusing typo on doc

Hello,

There is a small confusing typo in the documentation: in getting_started.lingspam.rst, in the Annotating a Dataset with a Reduced Workload section, the example command you show is :
SecuML_ILAB SpamHam lingspam --secuml-conf <conf_file >Ilab --auto --budget 500

I think it would be clearer if the right bracket of conf_file is not preceded by a space and not immediately followed by Ilab, like so:
SecuML_ILAB SpamHam lingspam --secuml-conf <conf_file> Ilab --auto --budget 500

Thanks,
Pavel

First experiment

Hello,
Thank you for your amazing work. I finished installing SecuML, yet I don't know how to start experimenting with it. I am required to try some experiments for a research project but I don't know how to start. What should be my first step to do after installing it?
Thanks again, and sorry if my question is quite silly.