faucetsdn / poseidon Goto Github PK

Hi @bradh41

I read your post here. Being a newbie to the world of Machine Learning based Cyber Security, I found it very informative. Thank you for that.

I found so many Jupyter Notebooks in the repo. Which one should I start with? Also how can I collect the raw hex dump of the headers (any tool which can help me) for my own custom use case?

Thank you

$uname -smprv
Darwin 14.5.0 Darwin Kernel Version 14.5.0: Tue Apr 11 16:12:42 PDT 2017; root:xnu-2782.50.9.2.3~1/RELEASE_X86_64 x86_64 i386
$docker -v
Docker version 17.03.1-ce, build c6d412e

$git clone git clone https://github.com/Lab41/poseidon.git && cd poseidon/ && make docs

(Edit: sorry about the formatting, MD is confused by the content)
.
.
.
`=================================== FAILURES ===================================
_____________________________ test_PoseidonStorage _____________________________

def test_PoseidonStorage():
    """
    test of PoseidonStorage class that
    brokers communication with the mongodb container.
    client.address tests connection to database and returns
    tuple of host and port. default port for mongodb
    is 27017.
    """
    ps = PoseidonStorage()

  assert isinstance(ps.client.address, type(()))

poseidon/poseidonStorage/test_poseidonStorage.py:77:

/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py:614: in address
return self._server_property('address')
/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py:580: in _server_property
writable_server_selector)
/usr/local/lib/python2.7/dist-packages/pymongo/topology.py:214: in select_server
address))

self = <pymongo.topology.Topology object at 0x7fe03c650890>
selector = <function writable_server_selector at 0x7fe044fb72a8>
server_selection_timeout = None, address = None

def select_servers(self,
                   selector,
                   server_selection_timeout=None,
                   address=None):
    """Return a list of Servers matching selector, or time out.

        :Parameters:
          - `selector`: function that takes a list of Servers and returns
            a subset of them.
          - `server_selection_timeout` (optional): maximum seconds to wait.
            If not provided, the default value common.SERVER_SELECTION_TIMEOUT
            is used.
          - `address`: optional server address to select.

        Calls self.open() if needed.

        Raises exc:`ServerSelectionTimeoutError` after
        `server_selection_timeout` if no matching servers are found.
        """
    if server_selection_timeout is None:
        server_timeout = self._settings.server_selection_timeout
    else:
        server_timeout = server_selection_timeout

    with self._lock:
        self._description.check_compatible()

        now = _time()
        end_time = now + server_timeout
        server_descriptions = self._description.apply_selector(
            selector, address)

        while not server_descriptions:
            # No suitable servers.
            if server_timeout == 0 or now > end_time:
                raise ServerSelectionTimeoutError(

                  self._error_message(selector))

E ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

/usr/local/lib/python2.7/dist-packages/pymongo/topology.py:189: ServerSelectionTimeoutError
____________________________ test_db_database_names ____________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c621c90>

def test_db_database_names(client):
    """
    tests retrieval of database names from
    poseidonStorage.
    """
    resp = client.get('/v1/storage')

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:106: AssertionError
___________________________ test_db_collection_names ___________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c5c9f10>

def test_db_collection_names(client):
    """
    tests retrieval of collection names for a
    given database.
    """
    resp = client.get('/v1/storage/local')

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:116: AssertionError
___________________________ test_db_collection_count ___________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c6502d0>

def test_db_collection_count(client):
    """
    tests retrieval of document count for
    a given database and collection.
    """
    resp = client.get('/v1/storage/local/startup_log')

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:131: AssertionError
_____________________________ test_db_retrieve_doc _____________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c6503d0>

def test_db_retrieve_doc(client):
    """
    tests retrieval of document from a
    given database and collection.
    If no document is found then error message
    is returned.
    """
    resp = client.get(
        '/v1/storage/doc/local/startup_log/ffffffffffffffffffffffff')

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:144: AssertionError
___________________________ test_db_collection_query ___________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c5c9110>

def test_db_collection_query(client):
    """
    tests response from query of database.
    response is a serialized dict with 'count'
    and 'docs' fields.
    """
    query = {}
    query = json.dumps(query)
    resp = client.get('/v1/storage/query/' + 'local/' + 'startup_log/' + query)

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:160: AssertionError
_____________________________ test_db_add_one_doc ______________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c650790>

def test_db_add_one_doc(client):
    """
    tests adding document to a database, then
    tests retrieving added document from the database
    using the returned id, then tests that the collection
    inserted into is listed under the database collections.
    """
    doc = {}
    doc['node_ip'] = 'TEST'
    doc['talked_to'] = {'machine_1': 2,
                        'machine_2': 1,
                        'machine_3': 1}
    doc['recieved_from'] = {'machine_1': 1,
                            'machine_6': 3,
                            'machine_2': 2}
    doc['packet_lengths'] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    doc['flow_id'] = '6a32984d2348e23894f3298'
    doc['dns_records'] = ['0.0.0.0', '1.1.1.1']
    doc['time_rec'] = {'first_sent': '0-0-0 00:00:00.000000',
                       'first_received': '0-0-0 00:00:00.000000',
                       'last_sent': '0-0-0 00:00:00.000000',
                       'last_received': '0-0-0 00:00:00.000000'}
    # adds doc to db
    uri = '/v1/storage/add_one_doc/' + 'test_poseidon_records/' + 'network_graph'
    resp = client.post(uri, data=json.dumps(doc))

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:196: AssertionError
____________________________ test_db_add_many_docs _____________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c524150>

def test_db_add_many_docs(client):
    """
    tests inserting several docs into database.
    encodes with bson for well-formatted url.
    """
    doc_one = {}
    doc_one['node_ip'] = '1.1.1.1'
    doc_one['packet_lengths'] = [1, 1, 2]

    doc_two = {}
    doc_two['node_ip'] = '2.2.2.2'
    doc_two['packet_lengths'] = [3, 5, 8]

    doc_thr = {}
    doc_thr['node_ip'] = '3.3.3.3'
    doc_thr['packet_lengths'] = [13, 21, 34]

    # insert 3 docs
    docs = [doc_one, doc_two, doc_thr]
    uri = '/v1/storage/add_many_docs/'+'test_poseidon_records/'+'network_graph/'
    resp = client.post(uri, data=json.dumps(docs))

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:263: AssertionError
____________________________ test_db_update_one_doc ____________________________

client = <pytest_falcon.plugin.Client object at 0x7fe03c650410>

def test_db_update_one_doc(client):
    """
    tests updating nonexistent doc,
    encodes filter and updated doc with
    bson.
    """
    filt = {'node_ip': '-1.-1.-1.-1'}
    update = {'node_ip': '-1.-1.-1.-1', 'new_field': 101}

    # non-existent document
    filt = json.dumps(filt)
    uri = '/v1/storage/update_one_doc/'+'test_poseidon_records/'+'network_graph/'+filt
    resp = client.post(uri, data=json.dumps(update))
    assert resp.status == falcon.HTTP_BAD_REQUEST
    resp = json.loads(resp.body)
    assert not resp['success']

    # insert doc
    doc = {'node_ip': 'GIRAFFE'}
    uri = '/v1/storage/add_one_doc/' + 'test_poseidon_records/' + 'network_graph'
    resp = client.post(uri, data=json.dumps(doc))

  assert resp.status == falcon.HTTP_OK

E AssertionError: assert '500 Internal Server Error' == '200 OK'
E - 500 Internal Server Error
E + 200 OK

poseidon/poseidonStorage/test_poseidonStorage.py:297: AssertionError
===================== 9 failed, 2 passed in 271.80 seconds =====================
The command '/bin/sh -c py.test -v --ignore=poseidon/poseidonMain --ignore=poseidon/poseidonMonitor --ignore=poseidon/periodically --ignore=poseidon/mockController --ignore=collectors --ignore=plugins/ --cov=poseidon/poseidonStorage --cov-report term-missing --cov-config .coveragerc' returned a non-zero code: 1
make: *** [build] Error 1`

NorthboundControllerAbstraction needs better coverage for testing.

add testing for get_endpoint_next to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for get_endpoint_ip to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for schedule_thread_worker to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for update_next_state to test_poseidonMonitor.py

-right now flow objects are passed to the callback - change to dict (or similar) of flows that uses correct flow based on rabbitmq message

NorthboundControllerAbstraction needs better coverage for testing.

add testing for signal_handler to test_poseidonMonitor.py

images should be organized under an img directory. README should filled in to include building the docs

NorthboundControllerAbstraction needs better coverage for testing.

add testing for make_endpoint_dict to test_NorthBoundControllerAbstraction.py

When I run the test, 2 errors as following,but the package was installed in /usr/local/lib/python2.7/dist-packages/pika，what can I do to resolve the errors:

ERROR collecting poseidon/poseidonMonitor/NorthBoundControllerAbstraction/test_NorthBoundControllerAbstraction.py
ImportError while importing test module '/poseidonWork/poseidon/poseidonMonitor/NorthBoundControllerAbstraction/test_NorthBoundControllerAbstraction.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
poseidon/poseidonMonitor/NorthBoundControllerAbstraction/test_NorthBoundControllerAbstraction.py:28: in
from poseidon.poseidonMonitor.NorthBoundControllerAbstraction.NorthBoundControllerAbstraction import controller_interface
poseidon/poseidonMonitor/NorthBoundControllerAbstraction/NorthBoundControllerAbstraction.py:30: in
import pika
E ImportError: No module named pika
___________________ ERROR collecting scripts/test_rabbit.py ____________________
ImportError while importing test module '/poseidonWork/scripts/test_rabbit.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
scripts/test_rabbit.py:12: in
import pika
E ImportError: No module named pika

• license
• maintainers
• contributing
• version
• authors
• changelog

NorthboundControllerAbstraction needs better coverage for testing.

add testing for get_q_item to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for get_endpoint_state to test_NorthBoundControllerAbstraction.py

thoughts?

NorthboundControllerAbstraction needs better coverage for testing.

add testing for make_known_endpoint to test_NorthBoundControllerAbstraction.py

Create an Endpoint.py in the same dir as poseidonMonitor.py
stores information like:

{
'tenant': u'FLOORPLATE', 
'mac': u'de:ad:be:ef:7f:12', 
'segment': u'prod', 
'name': None, 
'ip-address': u'102.179.20.100'
}

the class should have member variables
state(str) - current state the endpoint is in
next-state(str) - the next state for the endpoint
data(dict) - for storing the above example info

the class should have the following member functions
init by dictionary to make a new object .
make_hash (see NorthboundControllerAbstraction::make_hash)
to_str - make string representation of internals of object
to_json - should return a json view of the object
from_json -initialize object from json
update_state - state <- next_state, next_state <- 'NONE' or a string that is passed as a parameter.

NorthboundControllerAbstraction needs better coverage for testing.

add testing for get_rabbit_message to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for rabbit_callback to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for print_endpoint_state to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for find_new_machines to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for print_endpoint to test_poseidonMonitor.py

1. create a dockerfile called Dockerfile.docs (you may want to look at the pre-existing container)
2. Sphinx the project (see http://docs.readthedocs.io/en/latest/getting_started.html)
3. have output placed somewhere under docs

building the container should look like:
docker build -f ./Dockerfile.docs -t poseidon.docs .

running the container should look like:
docker run -v/someDir:/pathTo/docs poseidon.docs

someDir - would be where to write updated version of docs
/pathTo/docs - inside the container where things are being written

If you have questions please ask. (use the existing Dockerfile as a reference)

NorthboundControllerAbstraction needs better coverage for testing.

add testing for update_endpoint_state to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for configSelf to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for process to test_poseidonMonitor.py

under docs, the vent logo should be added into a folder called img and added to the main README.md page. the logo can be found here: https://blog.cyberreboot.org/hacktoberfest-is-upon-us-a8ff56a9d724

NorthboundControllerAbstraction needs better coverage for testing.

add testing for start_vent_collector to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for shutdown_endpoint to test_NorthBoundControllerAbstraction.py

Some of the functions within poseidonMonitor can be moved into a poseidonMonitor_helper.py file

1. create the poseidonMonitor_helper.py file in the same directory as poseidonMonitor.py
2. add the appropriate headers to the top of the file
3. move the helper functions into the new file
4. add the imports, call the functions correctly now that they are in a new location
5. put what functions were moved into the issue so I can update the Testing issues.

everything but two entries uses _ for separating works... make the config uniform.

rabbit-server = RABBIT_SERVER
rabbit-port = RABBIT_PORT

change to:

rabbit_server = RABBIT_SERVER
rabbit_port = RABBIT_PORT

fix code to look for the _ versions..

-pcap stats and dns verify both need to maintain list of in-network machines (from onos nab)
-integrate with rabbitmq and uncomment connection and callback handler installer

The machine learning process benefits from knowing some metadata about the data it is processing.

poseidonMonitor::start_vent_collector()
should add a 'metadata' field to the 'payload' dict.
'metadata' should be assigned the connection metadata associated with the dev_hash key passed into the function. get the metadata from uss.return_endpoint_state()

NorthboundControllerAbstraction needs better coverage for testing.

add testing for update_next to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for return_endpoint_state to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for start_vent_collector to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for miror_endpoint to test_NorthBoundControllerAbstraction.py

this is the collector that listens on a NIC and passes that traffic off to plugins

NorthboundControllerAbstraction needs better coverage for testing.

add testing for schedule_job_kickurl to test_poseidonMonitor.py

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:4444/swagger.yaml. (Reason: CORS header 'Access-Control-Allow-Origin' missi

NorthboundControllerAbstraction needs better coverage for testing.

add testing for schedule_job_reinvestigation to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for configSelf to test_poseidonMonitor.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for change_endpoint_state to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for first_run to test_NorthBoundControllerAbstraction.py

NorthboundControllerAbstraction needs better coverage for testing.

add testing for schedule_thread_worker to test_poseidonMonitor.py