Code Monkey home page Code Monkey logo

dissector's Introduction

        Concordia logo         DDoS Clearing House logo         NoMoreDDoS logo NoMoreDDoS logo

Python GitHub Issues Contributions welcome License Last commit

DDoS Dissector

The Dissector summarizes DDoS attack traffic from stored traffic captures (pcap/flows). The resulting summary is in the form of a DDoS Fingerprint; a JSON file in which the attack's characteristics are described.

How to use the Dissector

Option 1: in Docker

You can run DDoS Dissector in a docker container. This way, you do not have to install dependencies yourself and can start analyzing traffic captures right away. The only requirement is to have Docker installed and running.

  1. Pull the docker image from docker hub: docker pull ddosclearinghouse/dissector
  2. Run dissector in a docker container:
    docker run -i --network="host" \
    --mount type=bind,source=/abs-path/to/config.ini,target=/etc/config.ini \
    -v /abs-path/to/data:/data \
    ddosclearinghouse/dissector -f /data/capture_file [options]
    Note: We bind-mount the config file with DDoS-DB and MISP tokens to /etc/config.ini, and create a volume mount for the location of capture files. We use the local network to also allow connections to a locally running instance of DDoS-DB or MISP. Fingerprints are saved in your-data-volume/fingerprints

Option 2: Installed locally

  1. Install the dependencies to read PCAPs (tshark) and Flows (nfdump):

    1. https://tshark.dev/
    2. https://github.com/phaag/nfdump
  2. Clone the Dissector repository

    git clone https://github.com/ddos-clearing-house/ddos_dissector;
    cd ddos_dissector;
  3. [Advised] create a python virtual environment or conda environment for the dissector and install the python requirements:

    Venv:

    python -m venv ./python-venv
    source python-venv/bin/activate
    pip install -r requirements.txt

    Conda:

    conda create -n dissector python=3.10
    conda activate dissector
    pip install -r requirements.txt
  4. Get a traffic capture file to be analized (PCAP files should have the .pcap extension, Flows should have the .nfdump extension)

  5. Run the dissector:

    python src/main.py -f data/attack_traffic.nfdump --summary

Options

    ____  _                     __            
   / __ \(_)____________  _____/ /_____  _____
  / / / / / ___/ ___/ _ \/ ___/ __/ __ \/ ___/
 / /_/ / (__  |__  )  __/ /__/ /_/ /_/ / /    
/_____/_/____/____/\___/\___/\__/\____/_/     

usage: main.py [-h] -f FILES [FILES ...] [--summary] [--output OUTPUT] [--config CONFIG] [--nprocesses N] 
[--target TARGET] [--ddosdb] [--misp] [--noverify] [--debug] [--show-target]

options:
  -h, --help            show this help message and exit
  -f FILES [FILES ...], --file FILES [FILES ...]
                        Path to Flow / PCAP file(s)
  --summary             Optional: print fingerprint without source addresses
  --output OUTPUT       Path to directory in which to save the fingerprint (default ./fingerprints)
  --config CONFIG       Path to DDoS-DB and/or MISP config file (default /etc/config.ini)
  --nprocesses N        Number of processes used to concurrently read PCAPs (default is the number of CPU cores)
  --target TARGET       Optional: target IP address or subnet of this attack
  --ddosdb              Optional: directly upload fingerprint to DDoS-DB
  --misp                Optional: directly upload fingerprint to MISP
  --noverify            Optional: Don't verify TLS certificates
  --debug               Optional: show debug messages
  --show-target         Optional: Do NOT anonymize the target IP address / network in the fingerprint

Example: python src/main.py -f /data/part1.nfdump /data/part2.nfdump --summary --config ./localhost.ini --ddosdb --noverify

DDoS Fingerprint format

Example Fingerprints

Note: numbers and addresses are fabricated but are inspired by real fingerprints.

(Click to expand) Fingerprint from FLOW data: Multivector attack with LDAP amplification and TCP SYN flood
{
"attack_vectors": [
  {
    "service": "HTTPS",
    "protocol": "TCP",
    "source_port": 443,
    "fraction_of_attack": 0.21,
    "destination_ports": {
      "443": 1.0
    },
    "tcp_flags": {
      "......S.": 0.704,
      "others": 0.296
    },
    "nr_flows": 7946,
    "nr_packets": 39900000,
    "nr_megabytes": 34530,
    "time_start": "2022-01-30 12:49:09",
    "duration_seconds": 103,
    "source_ips": [
      "75.34.122.98",
      "80.83.200.214",
      "109.2.17.144",
      "22.56.34.108",
      "98.180.25.16",
      ...
    ]
  },
  {
    "service": "LDAP",
    "protocol": "UDP",
    "source_port": 389,
    "fraction_of_attack": 0.79,
    "destination_ports": {
      "8623": 0.837,
      "36844": 0.163
    },
    "tcp_flags": null,
    "nr_flows": 38775,
    "nr_packets": 31365000,
    "nr_megabytes": 101758,
    "time_start": "2022-01-30 12:49:01",
    "duration_seconds": 154,
    "source_ips": [
      "75.34.122.98",
      "80.83.200.214",
      "109.2.17.144",
      "22.56.34.108",
      "98.180.25.16",
      ...
    ]
  }
],
"target": "Anonymous",
"tags": [
  "Amplification attack",
  "Multi-vector attack",
  "TCP",
  "TCP flag attack",
  "UDP"
],
"key": "601fd86e43c004281210cb02d7f6d821",
"time_start": "2022-01-30 12:49:01",
"time_end": "2022-01-30 12:51:35",
"duration_seconds": 154,
"total_flows": 46721,
"total_megabytes": 102897,
"total_packets": 189744000,
"total_ips": 4397,
"avg_bps": 5193740008,
"avg_pps": 960028,
"avg_Bpp": 497
}
(Click to expand) Fingerprint from PCAP data: DNS amplification attack with fragmented packets
{
  "attack_vectors": [
    {
      "service": "Fragmented IP packets",
      "protocol": "UDP",
      "source_port": 0,
      "fraction_of_attack": null,
      "destination_ports": {
        "0": 1.0
      },
      "tcp_flags": null,
      "nr_packets": 4190,
      "nr_megabytes": 5,
      "time_start": "2013-08-15 01:32:40.901023+02:00",
      "duration_seconds": 0,
      "source_ips": [
        "75.34.122.98",
        "80.83.200.214",
        "109.2.17.144",
        "22.56.34.108",
        "98.180.25.16",
        ...
      ],
      "ethernet_type": {
        "IPv4": 1.0
      },
      "frame_len": {
        "1514": 0.684,
        "693": 0.173,
        "296": 0.057,
        "others": 0.086
      },
      "fragmentation_offset": {
        "0": 0.727,
        "1480": 0.247,
        "others": 0.026
      },
      "ttl": {
        "54": 0.159,
        "57": 0.142,
        "55": 0.123,
        "59": 0.119,
        "others": 0.457
      }
    },
    {
      "service": "DNS",
      "protocol": "UDP",
      "source_port": 53,
      "fraction_of_attack": 0.945,
      "destination_ports": "random",
      "tcp_flags": null,
      "nr_packets": 166750,
      "nr_megabytes": 21,
      "time_start": "2013-08-15 00:56:40.211654+02:00",
      "duration_seconds": 22,
      "source_ips": [
        "75.34.122.98",
        "80.83.200.214",
        "109.2.17.144",
        "22.56.34.108",
        "98.180.25.16",
        ...
      ],
      "ethernet_type": {
        "IPv4": 1.0
      },
      "frame_len": {
        "103": 0.695,
        "87": 0.208,
        "others": 0.097
      },
      "fragmentation_offset": {
        "0": 1.0
      },
      "ttl": {
        "120": 0.1,
        "119": 0.085,
        "121": 0.085,
        "118": 0.07,
        "others": 0.66
      },
      "dns_query_name": {
        "ddostheinter.net": 0.999
      },
      "dns_query_type": {
        "A": 0.999
      }
    }
  ],
  "target": "Anonymous",
  "tags": [
    "Fragmentation attack",
    "Amplification attack",
    "UDP"
  ],
  "key": "2e8c013d61ccaf88a1016828c16b9f0e",
  "time_start": "2013-08-15 00:56:40.211654+02:00",
  "time_end": "2013-08-15 00:57:03.199791+02:00",
  "duration_seconds": 22,
  "total_packets": 176393,
  "total_megabytes": 22,
  "total_ips": 8044,
  "avg_bps": 8039206,
  "avg_pps": 8017,
  "avg_Bpp": 125
}

Acknowledgment

EU Flag This project has received funding from the European Union's Horizon 2020
research and innovation program under grant agreement no. 830927.

dissector's People

Contributors

botlog avatar craig avatar dependabot[bot] avatar erev0s avatar jeroenh avatar jjsantanna avatar joaoceron avatar koenvh1 avatar poorting avatar santannajj avatar shahns avatar spirosmesa avatar swiftnode-linden avatar tvdhout avatar uyatashi avatar wqrld avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dissector's Issues

Provide config in login-style interaction

Currently, we require a config file with credentials for ddosdb to be passed each time running the dissector. It would be nicer if the user can provide those details once in a docker login-type fashion. An interactive prompt asking for ddosdb hostname and API key. The credentials would be saved locally.

If the dissector would be deployed as package, the login can be saved in the interpreter's share directory.

ip_proto as a string

Could we also have the ip_proto as a string in the json fingerprint file? Instead of or in addition to

"ip_proto":[ 17 ],
I'd need

"ip_proto_s":[ "UDP" ],

Thank you!

TCP flags should be parsed as hex values

For PCAP files, TCP flags are parsed from their numeric representation to the corresponding letters. However, the value obtained from tshark (tcp.flags) is casted as a decimal when it is an hexadecimal number.

For example, 0x00c2 will cause the program to crash although the correct result should be ....CE....S.. The error should be at this line: https://github.com/ddos-clearing-house/ddos_dissector/blob/main/src/attack.py#L88

I think the change could be as simple as changing it to:

int_flags = int(key, 16)

Alternatively, the string representation can be automatically calculated using tshark changing the parameter to be tcp.flags.str here: https://github.com/ddos-clearing-house/ddos_dissector/blob/main/src/reader.py#L35

I believe this is a bug, if you like one of the two proposed changes I can submit a PR.

Passing a negative integer is deprecated

FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.
pd.set_option('display.max_colwidth', -1)

exception not handled

INFO - Reprocessing attack based on protocols: ['ARP', 'BROWSER', 'CDP', 'DB-LSP-DISC', 'HTTP', 'ICMP', 'SSH', 'STP', 'TCP', 'UDP']
Traceback (most recent call last):
File "./ddos_dissector.py", line 1429, in
(fingerprint,json_file) = prepare_fingerprint_upload(df_fingerprint,df,fingerprint,n_type,labels)
File "./ddos_dissector.py", line 1266, in prepare_fingerprint_upload
initial_timestamp = datetime.utcfromtimestamp(initial_timestamp).strftime('%Y-%m-%d %H:%M:%S')
ValueError: Invalid value NaN (not a number)

software module named converter

There is no converters in the official repository. Although it is possible to find some repositories with some software (converter) built on the top of the database. The limitations here are:

  • No official converters from DDoSDB repository
  • The converters from external contributors are not integrated into the system
  • Lack of documentation

External repositories:

  1. BGP Flowspec Rules: https://github.com/joerikock/Master-Thesis-2019
  2. Berkeley Packet Filter and Express Data Path: https://github.com/HuubvanWieren/masterthesis
  3. Berkely Packet Filter: https://github.com/DirkKoelewijn/research-project
  4. BGP Flowspec Rules (2): https://github.com/DiedB/ResearchProject2019
  5. BRO https://github.com/boschma2702/DDoS_Scripts

Wrong analyzing tcpdump

Hi, i have a issue with analyzing own pcap recorded files (i recorded by the most simplest method), samples included in project works, the same problem it's in docker image.
root@node:~/ddos_dissector# ./ddos_dissector.py -f test.pcap --summary
/usr/lib/python3/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)


| __ | __ \ / | __ | _
| | | | | | | | ( | | | | |
) |
| | | | | | |/ _ \ ___ | | | | _ <
| || | || | (
) |
) | || | |_) |
|
/|/ _//|_/|____/

Configuration file provided [ddosdb.conf] not found
[▂] Loading network file: `test.pcap' Exception in thread process:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "./ddos_dissector.py", line 433, in pcap_to_df
df['ip.proto'] = df['ip.proto'].apply(lambda x: protocol_names[x] if (x > 0) else -1)
File "/usr/local/lib/python3.7/dist-packages/pandas/core/series.py", line 4138, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/_libs/lib.pyx", line 2467, in pandas._libs.lib.map_infer
File "./ddos_dissector.py", line 433, in
df['ip.proto'] = df['ip.proto'].apply(lambda x: protocol_names[x] if (x > 0) else -1)
KeyError: 112

pcap dump parsing issue - tokenizing data error

This is the same dump from #51 - unfortunately, it has more issues:

$ file BT-20220314.pcap 
BT-20220314.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65536)
[INFO] 
    ____  _                     __            
   / __ \(_)____________  _____/ /_____  _____
  / / / / / ___/ ___/ _ \/ ___/ __/ __ \/ ___/
 / /_/ / (__  |__  )  __/ /__/ /_/ /_/ / /    
/_____/_/____/____/\___/\___/\__/\____/_/     

[INFO] Loading "BT-20220314.pcap"...
[INFO] Error reading PCAP file: Error tokenizing data. C error: Expected 24 fields in line 145732, saw 25

[INFO] Skipping the offending lines...
Traceback (most recent call last):
  File "/home/sb/VCS/ddos_dissector/src/reader.py", line 125, in read_pcap
    data: pd.DataFrame = pd.read_csv(output_buffer, parse_dates=['frame.time'], low_memory=False, delimiter=',')
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 581, in _read
    return parser.read(nrows)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1250, in read
    index, columns, col_dict = self._engine.read(nrows)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 230, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 787, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 876, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 1960, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 24 fields in line 145732, saw 25


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sb/VCS/ddos_dissector/src/main.py", line 38, in <module>
    data: pd.DataFrame = pd.concat([read_file(f, filetype) for f in args.files])  # Read the FLOW file(s) into a dataframe
  File "/home/sb/VCS/ddos_dissector/src/main.py", line 38, in <listcomp>
    data: pd.DataFrame = pd.concat([read_file(f, filetype) for f in args.files])  # Read the FLOW file(s) into a dataframe
  File "/home/sb/VCS/ddos_dissector/src/reader.py", line 184, in read_file
    return read_pcap(filename)
  File "/home/sb/VCS/ddos_dissector/src/reader.py", line 129, in read_pcap
    data: pd.DataFrame = pd.read_csv(output_buffer, parse_dates=['frame.time'], low_memory=False, delimiter=',',
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 575, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 933, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1231, in _make_engine
    return mapping[engine](f, **self.options)
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 152, in __init__
    self._validate_parse_dates_presence(self.names)  # type: ignore[has-type]
  File "/home/sb/VCS/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/base_parser.py", line 228, in _validate_parse_dates_presence
    raise ValueError(
ValueError: Missing column provided to 'parse_dates': 'frame.time'

mix of absolute and relative paths for output directory

The method check_requirements checks whether the absolute path /output exists:

https://github.com/ddos-clearing-house/ddos_dissector/blob/b4385f0785af019441314c08768f2371bc4133f1/src/ddos_dissector_cli.py#L27

However, the directory output is created relative to the working directory if this condition is not met:

https://github.com/ddos-clearing-house/ddos_dissector/blob/b4385f0785af019441314c08768f2371bc4133f1/src/ddos_dissector_cli.py#L28

This bug makes it impossible to fulfill the check_requirements method (unless you run it from /). Using the same path in both lines would be a quick solution.

Filter obviously useless source IPs

Obviously spoofed IPs used in non-reflective attack vectors should be removed from the attack vector to avoid fingerprints with millions of source IP addresses in e.g. TCP SYN attacks.

Forking processes doesn't work quite well

When I run from the command line I get error messages about Forking. When I run everything manually from the interpreter, everything works fine.

Compare:

$ python ddos_dissector.py --input input4test/1.pcap
input file: /var/folders/jr/dzm6587x09s5v2d3mm4898b40000gp/T/tmp7rj671p0
output file: output/d0ccb10c7e08ac2d4a16804d53b30845.pcap
1375 packets (523467 bytes) written
objc[34625]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[34625]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
^CProcess ForkPoolWorker-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 335, in get
    res = self._reader.recv_bytes()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
  File "ddos_dissector.py", line 80, in <module>
    ddos_dissector(input_file)
  File "ddos_dissector.py", line 54, in ddos_dissector
    p.starmap(anonymize, items)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 638, in get
    self.wait(timeout)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 635, in wait
    self._event.wait(timeout)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 551, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
$ python
Python 3.6.5 (default, Apr 16 2018, 09:16:50)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import functions.attack_vector_anonymizer
>>> import functions.dataframe_analysis
>>> import functions.file_type_parser
>>> functions.file_type_parser.determine_file_type("input4test/1.pcap")
'pcap'
>>> df = functions.file_type_parser.convert_to_dataframe("input4test/1.pcap","pcap")
>>> functions.dataframe_analysis.analyze_dataframe(df,"pcap")

DISTRIBUTION OF DESTINATION IPS:
[...]

Make better use of argparse

Argparse is a really nice library for parsing arguments to the CLI tool.

I would suggest refactoring the code so that all settings currently in settings.example.py can be set by arguments, with sensible defaults. The idea is that you store these in a settings object, which you then pass along to other methods that possibly make use of it.

That will give you a much cleaner and flexible solution than the current file-based settings.

If necessary, you can extend the argparse later with a defaults file that we store in "~/.ddos_dissector.cfg" for example.

TypeError: expected str, bytes or os.PathLike object, not NoneType

I am unsure how to fix this error.

  ./ddos_dissector.py -f pcap_samples/sample1.pcap --summary,
 
 Loading network file: `pcap_samples/sample1.pcap' ▁CRITICAL - Tshark software not found
 Exception in thread process:
 Traceback (most recent call last):
 File ...
 ...

TypeError: expected str, bytes or os.PathLike object, not NoneType
 Loading network file: `pcap_samples/sample1.pcap' ▁ 

nfdump_modified

nfdump_modified is not installed automatically. Is this version of nfdump really needed? As far as I know this version is mainly used to anonymize the flow-file.

v3.01 uploads weird src_ips

version 3.01 seems to upload src_ips in this form:

"src_ips": {
"107.87.219.23": "107.87.219.23",
"121.237.58.117": "121.237.58.117",
"121.237.74.179": "121.237.74.179",
"111.227.142.123": "111.227.142.123",
"115.87.237.47": "115.87.237.47",
"109.229.134.219": "109.229.134.219",
(...)

which ddosdb does not understand.
Is there a reason for including src_ips in this way when the same information is also available in either amplifiers or attackers fields as a list?:

"amplifiers": [
"107.87.219.23",
"121.237.58.117",
"121.237.74.179",
"111.227.142.123",
(...)

If there is no immediate reason for including src_ips, I would suggest to simply leave it out :-)

(by the way: this is also the reason that with default settings elasticsearch produces an error of 'Limit of total fields [1000] in index [ddosdb] has been exceeded', it treats every IP address key as a new field).

ddos_dissector.py crashing with fragmented packets

I'm sending fragmented packets to my target system:

# hping3 -p 80 192.168.0.16 -E /etc/services -d 1 -f --flood

On my target system:

# tcpdump -ni bond0 -w ../cap.cap
^C
655466 packets captured
655466 packets received by filter
0 packets dropped by kernel

# ./ddos_dissector.py -f ../cap.cap --summary

 _____  _____        _____ _____  ____  
|  __ \|  __ \      / ____|  __ \|  _ \ 
| |  | | |  | | ___| (___ | |  | | |_) |
| |  | | |  | |/ _ \ ___ \| |  | |  _ < 
| |__| | |__| | (_) |___) | |__| | |_) |
|_____/|_____/ \___/_____/|_____/|____/ 

[✓] Using configuration file [ddosdb.conf]
[✓] Loading network file: `../cap.cap' 
[✓] Processing target IP address: 192.168.0.16
Traceback (most recent call last):
  File "./ddos_dissector.py", line 1589, in <module>
    (lst_attack_protocols, fragmentation_attack_flag) = infer_protocol_attack(df_target,n_type)
TypeError: cannot unpack non-iterable NoneType object

Debian 10 with:
Python 3.7.3
tshark 2.6.20-0+deb10u1
nfdump 1.6.17-1

pip3 installation:

# pip3 install -r requirements.txt
Collecting Pygments>=2.7.4 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/a6/c9/be11fce9810793676017f79ffab3c6cb18575844a6c7b8d4ed92f95de604/Pygments-2.9.0-py3-none-any.whl (1.0MB)
    100% |████████████████████████████████| 1.0MB 1.2MB/s 
Collecting cursor==1.3.4 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/b7/6a/c99e426288c8b0d229107a09137cf0518a84c9891af796802a5f888da0d0/cursor-1.3.4.tar.gz
Collecting ipaddr==2.2.0 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9d/a7/1b39a16cb90dfe491f57e1cab3103a15d4e8dd9a150872744f531b1106c1/ipaddr-2.2.0.tar.gz
Collecting numpy==1.17.0 (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/05/4b/55cfbfd3e5e85016eeef9f21c0ec809d978706a0d60b62cc28aeec8c792f/numpy-1.17.0-cp37-cp37m-manylinux1_x86_64.whl (20.3MB)
    100% |████████████████████████████████| 20.3MB 69kB/s 
Collecting pandas==1.0.3 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/4a/6a/94b219b8ea0f2d580169e85ed1edc0163743f55aaeca8a44c2e8fc1e344e/pandas-1.0.3-cp37-cp37m-manylinux1_x86_64.whl (10.0MB)
    100% |████████████████████████████████| 10.0MB 141kB/s 
Collecting requests==2.23.0 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 6.1MB/s 
Collecting pytz>=2017.2 (from pandas==1.0.3->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/70/94/784178ca5dd892a98f113cdd923372024dc04b8d40abe77ca76b5fb90ca6/pytz-2021.1-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 2.0MB/s 
Collecting python-dateutil>=2.6.1 (from pandas==1.0.3->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
    100% |████████████████████████████████| 235kB 3.9MB/s 
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.23.0->-r requirements.txt (line 6)) (2018.8.24)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests==2.23.0->-r requirements.txt (line 6)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.23.0->-r requirements.txt (line 6)) (2.6)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.23.0->-r requirements.txt (line 6)) (1.24.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.6.1->pandas==1.0.3->-r requirements.txt (line 5)) (1.12.0)
Building wheels for collected packages: cursor, ipaddr
  Running setup.py bdist_wheel for cursor ... done
  Stored in directory: /root/.cache/pip/wheels/bc/43/2f/1dcd2df9d75eb72d9eb8de232e8bd845a18cf9c18d15edc23d
  Running setup.py bdist_wheel for ipaddr ... done
  Stored in directory: /root/.cache/pip/wheels/c8/46/0b/5220490c49f714459a27e1eaafe3eef4c7a5e9bb4e9baf10ca
Successfully built cursor ipaddr
Installing collected packages: Pygments, cursor, ipaddr, numpy, pytz, python-dateutil, pandas, requests
  Found existing installation: requests 2.21.0
    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'requests'. No files were found to uninstall.
Successfully installed Pygments-2.9.0 cursor-1.3.4 ipaddr-2.2.0 numpy-1.17.0 pandas-1.0.3 python-dateutil-2.8.1 pytz-2021.1 requests-2.23.0

Bug Report: issue analyzing dumps

Hello, I've run into a problem when analysing a pcap dump:

$ python src/main.py -f BT-20220314.pcap --summary
[INFO] 
    ____  _                     __            
   / __ \(_)____________  _____/ /_____  _____
  / / / / / ___/ ___/ _ \/ ___/ __/ __ \/ ___/
 / /_/ / (__  |__  )  __/ /__/ /_/ /_/ / /    
/_____/_/____/____/\___/\___/\__/\____/_/     

[INFO] Loading "BT-20220314.pcap"...
Traceback (most recent call last):
  File "/home/user/ddos_dissector/src/main.py", line 38, in <module>
    data: pd.DataFrame = pd.concat([read_file(f, filetype) for f in args.files])  # Read the FLOW file(s) into a dataframe
  File "/home/user/ddos_dissector/src/main.py", line 38, in <listcomp>
    data: pd.DataFrame = pd.concat([read_file(f, filetype) for f in args.files])  # Read the FLOW file(s) into a dataframe
  File "/home/user/ddos_dissector/src/reader.py", line 178, in read_file
    return read_pcap(filename)
  File "/home/user/ddos_dissector/src/reader.py", line 124, in read_pcap
    data: pd.DataFrame = pd.read_csv(output_buffer, parse_dates=['frame.time'], low_memory=False)
  File "/home/user/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/user/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/user/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 581, in _read
    return parser.read(nrows)
  File "/home/user/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1250, in read
    index, columns, col_dict = self._engine.read(nrows)
  File "/home/user/ddos_dissector/python-venv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 230, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 787, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 876, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 1960, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 24 fields in line 145732, saw 25

I'm not really sure how to analyze this further. The dump loads fine in wireshark.

Run dissector as standalone script

Currently, to install the dissector, you'd need to clone the repository, install the requirements, and then run the main python script.

It would be neater if the project can be installed as a package as a whole. Either from pypi, or another package registry.

cannot process fingerprints using binary data

(df_remaining['_ws.col.Protocol']=='TCP')&(df_remaining['dstport']==80)&(df_remaining['tcp.flags.str']=='b'\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7A\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7'')
                                                                                                                                                                                                      ^

SyntaxError: unexpected character after line continuation character

500 error code is ignored

If uploading to a ddosdb, an 'internal server error' (50x) return code is ignored.
The submitter may want to know the ddosdb returned an error code; probably meaning the result wasn't stored successfully in the ddosdb.

duckdb.duckdb.BinderException with pcap input

With a fresh install of the dissector locally and trying to generate fingerprints from the sample pcaps previously found here, DuckDB raises the following error:

Traceback (most recent call last):
  File "/Users/thijs/Documents/DDoS/ddos_dissector/src/main.py", line 101, in <module>
    view = parquet_files_to_view(db, pqt_files, filetype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/thijs/Documents/DDoS/ddos_dissector/src/util.py", line 433, in parquet_files_to_view
    db.execute(sql)
duckdb.duckdb.BinderException: Binder Error: Cannot mix values of type VARCHAR and INTEGER_LITERAL in COALESCE operator - an explicit cast is required

Running on latest MacOS with these dependencies:

  • TShark (Wireshark) 4.2.2
  • duckdb==0.10.0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.