- changed background colour of GUI to white, its more pleasing to the eyes.
- implemented fix for bug when user runs manual mode without specifying cluster amount on kmeans. It crashes 1/5 times but works otherwise. I assume this is due to the problem of time stepping and can probably be fixed by sleeping threads at a certain point, or by the gap statistic returning very large cluster numbers.
- fixed manual mode in general, works as intended now.
- improved general efficiency
BUG: when using automatic mode and gap statistic is selected by the algorithm, the centroids will not be displayed correctly on the basic clusterings although the combined clustering will function as intended.FIXED
Using the university's hacklab nmap and nessus scans as a dataset the following results were acheived. More information can be found within the hacklab analyses folder.
twin mode: requires both datasets as input. This will take clusters with IP count < 3 and combine the vector features of both datasets to create a new clustering of 'vulnerable' IP's. In other words - Small clusterings (IP count less than 3) reclustered with combined datasets.
command:
cluster.py -s automatic -vv -p -t -N -tp "../hacklab analyses/hacklab_new.xml" "../hacklab analyses/hacklab_new.nessus"
full raw output which includes all relevant information can be found here: twin-auto-output.txt
clustering model on automatic mode with centroids using nmap xml output (left) and nessus scan output (right) the vectors have been normalised before PCA(2d).
the covariance matrices for each are as follows:
centroids covariance matrix for nmap output:
[[ 0.23001336 0.02479788]
[ 0.02479788 0.14788991]]
centroids covariance matrix for nessus output:
[[ 0.13216597 0.00459265]
[ 0.00459265 0.08221746]]
usage: cluster.py [-h] [-s {manual,automatic,assisted}]
[-c {kmeans,dbscan,agglomerative}]
[--metric {euclidean,cosine,jaccard}] [-N] [-n N_CLUSTERS]
[-e EPSILON] [-m MIN_SAMPLES] [-cent] [-t] [-tp twinpath]
[-p] [-v]
path [path ...]
Cluster NMap/Nessus Output
positional arguments:
path Paths to files or directories to scan
optional arguments:
-h, --help show this help message and exit
-s {manual,automatic,assisted}, --strategy {manual,automatic,assisted}
-c {kmeans,dbscan,agglomerative}, --method {kmeans,dbscan,agglomerative}
--metric {euclidean,cosine,jaccard}
-N, --nessus use .nessus file input
-n N_CLUSTERS, --n_clusters N_CLUSTERS
Number of kmeans clusters to aim for
-e EPSILON, --epsilon EPSILON
DBSCAN Epsilon
-m MIN_SAMPLES, --min_samples MIN_SAMPLES
DBSCAN Minimum Samples
-cent, --centroids plot only centroids graph, requires the use of "-p"
-t, --twin use both input formats to calculate vulnerable single
clusters, use with -tp and -N
-tp twinpath, --twinpath twinpath
path to nmap xml if using twin clustering
-p, --plot Plot clusters on 2D plane
-v, --verbosity increase output verbosity
Items in bold are currently being worked on.
Items ruled are completed.
Network capturerFeature extration and clusteringConsider labelsCreate a nessus xml file parsercode is messy but it works and is easily editable to incorporate any aspect of nessus file as featuresNessus clusteronly usable nessus file i have at the moment is my home network which is small and not very clusterable however it seems to work relatively well.- add multiple multiclass ROC curves for each clustering in main figure interface.
Using a nessus file from my home network scan.
192.168.0.4 and 192.168.0.7 are both recent windows machines whilst 192.168.0.1 is the home router and 192.168.0.8 is an android device.
calculate covariance matrix of centroidsonly works for kmeans due to agglo and dbscan python implementations not returning centroidsCompare the two clusters and generate a new cluster based off of themtakes all ips in clusters less than 3 ip's in size and combines each ips data from the 2 data sets and clusters based off of gap_statistic k-means
- automate the entire process not just clustering (scanning and retreival)
- possible integration with exploitation model
Click the following for clustering information without normalisation before PCA (old):
individual readme.md's for each model: