ebiquity / casie Goto Github PK

View Code? Open in Web Editor NEW

64.0 16.0 21.0 44.55 MB

CyberAttack Sensing and Information Extraction

Python 100.00%

cybersecurity-event information-extraction annotation-corpus

casie's Introduction

CASIE : CyberAttack Sensing and Information Extraction

Tool for extracting cyberattack event information

Components of CASIE are written in Python:

nug_arg_detection.py (applied Domain-Word2vec) or nug_arg_detection_bert.py (applied BERT) are for training the neural classifier to detect nugget and argument.
realis_identify.py is for training the neural classifier to identify realis for event nugget.
role_phrase.py is for training the neural classifier to assign roles to arguments.
link_coref.py is for identifying realis and linking role using in the trained models and grouping the mentions of the coreference events.
Others scripts are needed for running CASIE's components.

Annotation corpus of cybersecurity event in news articles

The corpus contains 1000 annotation and source files. Our cybersecurity focused on five event types: Databreach, Phishing, Ransom, Discover, and Patch.

More details of the annotation and CASIE's system are in the papers. If you use our data, please cite one of the following papers.

Taneeya Satyapanich, Francis Ferraro, and Tim Finin, "CASIE: Extracting Cybersecurity Event Information from Text", InProceedings, Proceeding of the 34th AAAI Conference on Artificial Intelligence, February 2020.
Taneeya Satyapanich, Tim Finin, and Francis Ferraro, "Extracting Rich Semantic Information about Cybersecurity Events", InProceedings, Second Workshop on Big Data for CyberSecurity, held in conjunction with the IEEE Int. Conf. on Big Data, December 2019.

Any problems found, please contact [email protected].

Publications

If you use any of the data, annotations or code, please cite one of the following papers.

Taneeya Satyapanich, Tim Finin and Francis Ferraro, CASIE: Extracting Cybersecurity Event Information from Text, 34th AAAI Conference on Artificial Intelligence, Feb. 2020.
Taneeya Satyapanich, Modeling and extracting information about cybersecurity events from text. Ph.D. Dissertation, University of Maryland, Baltimore County, December 2019.
Taneeya Satyapanich, Tim Finin and Francis Ferraro, Extracting Rich Semantic Information about Cybersecurity Events, 2nd Workshop on Big Data for CyberSecurity, held with the IEEE Int. Conf. on Big Data, Dec. 2019.

casie's People

Contributors

Stargazers

Watchers

casie's Issues

realis_identify.py: error:

Hi,
I got an error:
realis_identify.py: error: the following arguments are required: -trainfile, -testfile, -directory, -label
I am wondering how to solve this.
Any help is appreciated.
Thx

Json format of 10001.json is not consistent with other json files.

wd_search not found

The line "import wd_search" in parseJsontoFeatures.py fails with the following error:
ModuleNotFoundError: No module named 'wd_search'

In which package the module is present? Please suggest.

How to create content.nonstop.label and content.json

Hi,
can you please help me with data generation for content.nonstop.label and content.json

what's the 'wd_search'

In the parseJsontoFeatures.py(line 25 and 212), what’s the ”wd_search” ? I google nothing about it. I guess that’s your code. Would you like sharing your code to me ? Thank you very much.

"from tensorflow import set_random_seed" fails

The line "from tensorflow import set_random_seed" appearing in the following files:

  - nug_arg_detection.py 
  - parseJsontoFeatures.py 
  - prepare.py 
  - x2index.py 
  - nug_arg_detection_bert.py

causes the following error:
ImportError: cannot import name 'set_random_seed'

In TensorFlow2, tf.random.set_seed() should be used instead.

Are you using TensorFlow1, in contrary to what requirements.txt says?

报错了

nug_arg_detection.py: error: the following arguments are required: -trainfile, -testfile, -directory

Hi,
I was able to create .content.json (using stanford corenlp) and content.nostop.label files (with this repo). Now the program is looking for .ner.json files. Could you please provide a sample of these files so I know how to generate them? Many thanks in advance.

.content.nostop.label and .content.json

Any recommendation on how to address the absence of .content.nostop.label and .content.json file extensions in the provided corpus?

Can you please provide some help on this issue in order to reproduce the work?
Thank you.

Requirements installation fails

Requirements installation (pip install -r requirements.txt) fails due to packages versions that can not be found and packages versions conflicts.

For example (copied from pip output):

ERROR: Cannot install -r requirements.txt (line 1) and numpy==1.17.2 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested numpy==1.17.2
bert-embedding 1.0.1 depends on numpy==1.14.6

The version 1.0.1 of bert-embedding appears to be the only present in pypi.org. Therefore I removed the version of numpy to let pip choose the best one. But it caused another problem:
tensorflow-gpu 2.0.0 requires numpy<2.0,>=1.16.0, but you have numpy 1.14.6 which is incompatible.

Also, I could not install tensorflow-gpu 2.0.0 because pip said there is no compatible wheel. Fortunately, it succeeded when I downloaded the wheel to my local file system and installed it from there. But, in addition to the mentioned above, pip complains on the following:

tensorflow-gpu 2.0.0 requires keras-applications>=1.0.8, but you have keras-applications 1.0.7 which is incompatible.
tensorflow-gpu 2.0.0 requires tensorboard<2.1.0,>=2.0.0, but you have tensorboard 1.13.1 which is incompatible.
tensorflow-gpu 2.0.0 requires tensorflow-estimator<2.1.0,>=2.0.0, but you have tensorflow-estimator 1.13.0 which is incompatible.

keras-contrib is not present in pypi.org. It can be installed from Github, but the specified version is not found. (Actually, no specific version can be found in that repository.)
If the current version is OK, could you please replace the respective line in requirements.txt with the following:
git+https://[email protected]/keras-team/keras-contrib.git
BTW, the repository README now says they are migrating to tensorflow/addons. If it's OK to install it instead, please suggest which version to take.

Also, pywikibot==3.0.dev0 is not found in pypi.org anymore. I guess the most close would be pywikibot==3.0.20170403, but its installation fails, because the respective wheel meta-data says it is 3.0.dev0 that is different from the wheel name.

Also, requests==2.21.0 requires version of urllib3 different from the specified (1.25.7)

I am using Python 3.6.13 on Ubuntu 20.04. Do you recommend another version of Python? I might be wrong, but if I remember correctly, some of the listed packages versions need v3.6 specifically.

Anybody know how to create the json files in the annotation folder?

I want to use my own articles as the test data and generate all json, content.json, content.nostop.json.
Anyone know how to create the json files in the annotation folder provided by the author?

Not work

How to run it?Can you give detailed steps?

how to generate .ner.json files

Does anyone know how to generate .ner.json files?
I still don't understand how to generate the files after looking that example.

I don't know how to obtain the "casetype".
And the results I generated using Wikidata and DBpedia spotlight are not quite consistent with the example.

i don't konw how to get content.nostop.label,content.json by using standford corenlp ?

ModuleNotFoundError: No module named 'wd_search'

Question and ask for help

There is an ModuleNotFoundError: No module named 'wd_search', and I didn't find related package in PyPi. Are the related functions renamed?

The other three word2vec models is missing.

Dear author,
I reproduced the result preliminarily after supplementing some missing codes according to all the materials you provided.
However, there is only one Domain-Word2Vec model under the embedding folder you provide.
As for Transfer-Word2Vec, Cyber-Word2Vec and Pre-built BERT Word2Vec models are missing.
Is it convenient for you to supplement the missing three types of word2vec model files?
Your reply will be of great help to my research.
I am looking forward to your reply.
Thank you again!