houssamzenati / efficient-gan-anomaly-detection Goto Github PK

View Code? Open in Web Editor NEW

432.0 20.0 137.0 18.29 MB

License: MIT License

Python 100.00%

efficient-gan-anomaly-detection's Introduction

Efficient-GAN-Based Anomaly Detection

Official implementation of the prepublished article submitted to the ICLRW 2018: https://arxiv.org/abs/1802.06222

NEW! Updated version of this work in "Adversarially Learned Anomaly Detection" paper!

Anomaly Detection materials, by the Deep Learning 2.0 team in I2R, A*STAR, Singapore

Please reach us via emails or via github issues for any enquiries!

Please cite our work if you find it useful for your research and work:

@article{zenati2018,
  author    = {Houssam Zenati and
               Chuan Sheng Foo and
               Bruno Lecouat and
               Gaurav Manek and
               Vijay Ramaseshan Chandrasekhar},
  title     = {Efficient GAN-Based Anomaly Detection},
  year      = {2018},
  url       = {http://arxiv.org/abs/1802.06222},
  archivePrefix = {arXiv}
}

Prerequisites.

To run the code, follow those steps:

Install Python 3

sudo apt install python3 python3-pip

Download the project code:

git clone https://github.com/houssamzenati/Efficient-GAN-Anomaly-Detection

Install requirements (in the cloned repository):

pip3 install -r requirements.txt

Doing anomaly detection.

Running the code with different options

python3 main.py <gan, bigan> <mnist, kdd> run --nb_epochs=<number_epochs> --label=<0, 1, 2, 3, 4, 5, 6, 7, 8, 9> --w=<float between 0 and 1> --m=<'cross-e','fm'> --d=<int> --rd=<int>

To reproduce the results of the paper, please use w=0.1 (as in the original AnoGAN paper which gives a weight of 0.1 to the discriminator loss), d=1 for the feature matching loss.

efficient-gan-anomaly-detection's People

Contributors

Stargazers

Watchers

Forkers

yarqian lidan456 minkcho ykwon0407 dangxuanhong chidperi dineshresearch james-fu deepdrive dawdleryang hijuly bemoregt moonight10 gregcusack sdd9465 kianqunki hrnbot lurenlym rajpratim tartaruszen newbie-gk fasladodo zhengze2014 swordfishwjy musingxu will-rice nakaokarei qweshpd jaykimbravekjh zbn123 bianjiang1234567 sashirin westamine valeman lwpyh itaishchorry patrickcnkm leon651cd yanyushu william-vu samysung shubhampachori12110095 jiweitian yusukekyokawa jdc08161063 stevenfyy aisaturday shuangyumo medical-images-process chunchill youndoldman shudichen dorianxiao cjen07 jc7553a phuccuongngo99 david-zzy tiger115136 lishuai0000 snapcrab snowisland1 jzhao23 ai404 tanmdl aayush-1 xiongnudahan lm-jira gg00 xiangnanhe siayou judithcodes wzg16 yxinjiang jibanli gar1t dukedj wylx14 scampion xuanli98 lxmwust yoonkim97 aksarburak 15529523003 jwc92 marissaxiao yixiyixi5 cuirun johnny328 andyshields dgl547437235 sxjpage xuhan314 imygirl ntoron ajayarunachalam koapt mesarcik gaodawn manmeet3 ahsan-ul-haq30

efficient-gan-anomaly-detection's Issues

Request for the code: adversarially learned anomaly detection

BiGAN: intuitive idea behind the encoder training step

Hi, I'm reading your code and looking at the BiGAN implementation and I was wondering:
why the loss function for E is separate from the loss function of G?

In the standard adversarial training, we train D as in a min-max game, where the discriminator should judge between the real and fake samples and G uses the judgment of D on the generated samples to update its parameters.

Hence, can't the min-max game be played also from G and E? Both use the information generated by D and E must learn the opposite task of G. So, why the loss can't be something like

loss_ge = binary_cross_entropy( D( (G(z), z), 1) + binary_cross_entropy( D( (x, E(x)), 0)

I'm trying training using this loss function and the train is unstable, sometimes the generated images look fine, but after some epoch the model collapses and generates black images. After a few steps, the output still comes back to something realistic...

Probably there's something obvious that I'm missing.
Thank you for your help!

Normal and Anomalous Labels

Hello Houssam,

First of all I'd like to thank you for sharing the code and the amazing paper you and your fellow researchers produced. I have a few questions regarding the implementations, it would be great if you could take some time and look at these.

In kdd.py, it was assumed 1 for anomalous label and 0 for normal label throughout the code, but in the function _get_dataset the anomalous data was assigned with label 0 and the normal 1 (lines 46-47), which confused me, these two lines also contradict with the comment in line 128. I did reverse the label in these two lines (keeping other settings intact, cross entropy loss, L1 norm in reconstruction loss used) and the F1 score dropped drastically. Please correct me if I was wrong.
In BiGAN_run_kdd.py when you are evaluating the anomalous data in the test set, you used the fact that 20% of the test data are anomalous and thus assigned anomalous label to the data with top 20% scores in test set (lines 319-328). Is that allowed in anomaly detection? I think an anomaly detection algorithm should be able to detect unknown anomalies.

Thank you for your contribution, it's an exciting work!

Zheyu

Get_shape_input

could you kindly explain why the shape of dataset is mentioned as None,121 in get_shape_input function?

A question about the feature

Hello, thanks for your nice work! I have a question that has you tried to utilize the feature extracted from the latent layer? i.e. the output of E(x). Because in the works related to autoencoder, the authors usually used the latent layer as the feature extractor, but in the work related to BiGan, I do not find this kind of usages, though BiGan is equal to an autoencoder in the optimal situation. Maybe I do not explain my question here clearly, it is so nice if I can send you the email if you don't mind.

Need help on reproducing KDD result

Questions:

May I know what may have gone wrong? I downloaded the code and ran it as is. I also ran for 1000 Epoch but result was not good at all. I once even ran it for 5000 epoch and got this result:
Epoch 4999 | time = 15s | loss gen = 3.2600 | loss enc = 329.4751 | loss dis = 0.3632
Testing : Prec = 0.1924 | Rec = 0.5634 | F1 = 0.2869
How come the 2nd run, the loss gen, loss enc were much smaller but F1 got worse? Shouldn't be the lower the losses, the better F1? (given than: list_scores = (1-weight) * gen_score + weight * dis_score)
Why the loss enc always seem so big? Anyway to improve it? Note that I tried to add a layer_3 in encoder and/or to add dropout in encoder and generator but didn't help.

Thanks

training on GPU or CPU?

I installed your code and tried to run the example, but I found it ran on CPU, not on GPU. Do I need to change tensorflow into tensorflow-gpu in your requirement.txt?

At train AnoGAN with MNIST data, Does the train set and test set all contain only normal data?

Hi, @houssamzenati

In the case of Anomaly Gan, I knew that it trained with only normal datasets.

At train AnoGAN with MNIST data, Does the train set and test set all contain only normal data?

Or are both normal + abnormal divided into folders?
I wonder.

form @bemoregt.

No activation in the Encoder 1st Layer

No activation layer or nonlinearity is found in the block of code describing the first layer of the Encoder.

Unable to understand the encoder loss

Hello,

I have a question regarding the loss calculation for the encoder.
loss_encoder = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.zeros_like(l_encoder),logits=l_encoder))

As per my understanding, the encoder is being optimized to have the discriminator output "0" (Fake) for the input combination of (x, E(X)) i.e. (Real Image, Encoder output of the Real Image). Could you please elaborate on why this is so or point me towards some relevant material?

I have read the associated research paper and the BiGAN paper which states that the generator and encoder should be updated together but I'm not able to relate it to how the encoder loss is calculated here.

Am I missing something obvious?

Example parameters

Hi, I found your research quite interesting! Could you provide some example sets of parameters that you know to work well with these example datasets?

Troubles applying GAN for anomaly detection

Hi Mr Houssam Zenati,

My name is Norbert and I was amazed with the opportunities of your code. I was trying to use it in a real image dataset to classify inbetween one class of image and a diferent one (one good image and the rest, like "cat" and "no cat"). But I was unable to understand your readme instructions completely. Could you provide and easy example with a real dataset? I've lost a bunch of hours to figure out how to run your code, because I could not completely comprehend it.

If you have little time, also a chat or videochat would be a possibilty,

Thank you very much and I am here for any question or suggestion,

Norbert.

How the precision range was calculated?

First of all thanks for the amazing paper. Quickly wanted to know how the range for the below metrics was found. When i run the code, i see absolute value.
Our Modelσ 0.9200 ± 0.0740 0.9582 ± 0.0104 0.9372 ± 0.0440

Datasets with other dimensions

Hello Mr.,

First of all, congratulations on your work! It is very interesting and insightful.
I was able to run your code with the kddcup99 dataset. However, I would like to know if you could give me some help and directions on how to adapt your code for other datasets with different dimensions. For example, what I would have to do if I want to use only a few features of the kddcup99 dataset, so I have 15 features instead of 121?

Thank you very much for your help and assistance!
Best regards,
Paulo

why do you use abnormal data to train?

when I want to use my self dataset,I found your model just use abnormal dataset of KDD99 to train.But When I read some other paper, they said when they use GAN model to do abnormal detection ,they just use normal dataset to train,not abnormal dataset. Could you answer my question?

about contrib module

Hi.
In run_mnist, I get the error "no contrib".
Instead of tensorflow, I use

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

instead of tensorflow, but it does not improve the problem.
I don't want to change the version of tensorflow if possible. How can I make it work?

Libculas Issue when running command as mentioned in repo

python3 main.py bigan kdd run --nb_epochs=50 --label=5 --w=0.2 --m=cross-e --d=2 --rd=2

|_ .|_ _| / ___ . .' '.
| | . \ | | |_/___) | | .--. | | | | | | | _ .'____.' | | | | _| |_.' /_| |__/ | / /_____ _| --' |
|.'|__| |__|()'..'

[11:14:57 INFO @AnomalyDetection] Running script at bigan.run_kdd
[11:14:57 ERROR @AnomalyDetection] Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 40, in run
mod = importlib.import_module(mod_name)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/ubuntu/Efficient-GAN-Anomaly-Detection/bigan/run_kdd.py", line 3, in
import tensorflow as tf
File "/usr/local/lib/python3.7/dist-packages/tensorflow/init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
[11:14:57 ERROR @AnomalyDetection] Uhoh, the script halted with an error.

the problem about the code

hi，thanks for your sharing of the cool idea.
I have a question about the code，displaying as bellow：

ran_from = nr_batches_test * batch_size
ran_to = (nr_batches_test + 1) * batch_size
size = testx[ran_from: ran_to].shape[0]
fill = np.ones([batch_size - size, 121]) # why all ones
batch = np.concatenate([testx[ran_from:ran_to], fill], axis=0)

I cannot understand the meaning of creating such batch of the data.
can you give me a detailed explanation？

thank you！

anomalous label for the experiment

hi，
parser.add_argument('--label', nargs="?", type=int, help='anomalous label for the experiment')
--label=<0, 1, 2, 3, 4, 5, 6, 7, 8, 9>
I don't know the real meaning of anomalous labels. Does the label choice (0-9) have some effect on the detection performance?

Hi, I have something problem in main.py

I try to run your program today.
When I use this command "python main.py gan kdd run --nb_epochs=1 --label=1",
it will be error.
The following message is the content of this error
"""
Traceback (most recent call last):
File "main.py", line 67, in
parser.add_argument('--d', nargs="?", default=1, defaulttype=int, help='degree for the L norm')
File "/anaconda3/lib/python3.6/argparse.py", line 1334, in add_argument
action = action_class(**kwargs)
TypeError: init() got an unexpected keyword argument 'defaulttype'
"""
This is caused by the "defaulttype".
I think it is a mistake, since other parameters is used "type", but the parameter "--d" is not.
Is it right?

k.w.

Cannot run the script

Hello,
I am trying to run this anomaly detection code provided by you on a mac (so tensorflow with CPU). However, I am using issues. For convenience, I am providing the command line instruction I used to run the code and the Traceback of the issue I am getting.

command line command:
python3 main.py 'bigan' 'mnist' 'run' --nb_epochs=10 --w=0.1 --m='fm' --d=1 --rd=44

Traceback

[10:25:58 INFO @AnomalyDetection] Running script at bigan.run_mnist
[10:25:58 ERROR @AnomalyDetection] Traceback (most recent call last):
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: dlopen(/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 10): Library not loaded: @rpath/libcublas.8.0.dylib
  Referenced from: /anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
  Reason: image not found


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: dlopen(/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 10): Library not loaded: @rpath/libcublas.8.0.dylib
  Referenced from: /anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 40, in run
    mod = importlib.import_module(mod_name)
  File "/anaconda3/envs/py36/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tridibdutta/Documents/GAN_ANOMALY_PAPER/Efficient-GAN-Anomaly-Detection/bigan/run_mnist.py", line 3, in <module>
    import tensorflow as tf
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 51, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/anaconda3/envs/py36/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: dlopen(/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 10): Library not loaded: @rpath/libcublas.8.0.dylib
  Referenced from: /anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
  Reason: image not found


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
[10:25:58 ERROR @AnomalyDetection] Uhoh, the script halted with an error.

Can't figure out how to run the code properly and find the anomaly.

Thanks.