zc8340311 / robustautoencoder Goto Github PK

Hi Chong,

First of all, thank you for your interesting article. We tried to implement your idea in Pytorch, and use it in production for anomaly detection. In our case we have a huge dataset of measurements without labels or ground truth. We know that the data has some outliers which we could not catch with linear models. So we would like to train or model (AE) to learn on non anomalous data if possible. However, we ran into some issues, and I would like to ask some questions, if you don't mind. I'm trying to use the outlier detection model with l21 regularization.

So the idea is to separate anomalous data into the S matrix. In my understanding it is only possible to use it for the training data to make prediction of anomalous instances, right? What do you recommend for classifying on unseen data? Normally autoencoders are used in a way, that the reconstruction error tells how much an instance is an outlier. In this case, what is the difference between relying on the S matrix instead of the reconstruction error? Should be there any correlation between these? (like high reconstruction error <-> S is non zero in that row)

In the article you stated that "non zero rows are outliers" in S. Does it mean if at least one element in a row of S is greater than zero, then the whole row(data instance) is an anomaly? Or the whole row has to be > 0 to consider it as an outlier?
In my case I cannot choose a correct lambda value which would give at least reasonable results. The problem is that always the same columns have non zero values. If I make lambda larger I will have more zeros in S but still all non zero elements will be in the same columns, only difference is that some additional columns will become 0. This means I will never get totally zero rows only if the whole matrix is zero (a little smaller lamdba would still give at least on column full of non zero elements)

As I understood, lamdba depends on the actual input data, in my case it ranges around 10-140. (Above 140 the whole S is 0, below 0-10 S does not have any 0.)
What seems strange is that in your case the plotted loss function decreases gradually with steps, every time you minimalize the L,S matrices, but in my case right after the first model training (so basically the first time L-S happens when S is not the zero matrix), the loss drops drastically to very small values (like 0.001) then the model hardly improves, sometimes the second L-S even makes the model diverge.

Also, I've made some modifications on the shrink methods, to be able to use them on high amount of input data. Basically, I've changed them to vectorized functions:

def l1shrink(mat, eps):
    output = np.zeros_like(mat)
    cond_pos = mat > eps
    cond_neg = mat < -eps
    output[cond_pos] = mat[cond_pos] - eps
    output[cond_neg] = mat[cond_neg] + eps
    return output

def l21shrink(mat, eps):
    output = np.zeros_like(mat)
    norm = np.linalg.norm(mat, ord=2, axis=0)
    condition = norm > eps
    output[:, condition] = mat[:, condition] - eps * mat[:, condition] / norm[condition]
    return output

Thank you,
Daniel

Hello Dear,
I'm currently working denosing GPR B-Scan images. I have GPR B-Scan dataset that is collected from real environment. The dataset is working with RPCA, I mean, original image (https://ibb.co/r6ffTDd) is denoised very well. But when I try same images with this your algorithm, It does not give any result. I changed lambda and other parameters but nothing changed. The output S same with input image. Can you give any advice to me about this ?

Hi there,

I would like to plot the convergence of ADMM training method, but am not sure what I should do.

In "l21RobustDeepAutoencoderOnSt.py",
for it in range(iteration):
if verbose:
print("Out iteration: " , it)
## alternating project, first project to L
self.L = X - self.S
## Using L to train the auto-encoder
self.AE.fit(self.L, sess = sess,
iteration = inner_iteration,
learning_rate = learning_rate,
batch_size = batch_size,
init = re_init,
verbose = verbose)
## get optmized L
self.L = self.AE.getRecon(X = self.L, sess = sess)
## alternating project, now project to S and shrink S
self.S = SHR.l21shrink(self.lambda_, (X - self.L).T).T

        **WRITE some codes to compute the objective function min||self.L - D(E(self.L))||2 + lambda||self.S||2,1 HERE?**

Any comments will be appreciated.

Hi,

I have one question regarding the calculation of the loss at the end of each epoch in the DAE script (RobustAutoencoder/experiments/Outlier Detection/DAE_tensorflow.py).

Currently the code look like:

# ....
        error = []
        sample_size = X.shape[0]
        turns = sample_size / batch_size

        for i in xrange(iteration):
            for turn in xrange(turns):

                start = (batch_size*turn) % sample_size
                end = (batch_size*(turn+1)) % sample_size

                sess.run(train_step,feed_dict = {input_x:X[start:end]})
            e = cost.eval(session = sess,feed_dict = {input_x: X[start:end]})
            if verbose and i%20==0:
                print "    iteration : ", i ,", cost : ", e
            error.append(e)
# ....

As the variable "start" and "end" still point to last used batch in the epoch the loss value will always be calculated only over the records of the last batch.
Wouldn't it make more sense to calculate it using the full dataset X - like

# ....
        error = []
        sample_size = X.shape[0]
        turns = sample_size / batch_size

        for i in xrange(iteration):
            for turn in xrange(turns):

                start = (batch_size*turn) % sample_size
                end = (batch_size*(turn+1)) % sample_size

                sess.run(train_step,feed_dict = {input_x:X[start:end]})
            e = cost.eval(session = sess,feed_dict = {input_x: X})
# ....

?

Best, Roberto

BTW: extremely interesting paper - thanks for making the code available

elif norm[i] < -epsilon should remove

Could you please provide the outlier MNIST dataset and scripts to produce the results in the paper? After training, how ca we detect the outliers according to the sparse matrix S? If it is based on the norm of S, how did you set the threshold?

Hi Thank you for the interesting paper. Do you know if there is any pytorch implementation of this paper? Thanks.

Hi,

Is it possible to share the slides you presented at KDD 2017?
If so, I would be highly appreciated.
Thanks in advance.

JM

Hi,

I have one question regarding the inference/prediction method "getRecon" (RobustAutoencoder/model/l21RobustDeepAutoencoder.py or RobustAutoencoder/experiments/Outlier Detection/RDAE_tensorflow.py):

This method looks like

def getRecon(self, X, sess):
        return self.AE.getRecon(self.L, sess = sess)

So it will create the reconstruction using the trained autoencoder on the "cleaned" X which is L (or LD in the paper).
But shouldn't that be

def getRecon(self, X, sess):
        return self.AE.getRecon(X, sess = sess)

?
Here we use the full X dataset. That way we can use the trained robust autoencoder to make inference on new data (test, validation data). The resulting reconstruction can be compared to the input and we can define any anomaly score based on the difference of those two.

Best, Roberto

When I run the basic AE code, I got the error loading the file
~/Desktop/RobustAutoencoder/model/BasicAutoencoder$ python3 DeepAE.py
Traceback (most recent call last):
File "/home/daozhang/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 428, in load
return pickle.load(fid, **pickle_kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb8 in position 598: ordinal not in range(128)

Traceback (most recent call last):
File "/home/daozhang/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 428, in load
return pickle.load(fid, **pickle_kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb8 in position 598: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "DeepAE.py", line 94, in
x = np.load(r"./data/data.npk")
File "/home/daozhang/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 431, in load
"Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file './data/data.npk' as a pickle

In script DeepAE.py, i think the last layer of the AE should not pass sigmoid. can you give me your suggestions in your time, thx.