I need to mining through windows events to capture the events from them.
Unfortunately, your windows events are without a label.
Do you have any metadata about them?
Or doe you know any paper used these events and preparing the labels or metadata?

Thanks

This only happens when processing Android logs with more than 10M.
The exception causes IPLoM finishes much faster than usual. Hence maybe the rest of the logs are not processed.

There are some companies like Zebrium around saying that they get an AI to do root cause analysis of logs. I think root cause analysis is not just about splitting the data between normal and abnormal. But I can only find binary classification data. Is there any multi-class data?

Hi,

Can I ask whether there is any other dataset including clear abnormal items inside logs except HDFS? Thanks.

I notice BGl is labelled dataset. But I did not find the clear labels regarding it from the hub.

Please let me know more details.

Thanks

Hi,

We need to generate a synthetic dataset for an experiment in our upcoming research work. In order to provide any future work in this direction a fair ground for comparison, we want to make the dataset available for download. We are using the sample logs from this repository to generate these logs, however, since there is no licensing information available for the sample logs I wanted to know if we can host this synthetic dataset on our github repository or can loghub help in hosting this dataset.

Thank you.

Hi.

I tried to download the dataset from https://zenodo.org/record/3227177 but could not download any file.
After clicking and waiting for a while, I receive a 503 Service Unavailable error.

Could you please verify if the data is still publically available.

Thank you in advance.
Best,
Marius

Hi,

Thanks for sharing the log files! However, I didn't find the anomaly_labels for datasets in the repo, could you please share the link?

I find the label for HDFS in following link but I cannot find other labels such as the label for BGL dataset.
https://github.com/logpai/loghub/tree/bba4876fb9e45c78501e598950bf5ff68dfef7bf/HDFS

Thank you very much for kindly help!

Sincerely

I have made some research about how to check the event logs to do digital forensic. I found the security logs have the largest relationship with hacker attacks.
When a unauthenticated access or login happen, you can find the record on security logs. While there is nothing showed in CBS logs.

So I have the doubt whether such kind of dataset is really useful when doing the anomaly detection.

2016-01-13 07:48:28,240 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0x13de8a8372744c, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 0 msec to generate and 1 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.

Can you give me information about feature names of the files mentioned in the title? For example

081109 203518 143 INFO dfs.DataNode$DataXceiver: Receiving block blk_-1608999687919862906 src: /10.250.19.102:54106 dest: /10.250.19.102:50010

What does above row say?

in Loghub, HDFS1 has 11,175,629 entries.
in Deeplog, the number is 11,197,954 logs.
I am confused.

thanks

你好何教授，

我想最近在做日志相关的探索，想详细了解下OpenStack数据集里的日志每个文件是做什么的？

openstack_abnormal 这个文件里的instance全部是异常的吗？

openstack_normal1 和 openstack_normal2 两个日志有什么区别吗？

labels 文件里为什么只有4个注入异常类的instance， openstack_abnormal 里的instance数远超4个，那么其他的instance是什么异常呢？这4个标出来的instance有什么特殊吗？

多谢！

I can't find the labels of the Thunderbird logs. In the README.md file of logpai/loghub, it shows that the Thunderbird log is labeled. However, I can't find the labels from the raw log files, which include "https://zenodo.org/record/3227177/files/Thunderbird.tar.gz?download=1", "http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.r43.cf2.rackcdn.com/hpc4/tbird2.gz". Could you please help me?

I applied for access to the LogPai team because I was specifically interested in the OpenSSH logs for my work. Is there any way I can get access to them for my dissertation work? If not tons of logs, at least a few MBs would be great. Anything more than the 2K that is present. The help is much appreciated.

Hi,

I need to run a few experiments on the andorid dataset mentioned in the paper "Tools and Benchmarks for Automated Log Parsing". Would it be possible to share that dataset?

Thank you.

Do we have samples of authentication logs here?

How can i get the raw Spirit dataset(172 million log messages)

Could you provide the message types for these system logs and they could be used for ground truth for the effectiveness of a log parser

I want to download labeled hdfs logs to test my code , but I can't link to zenodo.org. Have another way to download the logs?

Not sure whether it's an issue from here. But when try to read the current Linux.log (zenodo, md5:6d1802d7778126f21c001c6aa7b6b106) with python i got

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 20: invalid start byte

can you confirm that or is that something probably going wrong on my side?

Dear LogPai team,

Thank you for maintaining this wonderful log datasets.

I just wonder if it’s okay to redistribute a few of your 2K logs (e.g., https://github.com/logpai/loghub/blob/master/HDFS/HDFS_2k.log) just as an example log dataset in my replication package.

Though you kindly noted here that “the log datasets are freely available for research purposes”, it’s not clear to me if this includes the redistribution right of the log datasets. If possible, I will present a clear reference to this dataset repository and then include a few sample logs in my replication package as examples.

Looking forward to hearing from you soon.

Thanks,
Donghwan

Not all the logs are available on zenodo. Is there some way to download all of them?

i applied logs on zendoo but i didn't find windows logs which i need. So where can i get them? please do tell, thanks a lot.

Very cool dataset and impressive to see how much use/impact it is having! It would be nice if it was also possible to access it via the Hugging Face Hub (https://huggingface.co/datasets). There are a few possible approaches to doing this (and it's also possible to have gated acces). Happy to help with this if it is of interest!

感谢您的付出，点赞！

Hey Im having problem finding the complete file cause 2k is not enough data for a model Im testing on. Can you provide the link to get the complete HDFS.log raw log file?

How to obtain the regular expressions used for various datasets

Could you please provide the tables for the F-1 measure for the supervised anomaly detection models (SVM, LR, and DT)?

Hello!

In LogHub, Hadoop log dataset are divided into different application parts with different id and have labels very clearly for every type but in LogPub(logHub 2.0), you mixed them only in one file but i want to get just WORDCOUNT application part with the same file path format of that as in LogHub. Can you help me with that?

Thank you very much!