And Why the values of the field 'hour' in this avazu version are different from the field 'hour' in the standard avazu dataset?

Hi, I came across CIKM2021-SimpleX: A Simple and Strong Baseline forCollaborative Filtering, Nice work!
However, in terms of Table 1, I could not reproduce the result of BCELoss on yelp2018, Looking forward to your help!
Thanks a lot @zhujiem @xpai

aobaoAd_x1
Dataset description

Taobao is a dataset provided by Alibaba, which contains 8 days of ad click-through data (26 million records) that are randomly sampled from 1140000 users. Following the original data split, we use the first 7 days (i.e., 20170506-20170512) of samples for training, and the last day's samples (i.e., 20170513) for testing. We follow the preprocessing steps that have been applied to reproducing the DMR work. Note that a small part (~5%) of samples have been dropped during preprocessing due the missing of user or item profiles. The preprocessed data can be accessed from the BARS benchmark.

上述描述部分指出： a small part (~5%) of samples have been dropped during preprocessing due to the missing of user or item profiles.

可以请教下，这部分sample是如何去除的吗？是否在 DMR data preprocessing 所提供代码的基础上引入了新的逻辑呀？感谢~！

Hello, in taobao dataset it contain raw behavior dataset。 According to L36 - L56 https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Taobao/Taobao_x1/split_taobao_x1.py#L36， it create click history sequence from raw_sample dataset, not use the raw behavior dataset。

I am trying to reproduce the results of DeepFM on criteo_4x_001 dataset. I have setup my enviroment as follows:

python             3.6.13
cuda                11.7
torch               1.10.2
fuxictr             1.0.2
h5py                3.1.0
numpy               1.19.5
pandas              1.1.5
scipy               1.5.4

Is is not exactly the same as the environment in this repo, but at least I have set up fuxictr version 1.0.2 exactly.
Then I followed the config as
https://github.com/reczoo/BARS/tree/main/ranking/ctr/DeepFM/DeepFM_criteo_x4_001

The results were slightly differerent. I noticed that the AUC results in the original exepreiment had a big jump from 0.809407 to 0.813303 at epoch 4 to 5. My results also had such kind of a jump, but it came later at epoch 8 to 9, where AUC jump from 0.809898 to 0.813443.

benchmark里面的代码何时可以开源？

谢谢！

I directly downloaded the preprocessed Avazu_x4 dataset, then discarded the id feature, and processed all other features as string categorical features. However, it is very strange that it needs a huge amount of GPU memory to be loaded (about 31G !!!!), and I can’t even use this dataset on Nvidia V100 32G because of OOM.
Is this normal? Is there any way to fix this?

When will sequential models like din/dien to be evaluated on these public benchmarks? Do you have a plan?

https://zenodo.org/record/6324454/files/Movielenslatest_x1.zip?download=1
The above link I found at https://github.com/openbenchmark/BARS/blob/main/datasets/MovieLens/README.md is not usable, would you please provide a new one?

torch.jit.trace(fibinet, batch_data)

File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/jit/_trace.py", line 794, in trace
return trace_module(
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/jit/_trace.py", line 1056, in trace_module
module._c._create_method_from_trace(
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given

Here is the config:

{
"batch_norm": "True",
"batch_size": "4096",
"block1_dropout": "0.1",
"block1_hidden_activations": "ReLU",
"block1_hidden_units": "[400, 400]",
"block2_dropout": "0",
"block2_hidden_activations": "ReLU",
"block2_hidden_units": "[400, 400]",
"block_type": "2B",
"data_format": "csv",
"data_root": "/data/workspace/rec/dataset/Criteo_x1/",
"dataset_id": "criteo_x1_fuxictr2",
"debug_mode": "False",
"early_stop_patience": "2",
"embedding_dim": "10",
"embedding_regularizer": "1e-05",
"epochs": "100",
"eval_interval": "1",
"eval_steps": "None",
"feature_cols": "[{'active': True, 'dtype': 'float', 'name': ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], 'type': 'numeric'}, {'active': True, 'dtype': 'float', 'name': ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], 'type': 'categorical'}]",
"feature_specs": "None",
"gpu": "0",
"group_id": "None",
"label_col": "{'dtype': 'float', 'name': 'label'}",
"learning_rate": "0.001",
"loss": "binary_crossentropy",
"metrics": "['AUC', 'logloss']",
"min_categr_count": "1",
"model": "FinalNet",
"model_id": "FINAL_criteo_x1_028_b6c861e4",
"model_root": "./checkpoints/FINAL_criteo_x1/",
"monitor": "AUC",
"monitor_mode": "max",
"net_regularizer": "0",
"num_workers": "3",
"optimizer": "adam",
"ordered_features": "None",
"pickle_feature_encoder": "True",
"save_best_only": "True",
"seed": "2021",
"shuffle": "True",
"task": "binary_classification",
"test_data": "/data/workspace/rec/dataset/Criteo_x1/test.csv",
"train_data": "/data/workspace/rec/dataset/Criteo_x1/train.csv",
"use_field_gate": "True",
"valid_data": "/data/workspace/rec/dataset/Criteo_x1/valid.csv",
"verbose": "1"
}

Here is the log:
FINAL_criteo_x1_028_b6c861e4.log

Hello,

Thank you so much for your very useful work.

I have a question about results on Gowalla dataset, based on the log in https://github.com/openbenchmark/BARS/tree/master/candidate_matching/benchmarks/SimpleX/SimpleX_gowalla_x1

How do you divide training data into training and validation sets? From the log, test set is currently treated as validation set and reported results are selected as the best iteration on test set, which causes test data leakage. A similar problem was reported in this paper https://arxiv.org/pdf/2005.09683.pdf?

Look forward to hearing from you soon,
Thank you so much.

您好，我之前有看到开源openctr的代码，是什么原因删了，可以提供一下openctr的代码吗

How to choose the method of embedding table Initialization? if I choose the xavier_uniform method, the AUC is always 0.5 when I run the code of SAM model.But when I run the code of other models (AutoInt, FiGNN, AFN)using the xavier_uniform method, the results are good and normal.
I see that the method using in the project is normal initialization(embedding_initializer="partial(nn.init.normal_, std=1e-4)"). Is all model the mehod?

I want to run datasets such as Criteo and Avazu on the DIEN sequence model. How should I modify the parameters in the model_config.yaml file? Looking forward to your reply！

when I download Avazu dataset in kaggle, the format of Avazu is csv
but when I run the code and the format of DeepFM_avazu_x4_tuner_config_01.yaml is h5
how can I transform csv to h5?

Hi,

Excellent work, thanks for sharing the code. The link of KuaiVideo_x1 and TaoAd_x1 in DIN and DIEN seems not work.
Could you provide a new link to download datasets?

FuxiCTR 何时开源？

谢谢！

I noticed that the latest version of fuxictr was 2.3.x. Does this repo run on fuxictr 2.3.x?
By the way, I nocited that many of the requirement file in this git group do not specify version.
I strongly suggest doing it. I have encountered problems of version compatiblity of some depended libraries.
I have to try downgrading version by version to test which version is compatible.

This is a nice project. Just one question. I can find very few introduction about the implementation.
Are all the methods implemented on the same base framework like OpenMMlab projects, e,g., mmdetection?
Or the implementation of each method is a standalone bunch of code, borrowed from other sources?

Though BGCF KDD'20 provides the dataset splitting details, it is still hard to fully reproduce it (due to the lack of the random seed). Could you provide official links of AmazonCDs/Movies/Beauty_m1 to download?

Thanks.

DNN_avazu_x1 log shows "2022-02-08 10:25:00,299 P50417 INFO [Metrics] AUC: 0.763019 - logloss: 0.368178".
But the results in my machine is "2023-04-25 19:35:46,393 P47366 INFO [Metrics] AUC: 0.755872 - logloss: 0.371449"

The reason may be the inconsistency of the network, where the number of parameters in DNN_avazu_x1 log is 13805192 while mine is 13805191. I can't figure out what the 1 parameter is.
The detail of my parameters is as follows:
embedding_layer.embedding_layer.embedding_layer.feat_1.weight torch.Size([8, 10])
embedding_layer.embedding_layer.embedding_layer.feat_2.weight torch.Size([8, 10])
embedding_layer.embedding_layer.embedding_layer.feat_3.weight torch.Size([3479, 10])
embedding_layer.embedding_layer.embedding_layer.feat_4.weight torch.Size([4270, 10])
embedding_layer.embedding_layer.embedding_layer.feat_5.weight torch.Size([25, 10])
embedding_layer.embedding_layer.embedding_layer.feat_6.weight torch.Size([4863, 10])
embedding_layer.embedding_layer.embedding_layer.feat_7.weight torch.Size([304, 10])
embedding_layer.embedding_layer.embedding_layer.feat_8.weight torch.Size([32, 10])
embedding_layer.embedding_layer.embedding_layer.feat_9.weight torch.Size([228185, 10])
embedding_layer.embedding_layer.embedding_layer.feat_10.weight torch.Size([1048284, 10])
embedding_layer.embedding_layer.embedding_layer.feat_11.weight torch.Size([6514, 10])
embedding_layer.embedding_layer.embedding_layer.feat_12.weight torch.Size([5, 10])
embedding_layer.embedding_layer.embedding_layer.feat_13.weight torch.Size([5, 10])
embedding_layer.embedding_layer.embedding_layer.feat_14.weight torch.Size([1939, 10])
embedding_layer.embedding_layer.embedding_layer.feat_15.weight torch.Size([9, 10])
embedding_layer.embedding_layer.embedding_layer.feat_16.weight torch.Size([10, 10])
embedding_layer.embedding_layer.embedding_layer.feat_17.weight torch.Size([348, 10])
embedding_layer.embedding_layer.embedding_layer.feat_18.weight torch.Size([5, 10])
embedding_layer.embedding_layer.embedding_layer.feat_19.weight torch.Size([60, 10])
embedding_layer.embedding_layer.embedding_layer.feat_20.weight torch.Size([170, 10])
embedding_layer.embedding_layer.embedding_layer.feat_21.weight torch.Size([51, 10])
embedding_layer.embedding_layer.embedding_layer.feat_22.weight torch.Size([25, 10])
dnn.dnn.0.weight torch.Size([400, 220])
dnn.dnn.0.bias torch.Size([400])
dnn.dnn.3.weight torch.Size([400, 400])
dnn.dnn.3.bias torch.Size([400])
dnn.dnn.6.weight torch.Size([400, 400])
dnn.dnn.6.bias torch.Size([400])
dnn.dnn.9.weight torch.Size([1, 400])
dnn.dnn.9.bias torch.Size([1])

I followed the instruction in this website: https://github.com/openbenchmark/Open-CTR-Benchmark/blob/master/benchmarks/AFM/AFM_Avazu_x4_001.md, but I found data format mismatch.

In data preprocess:https://github.com/openbenchmark/Open-CTR-Benchmark/blob/master/datasets/Avazu/Avazu_x4/split_avazu_x4.py, the format is '.csv'.

But in running config: https://github.com/openbenchmark/Open-CTR-Benchmark/blob/master/benchmarks/AFM/AFM_avazu_x4_tuner_config_01.yaml, the format is '*.h5'

Hi,

I am trying to use this to reproduce results of some benchmarks. But h5 file does not work well on our computation system.
Is there any reason why using h5 file? Could we directly use csv file in constructing train, val and test data?
directly using csv is not efficient?

Thanks~

您好，
在DCN_criteo_x4_001中，按照Running steps做出的结果是
"data_format": "csv",
"dataset_id": "criteo_x4_9ea3bdfc",
"model_id": "DCN_criteo_x4_012_dc8ab363",
"model_root": "./Criteo/DCN_criteo_x4_001/",
最终结果 logloss:0.437622 - AUC:0.814319

但README中的logs信息为
"data_format": "h5",
"dataset_id": "criteo_x4_5c863b0f",
"model_id": "DCN_criteo_x4_5c863b0f_012_56382345",
"model_root": "./Criteo/DCN_criteo/min10/",
最终结果 logloss: 0.437612 - AUC: 0.814437
且项目中没有找到README中包含的内容

@zhujiem @xpai 感谢解答！

• [TKDD'2020] Core Interest Network for Click-Through Rate Prediction

• [CIKM2020] Deep Multi-Interest Network for Click-through Rate Prediction

• [CIKM'21] Efficient Learning to Learn a Robust CTR Model for Web-scale Online Sponsored Search Advertising

• [CIKM'21] Enhancing Explicit and Implicit Feature Interactions via Information Sharing for Parallel Deep CTR Models

• [CIKM'21] One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction

• [CIKM'21] AutoIAS: Automatic Integrated Architecture Searcher For Click-Trough Rate Prediction

• [CIKM'21] Click-Through Rate Prediction with Multi-Modal Hypergraphs

• [CIKM'21] AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation

• [CIKM'21] Disentangled Self-Attentive Neural Networks for Click-Through Rate Prediction

I found that results of NIA-GCN are reported in your survey paper, I was wondering if you could share the source code for NIA-GCN?

I want to reproduct AutoInt+ with KKBox config,here https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/benchmarks/AutoInt/AutoInt%2B_kkbox_x1/AutoInt%2B_kkbox_x1_tuner_config_06.yaml .but i can't get the result logloss: 0.476795 - AUC: 0.852967.myr result is here
.
My config is

您好！ 我在使用fuxictr_1.0.2+BARS时，按FGCNN_avazu_x4_001的配置文件运行代码，AUC始终为0.5。将配置文件中的conv_batch_norm与dnn_batch_norm全部设置为false时才能运行出有效结果（库中原配置文件conv_batch_norm为true），这是为什么？
或者将学习率由0.001调整到0.0001也能解决此问题

Hello~ I find there are no dataset config files of Taobao dataset in the folder of benchmarks, would you provide them in a further version?

感谢作者建立并开源了BARS项目，是 CTR prediction 领域巨人的肩膀

在 model_config.yaml 中会有多个模型配置，如：

• DIN_taobaoad_x1_001_6c779213
• DIN_taobaoad_x1_002_5f3df4a0
• ...
• DIN_taobaoad_x1_012_13c23a36

请问这里的八位数字/字母组合得到的后缀，有什么含意吗？以及他们是怎么生成的

感谢~！