Comments (10)
After checking the training log, ST-Norm's overall MAPE is indeed problematic.
The correct number is 8.56%.
Here are the full training results:
2022-11-03 12:55:25,075 - easytorch-training - INFO - Epoch 95 / 100
2022-11-03 12:56:18,227 - easytorch-training - INFO - Result <train>: [train_time: 53.15 (s), lr: 5.00e-04, train_MAE: 18.2307, train_RMSE: 30.4419, train_MAPE: 0.0808 ]
2022-11-03 12:56:18,228 - easytorch-training - INFO - Start validation.
2022-11-03 12:56:22,276 - easytorch-training - INFO - Result <val>: [val_time: 4.05 (s), val_MAE: 19.8844, val_RMSE: 32.5213, val_MAPE: 0.0858]
2022-11-03 12:56:22,307 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_best_val_MAE.pt saved
2022-11-03 12:56:26,161 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 17.2057, Test RMSE: 27.4493, Test MAPE: 0.0720
2022-11-03 12:56:26,164 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 18.3410, Test RMSE: 29.7707, Test MAPE: 0.0766
2022-11-03 12:56:26,166 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 19.1426, Test RMSE: 31.5075, Test MAPE: 0.0798
2022-11-03 12:56:26,169 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 19.7441, Test RMSE: 32.8585, Test MAPE: 0.0830
2022-11-03 12:56:26,172 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 20.2191, Test RMSE: 34.0451, Test MAPE: 0.0849
2022-11-03 12:56:26,174 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 20.6346, Test RMSE: 34.9905, Test MAPE: 0.0861
2022-11-03 12:56:26,177 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 21.0152, Test RMSE: 35.8100, Test MAPE: 0.0873
2022-11-03 12:56:26,180 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 21.3745, Test RMSE: 36.5240, Test MAPE: 0.0892
2022-11-03 12:56:26,182 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 21.6551, Test RMSE: 37.0576, Test MAPE: 0.0901
2022-11-03 12:56:26,185 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 21.9236, Test RMSE: 37.5292, Test MAPE: 0.0913
2022-11-03 12:56:26,187 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 22.1922, Test RMSE: 38.0034, Test MAPE: 0.0926
2022-11-03 12:56:26,190 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 22.5857, Test RMSE: 38.5200, Test MAPE: 0.0942
2022-11-03 12:56:26,218 - easytorch-training - INFO - Result <test>: [test_time: 3.91 (s), test_MAE: 20.5028, test_RMSE: 34.6671, test_MAPE: 0.0856]
2022-11-03 12:56:26,243 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_095.pt saved
Thanks very much for reporting this issue, and I will update the results.
from basicts.
The results have been updated in the commit 08997c2.
from basicts.
After checking the training log, ST-Norm's overall MAPE is indeed problematic. The correct number is 8.56%. Here are the full training results:
2022-11-03 12:55:25,075 - easytorch-training - INFO - Epoch 95 / 100 2022-11-03 12:56:18,227 - easytorch-training - INFO - Result <train>: [train_time: 53.15 (s), lr: 5.00e-04, train_MAE: 18.2307, train_RMSE: 30.4419, train_MAPE: 0.0808 ] 2022-11-03 12:56:18,228 - easytorch-training - INFO - Start validation. 2022-11-03 12:56:22,276 - easytorch-training - INFO - Result <val>: [val_time: 4.05 (s), val_MAE: 19.8844, val_RMSE: 32.5213, val_MAPE: 0.0858] 2022-11-03 12:56:22,307 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_best_val_MAE.pt saved 2022-11-03 12:56:26,161 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 17.2057, Test RMSE: 27.4493, Test MAPE: 0.0720 2022-11-03 12:56:26,164 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 18.3410, Test RMSE: 29.7707, Test MAPE: 0.0766 2022-11-03 12:56:26,166 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 19.1426, Test RMSE: 31.5075, Test MAPE: 0.0798 2022-11-03 12:56:26,169 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 19.7441, Test RMSE: 32.8585, Test MAPE: 0.0830 2022-11-03 12:56:26,172 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 20.2191, Test RMSE: 34.0451, Test MAPE: 0.0849 2022-11-03 12:56:26,174 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 20.6346, Test RMSE: 34.9905, Test MAPE: 0.0861 2022-11-03 12:56:26,177 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 21.0152, Test RMSE: 35.8100, Test MAPE: 0.0873 2022-11-03 12:56:26,180 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 21.3745, Test RMSE: 36.5240, Test MAPE: 0.0892 2022-11-03 12:56:26,182 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 21.6551, Test RMSE: 37.0576, Test MAPE: 0.0901 2022-11-03 12:56:26,185 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 21.9236, Test RMSE: 37.5292, Test MAPE: 0.0913 2022-11-03 12:56:26,187 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 22.1922, Test RMSE: 38.0034, Test MAPE: 0.0926 2022-11-03 12:56:26,190 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 22.5857, Test RMSE: 38.5200, Test MAPE: 0.0942 2022-11-03 12:56:26,218 - easytorch-training - INFO - Result <test>: [test_time: 3.91 (s), test_MAE: 20.5028, test_RMSE: 34.6671, test_MAPE: 0.0856] 2022-11-03 12:56:26,243 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_095.pt saved
Thanks very much for reporting this issue, and I will update the results.
Thanks for your quick response!
I also found Graph WaveNet's MAPE at Horizon 6 (8.24%) is extremely low(even smaller than which at Horizon 3 for 8.63%).
There might be similar problem for DCRNN's MAPE at Horizon 3 (6.86%), which is better than MTGNN's performance(6.89%). However, DCRNN's performance is much worse than MTGNN under other Horizons. I can't reproduce it either.
from basicts.
Thanks very much for your further suggestions.
The performance of Graph WaveNet at Horizon 6 should be 9.24%, instead of 8.24%.
Here is the training log:
2022-11-01 17:27:27,131 - easytorch-training - INFO - Epoch 180 / 200
2022-11-01 17:27:37,069 - easytorch-training - INFO - Result <train>: [train_time: 9.94 (s), lr: 2.50e-04, train_MAE: 14.0349, train_RMSE: 23.2042, train_MAPE: 0.0936]
2022-11-01 17:27:37,070 - easytorch-training - INFO - Start validation.
2022-11-01 17:27:38,466 - easytorch-training - INFO - Result <val>: [val_time: 1.40 (s), val_MAE: 14.6842, val_RMSE: 23.8004, val_MAPE: 0.1044]
2022-11-01 17:27:38,491 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_best_val_MAE.pt saved
2022-11-01 17:27:39,932 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 12.4887, Test RMSE: 19.5742, Test MAPE: 0.0803
2022-11-01 17:27:39,933 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 13.0537, Test RMSE: 20.7378, Test MAPE: 0.0835
2022-11-01 17:27:39,934 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 13.5009, Test RMSE: 21.6306, Test MAPE: 0.0863
2022-11-01 17:27:39,935 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 13.8442, Test RMSE: 22.3552, Test MAPE: 0.0882
2022-11-01 17:27:39,936 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 14.1508, Test RMSE: 22.9266, Test MAPE: 0.0902
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 14.4118, Test RMSE: 23.4517, Test MAPE: 0.0924
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 14.6821, Test RMSE: 23.9416, Test MAPE: 0.0937
2022-11-01 17:27:39,938 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 14.9098, Test RMSE: 24.3487, Test MAPE: 0.0953
2022-11-01 17:27:39,939 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 15.1343, Test RMSE: 24.6971, Test MAPE: 0.0969
2022-11-01 17:27:39,940 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 15.3347, Test RMSE: 25.0472, Test MAPE: 0.0983
2022-11-01 17:27:39,941 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 15.5267, Test RMSE: 25.3649, Test MAPE: 0.0992
2022-11-01 17:27:39,942 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 15.8140, Test RMSE: 25.7723, Test MAPE: 0.1011
2022-11-01 17:27:39,946 - easytorch-training - INFO - Result <test>: [test_time: 1.45 (s), test_MAE: 14.4043, test_RMSE: 23.3941, test_MAPE: 0.0921]
2022-11-01 17:27:39,970 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_180.pt saved
2022-11-01 17:27:39,977 - easytorch-training - INFO - The estimated training finish time is 2022-11-01 17:31:55
The performance of DCRNN on METR-LA is correct. DCRNN outperforms Graph WaveNet and MTGNN at short-term horizons. Here is the training log:
2022-11-02 12:24:40,516 - easytorch-training - INFO - Epoch 94 / 100
2022-11-02 12:28:57,768 - easytorch-training - INFO - Result <train>: [train_time: 257.25 (s), lr: 1.00e-06, train_MAE: 2.7032, train_RMSE: 5.5099, train_MAPE: 0.0711]
2022-11-02 12:28:57,768 - easytorch-training - INFO - Start validation.
2022-11-02 12:29:06,105 - easytorch-training - INFO - Result <val>: [val_time: 8.34 (s), val_MAE: 2.7504, val_RMSE: 5.4149, val_MAPE: 0.0765]
2022-11-02 12:29:06,115 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_best_val_MAE.pt saved
2022-11-02 12:29:21,572 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 2.1762, Test RMSE: 3.7696, Test MAPE: 0.0519
2022-11-02 12:29:21,574 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 2.4718, Test RMSE: 4.5959, Test MAPE: 0.0615
2022-11-02 12:29:21,575 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 2.6695, Test RMSE: 5.1683, Test MAPE: 0.0686
2022-11-02 12:29:21,576 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 2.8237, Test RMSE: 5.6119, Test MAPE: 0.0745
2022-11-02 12:29:21,577 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 2.9533, Test RMSE: 5.9699, Test MAPE: 0.0797
2022-11-02 12:29:21,578 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 3.0657, Test RMSE: 6.2738, Test MAPE: 0.0842
2022-11-02 12:29:21,579 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 3.1666, Test RMSE: 6.5409, Test MAPE: 0.0882
2022-11-02 12:29:21,580 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 3.2560, Test RMSE: 6.7684, Test MAPE: 0.0917
2022-11-02 12:29:21,581 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 3.3370, Test RMSE: 6.9717, Test MAPE: 0.0950
2022-11-02 12:29:21,582 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 3.4080, Test RMSE: 7.1505, Test MAPE: 0.0978
2022-11-02 12:29:21,583 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 3.4761, Test RMSE: 7.3172, Test MAPE: 0.1006
2022-11-02 12:29:21,584 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 3.5426, Test RMSE: 7.4758, Test MAPE: 0.1032
2022-11-02 12:29:21,592 - easytorch-training - INFO - Result <test>: [test_time: 15.48 (s), test_MAE: 3.0289, test_RMSE: 6.2336, test_MAPE: 0.0831]
2022-11-02 12:29:21,607 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_094.pt saved
from basicts.
I will update the results later.
Any further questions are welcome!
from basicts.
The results have been updated, and I have fixed a typo in DCRNN configs on metr-la and pems03 datasets. Now you should can reproduce the results.
from basicts.
Thanks for that! That would solved my problem.
Looking forward you checking the metrics at a proper time.
from basicts.
You are welcome. I have tested DCRNN on the mete-la dataset, and the results are the same as mentioned above.
Do you have any other concerns about results and metrics?
from basicts.
I have no more questions for now. Thanks again for your excellent work!
from basicts.
You are welcome! If you have any other questions, please feel free to create a new issue, and I will reply quickly.
from basicts.
Related Issues (20)
- some issue about model performance HOT 1
- thanks HOT 1
- AGCRN,PEMS08数据集的配置问题 HOT 1
- 请问如何使用inference程序,需要改哪些部分 HOT 15
- Add contributors. HOT 16
- 请问如何修改学习率衰减器 HOT 1
- 修改数据归一化方式。 HOT 1
- MS (multivariate -> univariate)? HOT 2
- 关于数据输入和输出特征个数的问题? HOT 2
- dataloader的droplast问题 HOT 2
- STGCN 模型不能跑第二次 HOT 1
- implementation of spatial indistinguishability HOT 1
- 能否支持early stop? HOT 1
- 您好,请教一下模型训练好后如何获取预测值与真实值的数据 HOT 5
- 怎么快速的修改输入输出长度,完成不同实验? HOT 1
- 关于训练速度 HOT 3
- 如何多卡训练 HOT 1
- Cannot run inference.py on VMWare 2016 workstation. HOT 20
- 复现代码中的参数设置问题 HOT 3
- 如何修改数据集预测的时间长度,我修改了生成数据集的参数和模型的参数后存在如下问题 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from basicts.