Code Monkey home page Code Monkey logo

Comments (10)

zezhishao avatar zezhishao commented on June 20, 2024

After checking the training log, ST-Norm's overall MAPE is indeed problematic.
The correct number is 8.56%.
Here are the full training results:

2022-11-03 12:55:25,075 - easytorch-training - INFO - Epoch 95 / 100
2022-11-03 12:56:18,227 - easytorch-training - INFO - Result <train>: [train_time: 53.15 (s), lr: 5.00e-04, train_MAE: 18.2307, train_RMSE: 30.4419, train_MAPE: 0.0808 ]
2022-11-03 12:56:18,228 - easytorch-training - INFO - Start validation.
2022-11-03 12:56:22,276 - easytorch-training - INFO - Result <val>: [val_time: 4.05 (s), val_MAE: 19.8844, val_RMSE: 32.5213, val_MAPE: 0.0858]
2022-11-03 12:56:22,307 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_best_val_MAE.pt saved
2022-11-03 12:56:26,161 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 17.2057, Test RMSE: 27.4493, Test MAPE: 0.0720
2022-11-03 12:56:26,164 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 18.3410, Test RMSE: 29.7707, Test MAPE: 0.0766
2022-11-03 12:56:26,166 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 19.1426, Test RMSE: 31.5075, Test MAPE: 0.0798
2022-11-03 12:56:26,169 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 19.7441, Test RMSE: 32.8585, Test MAPE: 0.0830
2022-11-03 12:56:26,172 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 20.2191, Test RMSE: 34.0451, Test MAPE: 0.0849
2022-11-03 12:56:26,174 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 20.6346, Test RMSE: 34.9905, Test MAPE: 0.0861
2022-11-03 12:56:26,177 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 21.0152, Test RMSE: 35.8100, Test MAPE: 0.0873
2022-11-03 12:56:26,180 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 21.3745, Test RMSE: 36.5240, Test MAPE: 0.0892
2022-11-03 12:56:26,182 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 21.6551, Test RMSE: 37.0576, Test MAPE: 0.0901
2022-11-03 12:56:26,185 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 21.9236, Test RMSE: 37.5292, Test MAPE: 0.0913
2022-11-03 12:56:26,187 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 22.1922, Test RMSE: 38.0034, Test MAPE: 0.0926
2022-11-03 12:56:26,190 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 22.5857, Test RMSE: 38.5200, Test MAPE: 0.0942
2022-11-03 12:56:26,218 - easytorch-training - INFO - Result <test>: [test_time: 3.91 (s), test_MAE: 20.5028, test_RMSE: 34.6671, test_MAPE: 0.0856]
2022-11-03 12:56:26,243 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_095.pt saved

Thanks very much for reporting this issue, and I will update the results.

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

The results have been updated in the commit 08997c2.

from basicts.

HankLiu10 avatar HankLiu10 commented on June 20, 2024

After checking the training log, ST-Norm's overall MAPE is indeed problematic. The correct number is 8.56%. Here are the full training results:

2022-11-03 12:55:25,075 - easytorch-training - INFO - Epoch 95 / 100
2022-11-03 12:56:18,227 - easytorch-training - INFO - Result <train>: [train_time: 53.15 (s), lr: 5.00e-04, train_MAE: 18.2307, train_RMSE: 30.4419, train_MAPE: 0.0808 ]
2022-11-03 12:56:18,228 - easytorch-training - INFO - Start validation.
2022-11-03 12:56:22,276 - easytorch-training - INFO - Result <val>: [val_time: 4.05 (s), val_MAE: 19.8844, val_RMSE: 32.5213, val_MAPE: 0.0858]
2022-11-03 12:56:22,307 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_best_val_MAE.pt saved
2022-11-03 12:56:26,161 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 17.2057, Test RMSE: 27.4493, Test MAPE: 0.0720
2022-11-03 12:56:26,164 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 18.3410, Test RMSE: 29.7707, Test MAPE: 0.0766
2022-11-03 12:56:26,166 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 19.1426, Test RMSE: 31.5075, Test MAPE: 0.0798
2022-11-03 12:56:26,169 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 19.7441, Test RMSE: 32.8585, Test MAPE: 0.0830
2022-11-03 12:56:26,172 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 20.2191, Test RMSE: 34.0451, Test MAPE: 0.0849
2022-11-03 12:56:26,174 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 20.6346, Test RMSE: 34.9905, Test MAPE: 0.0861
2022-11-03 12:56:26,177 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 21.0152, Test RMSE: 35.8100, Test MAPE: 0.0873
2022-11-03 12:56:26,180 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 21.3745, Test RMSE: 36.5240, Test MAPE: 0.0892
2022-11-03 12:56:26,182 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 21.6551, Test RMSE: 37.0576, Test MAPE: 0.0901
2022-11-03 12:56:26,185 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 21.9236, Test RMSE: 37.5292, Test MAPE: 0.0913
2022-11-03 12:56:26,187 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 22.1922, Test RMSE: 38.0034, Test MAPE: 0.0926
2022-11-03 12:56:26,190 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 22.5857, Test RMSE: 38.5200, Test MAPE: 0.0942
2022-11-03 12:56:26,218 - easytorch-training - INFO - Result <test>: [test_time: 3.91 (s), test_MAE: 20.5028, test_RMSE: 34.6671, test_MAPE: 0.0856]
2022-11-03 12:56:26,243 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_095.pt saved

Thanks very much for reporting this issue, and I will update the results.

Thanks for your quick response!

I also found Graph WaveNet's MAPE at Horizon 6 (8.24%) is extremely low(even smaller than which at Horizon 3 for 8.63%).

There might be similar problem for DCRNN's MAPE at Horizon 3 (6.86%), which is better than MTGNN's performance(6.89%). However, DCRNN's performance is much worse than MTGNN under other Horizons. I can't reproduce it either.

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

Thanks very much for your further suggestions.
The performance of Graph WaveNet at Horizon 6 should be 9.24%, instead of 8.24%.
Here is the training log:

2022-11-01 17:27:27,131 - easytorch-training - INFO - Epoch 180 / 200
2022-11-01 17:27:37,069 - easytorch-training - INFO - Result <train>: [train_time: 9.94 (s), lr: 2.50e-04, train_MAE: 14.0349, train_RMSE: 23.2042, train_MAPE: 0.0936]
2022-11-01 17:27:37,070 - easytorch-training - INFO - Start validation.
2022-11-01 17:27:38,466 - easytorch-training - INFO - Result <val>: [val_time: 1.40 (s), val_MAE: 14.6842, val_RMSE: 23.8004, val_MAPE: 0.1044]
2022-11-01 17:27:38,491 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_best_val_MAE.pt saved
2022-11-01 17:27:39,932 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 12.4887, Test RMSE: 19.5742, Test MAPE: 0.0803
2022-11-01 17:27:39,933 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 13.0537, Test RMSE: 20.7378, Test MAPE: 0.0835
2022-11-01 17:27:39,934 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 13.5009, Test RMSE: 21.6306, Test MAPE: 0.0863
2022-11-01 17:27:39,935 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 13.8442, Test RMSE: 22.3552, Test MAPE: 0.0882
2022-11-01 17:27:39,936 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 14.1508, Test RMSE: 22.9266, Test MAPE: 0.0902
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 14.4118, Test RMSE: 23.4517, Test MAPE: 0.0924
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 14.6821, Test RMSE: 23.9416, Test MAPE: 0.0937
2022-11-01 17:27:39,938 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 14.9098, Test RMSE: 24.3487, Test MAPE: 0.0953
2022-11-01 17:27:39,939 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 15.1343, Test RMSE: 24.6971, Test MAPE: 0.0969
2022-11-01 17:27:39,940 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 15.3347, Test RMSE: 25.0472, Test MAPE: 0.0983
2022-11-01 17:27:39,941 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 15.5267, Test RMSE: 25.3649, Test MAPE: 0.0992
2022-11-01 17:27:39,942 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 15.8140, Test RMSE: 25.7723, Test MAPE: 0.1011
2022-11-01 17:27:39,946 - easytorch-training - INFO - Result <test>: [test_time: 1.45 (s), test_MAE: 14.4043, test_RMSE: 23.3941, test_MAPE: 0.0921]
2022-11-01 17:27:39,970 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_180.pt saved
2022-11-01 17:27:39,977 - easytorch-training - INFO - The estimated training finish time is 2022-11-01 17:31:55

The performance of DCRNN on METR-LA is correct. DCRNN outperforms Graph WaveNet and MTGNN at short-term horizons. Here is the training log:

2022-11-02 12:24:40,516 - easytorch-training - INFO - Epoch 94 / 100
2022-11-02 12:28:57,768 - easytorch-training - INFO - Result <train>: [train_time: 257.25 (s), lr: 1.00e-06, train_MAE: 2.7032, train_RMSE: 5.5099, train_MAPE: 0.0711]
2022-11-02 12:28:57,768 - easytorch-training - INFO - Start validation.
2022-11-02 12:29:06,105 - easytorch-training - INFO - Result <val>: [val_time: 8.34 (s), val_MAE: 2.7504, val_RMSE: 5.4149, val_MAPE: 0.0765]
2022-11-02 12:29:06,115 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_best_val_MAE.pt saved
2022-11-02 12:29:21,572 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 2.1762, Test RMSE: 3.7696, Test MAPE: 0.0519
2022-11-02 12:29:21,574 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 2.4718, Test RMSE: 4.5959, Test MAPE: 0.0615
2022-11-02 12:29:21,575 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 2.6695, Test RMSE: 5.1683, Test MAPE: 0.0686
2022-11-02 12:29:21,576 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 2.8237, Test RMSE: 5.6119, Test MAPE: 0.0745
2022-11-02 12:29:21,577 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 2.9533, Test RMSE: 5.9699, Test MAPE: 0.0797
2022-11-02 12:29:21,578 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 3.0657, Test RMSE: 6.2738, Test MAPE: 0.0842
2022-11-02 12:29:21,579 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 3.1666, Test RMSE: 6.5409, Test MAPE: 0.0882
2022-11-02 12:29:21,580 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 3.2560, Test RMSE: 6.7684, Test MAPE: 0.0917
2022-11-02 12:29:21,581 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 3.3370, Test RMSE: 6.9717, Test MAPE: 0.0950
2022-11-02 12:29:21,582 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 3.4080, Test RMSE: 7.1505, Test MAPE: 0.0978
2022-11-02 12:29:21,583 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 3.4761, Test RMSE: 7.3172, Test MAPE: 0.1006
2022-11-02 12:29:21,584 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 3.5426, Test RMSE: 7.4758, Test MAPE: 0.1032
2022-11-02 12:29:21,592 - easytorch-training - INFO - Result <test>: [test_time: 15.48 (s), test_MAE: 3.0289, test_RMSE: 6.2336, test_MAPE: 0.0831]
2022-11-02 12:29:21,607 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_094.pt saved

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

I will update the results later.
Any further questions are welcome!

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

The results have been updated, and I have fixed a typo in DCRNN configs on metr-la and pems03 datasets. Now you should can reproduce the results.

from basicts.

HankLiu10 avatar HankLiu10 commented on June 20, 2024

Thanks for that! That would solved my problem.

Looking forward you checking the metrics at a proper time.

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

You are welcome. I have tested DCRNN on the mete-la dataset, and the results are the same as mentioned above.
Do you have any other concerns about results and metrics?

from basicts.

HankLiu10 avatar HankLiu10 commented on June 20, 2024

I have no more questions for now. Thanks again for your excellent work!

from basicts.

zezhishao avatar zezhishao commented on June 20, 2024

You are welcome! If you have any other questions, please feel free to create a new issue, and I will reply quickly.

from basicts.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.