wenjiedu / awesome_imputation Goto Github PK

谢谢

Dear Authors,

Thank you for your invaluable contributions to this repository. I am currently exploring the field of time series imputation and have encountered some aspects regarding the evaluation protocols that I believe could benefit from further discussion.

1. Dataset Splitting: The choice to split the dataset chronologically is well-suited for time series forecasting to prevent data leakage. However, for imputation tasks where the goal is to address the missingness in available data, such splitting may not be necessary. Given that the primary concern in imputation is dealing with inherently missing data, a non-chronological split might be more appropriate as it reflects real-world scenarios where all available data is subject to imputation, instead of the recent ones.
2. Evaluation Comparisons: The evaluation process raises somewhat questions about fairness and consistency across different methods. We compare for instance the Transformer and mean imputer. While the Transformer model is assessed using test data, the approach for evaluating a mean imputer remains unclear. Should the mean imputer also have access to the test data since the non-missing data in the test data should also be available in model serving? There are two options:

• training the mean imputer on the train-eval set is unfair since the non-missing data in the test set should be available for the mean imputer too, which does not cause leakage and has been exploited by nn models.
• training the mean imputer exclusively on the test set does not leverage the potentially informative train-eval sets, which seems equally unfair.

In view of these points, I suggest the following:

• For generalized imputation methods like those in HyperImpute, should we maintain merely the unavailability of missing values in the test set while considering the rest of the data as usable (including the non-missing values in the test data, the train and eval data)?
• Could we use a non-chronological train-val-test split, given that in practical applications, the emphasis is on imputing the entire dataset rather than the recent months? More importantly, in the case of missing value imputation, the non-missing data is often unavailable (kindly see the protocol of HyperImpute for reference)

I look forward to your insights and any suggestions you might have on aligning the evaluation framework with real-world imputation tasks.

Best regards,

Hao