sxjscience / automl_multimodal_benchmark Goto Github PK

Hi, thanks for the great benchmark!

I wanted to ask about something I noticed -- this seems to be a bug, but not quite positive. It looks like the 'price' column in the airbnb dataset, which is used to generate the 'price_label' column (which is a binned prediction target) is included in the input features. This means that 'price' can be used to predict 'price_label' -- which should allow for perfect accuracy for a moderately-flexible classifier from the price column alone.

https://github.com/sxjscience/automl_multimodal_benchmark/blob/c3478b1f595eeb593f367de42196a31dbf223e1d/multimodal_text_benchmark/scripts/data_processing/process_airbnb.py

Is this intended? And if so, maybe I am also unclear why the baseline models in the paper aren't getting 100% on this dataset?