apify-projects / product-mapping-engine Goto Github PK

When we find a pair of products, we should stash that result in some cache and use the cached result whenever we can in order to conserve computational resources.

Enable to specify training and executing by parameters to facilitate search for optimal parameters.

Compute similarities based on the multiple datasets with product pairs.

Maybe we should get a list of currencies and add them all to the units file?

But this after we will have stable version of code.

Currently, we basically only use price as a filter. This however won't work in case when the price is not there, and moreover, will still generate too many candidate product pairs to go through. We need to create smarter filters before we deploy the system in practice.

For example CNN based or other NN based approaches.

More advanced filters (e.g. filtering according to the descriptive words) require preprocessing to be done before they can be applied. Therefore, preprocessing has to be done before that portion of filtering. However, to make sure we are not preprocessing products that cannot possibly be paired with anything, the most basic filters that do not require any preprocessing (e.g. the price-based filtering) should be done beforehand.
Also make sure we are not preprocessing any product multiple times, which is the case now.

Since it is possible that in some datasets, some data columns will be missing both altogether or just be missing in some rows, we need to make sure the engine can deal with such cases without failing.
To do so properly, we also need to make sure there is a difference between the available texts (for instance) not matching and the texts not being there at all. To do this, lets shift every single probability we are calculating to the <-1,1> range, with values close to -1 signifying total mismatch, values around 0 signifying either ambiguity or the data not being present and values close to 1 meaning total match.

Create automated tests to verify whether new changes did not break anything else.

Currently, we are only using image hashing to compare images. However, it only cares about the visual features, but not about the semantics of what is depicted on the image.

Currently, parameter names are simply compared directly, but that will almost never work - e.g. "RAM memory" vs "Operational memory (RAM)". We need to devise a smarter way to compare them.

Currently, our filtering requires the products to have a price specified, but that isn't necessarily true, it might not be sold yet for instance, but we still want to be able to work with it.