josteinstraume / deduplication Goto Github PK

##Business Problem## This retail gaint deals with internal structured data pertainig to customer biz addressess, SKU product codes & description and product heirarchy descriptions for each SKU. It buys/collects data from external vendors to augument its marketing reach and in the process, the external datasets have thier owm nomenclature. i.e. fields and descriptions are bit different but pointing to same data row/record. Hence data needs to be matched accurately to get meaningful analytics. Problem needs to address 3 datasets namely Address, SKU and Product Hierarchy (classifications).

##Proposed Solution and ShinyR PoC Web App## [ShinyR Web App code is in the above folders submitted on Apr 2016 to dextra.sg platform and also hosted in RStudio's cloud app platform ShinyApps.io]

Data blending in this context is basically a subset of Record Linkage and De-Duplication problem. Given the constraints of data which can’t be fitted into a definitive way of comparing individual fields or use them to narrow the amount of search is a unique problem to be solved.

Address fields are free form data which can’t leverage ‘Blocking’ which is the reduction of the amount of data pairs through focusing on specified agreement patterns. Whereas SKU dataset can leverage this feature to a limited extend and Product Hierarchy utilizes a pure simple weighted distance ensemble.

Solution aspects are discussed against each dataset with PoC wep app hosted seperately.

Team

Nachi Nachiappan, a data practioner, has worked in enterprise IT industry for 15 years and possess diverse business domain expertise including exposure to retail & financial industries. Data science being my passion, I keep dabbling on such projects and I hope this proposal is worth that effort to positively make some impact on data blending in enterprise and public realm.

1. Methodology and Effectiveness

2. Accuracy Measurement

3. Implementation & Adoption

4. Conclusion

5. References & Consideration

Data blending for Unilever’s problem broadly involves the following steps:

1. Data Preparation

2. Basic Stochastic Distance Measurement / Blocking

3. Unsupervised Learning

4. Ensemble of Distance Measurements

Above process is used fully and sparingly based on each dataset properties and blending requirements

##1. Methodology and Effectiveness##

Data Preparation:

Basic Stochastic Distance Measurement / Blocking:

Unsupervised Learning

Ensemble of Distance Measurements

How does it solve the problem?

Basic data prep, distance measurements and unsupervised learning provide a base set upon which reduction is performed using ‘Ensemble Distance Measurement’ to achieve the results. No supervised learning is used as the dataset lacks huge training sample and better domain knowledge. Whereas our strategy is a generic one and can be applied to any dataset without prior domain knowledge but can be optimized and fine-tuned further with more domain knowledge and also can explore supervised training if that is possible.

What makes the solution robust?

Huge datasets may have an issue on mxn unrestricted distance calculation but may not be heavy compute cost given the cloud compute is very low.

How does the solution handle various cases of mapping?

A full-fledged solution may need to match the level of functionality as open source FEBRL application which has all permutations. Due to time constraint mapping is performed in the code and not exposed as a generic interface and can be co-developed if this solution/algorithm is selected.

##2. Accuracy Measurement##

SKU :

Product Hierarchy:

##3. Implementation & Adoption##

Prototypes hosted @

You can access the apps as here:

How easily the solution can be implemented?

It is a simple yet powerful solution strategy using standard record linkage and de-duplication concepts and distance metrics. Can be implemented as Shiny.R app and can be deployed for enterprise use. This prototype uses Shiny.R to demonstrate the algorithm interface and utilizes various R packages like RecordLinkage, stringDist, BioString, etc.

Any risk in implementation?

Every project and endeavor has some risk and here are the things to consider:

1. More data points to test the algorithm’s consistency

2. Does data structure change a lot – need to be studied to this tailored algorithm

3. Huge dataset performance to be measured

4. Data matching ambiguity needs to have a proper user interface to mark proper matching and store it for future supervised or unsupervised algorithms.

Experimental Frameworks Considered:

1. Simple Distance measurements won’t suffice

2. Supervised were considered but training dataset is unavailable

3. Unsupervised – Kmeans and Bclust

4. Stochastic – epiWeights and emiWeights

5. Combination of Distances – Simple Distance Ensemble

##4. Conclusion##

For this retail giant's Data Blending – a simple, powerful yet uncomplicated algorithm that used unsupervised reduction using Distance Ensemble is proposed which gives consistent and good results.

##5. References & Consideration##

Books

Data Matching – Springer

High Performance Record Linkage