This repository contains code and extensive prompt examples to reproduce and extend the experiments in our paper "Entity Matching using Large Language Models".
-
Requirements
-
Building the conda environment
To build the exact conda environment used for the experiments, navigate to the project root folder where the file llm_match.yml is located and run
conda env create -f llm_match.yml -
Jupyter Notebooks
The code is split into three main notebooks:
- LLMMatch_* - These notebooks contains the code for running the different tasks in zero-shot and with rules either against the OpenAI API or locally for the open source LLMs.
- LLMMatch_*-context - Same as 1 but specifically for the in-context experiments
- LLMMatch_*-finetune - These notebooks contains the code for running the different tasks in zero-shot configuration using the ChatGPT models we fine-tuned.
- Optional Notebooks for reproduction purposes
- LLMMatch_*_task - This optional notebook contains the code for downsampling the original WDC Products 80% corner-case set and the other benchmarks.
- prep_em_tasks - This notebook contains the code for preparing the tasks (prompt designs) as json files for usage in the MatchGPT notebooks.
- prep_finetune_* - Code for preparing the datasets for fine-tuning using the format required by OpenAI.
- fine-tune-* - The code for fine-tuning the OpenAI ChatGPT models that are used in 3.
- Detailed Prompt Designs
Below, we provide examples of the different prompt designs. For simplicity we provide the "title-only" versions of the prompts as well as only single positive and negative examples for the in-context learning prompts.
general-complex-free
Do the two entity descriptions refer to the same real-world entity?
Entity 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Entity 2: 'DYMO D1 Tape 12mm x 7m'
general-simple-free
Do the two entity descriptions match?
Entity 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Entity 2: 'DYMO D1 Tape 12mm x 7m'
general-complex-force
Do the two entity descriptions refer to the same real-world entity? Answer with 'Yes' if they do and 'No' if they do not.
Entity 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Entity 2: 'DYMO D1 Tape 12mm x 7m'
general-simple-force
Do the two entity descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Entity 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Entity 2: 'DYMO D1 Tape 12mm x 7m'
domain-complex-free (Product)
Do the two product descriptions refer to the same real-world product?
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-simple-free (Product)
Do the two product descriptions match?
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-complex-force (Product)
Do the two product descriptions refer to the same real-world product? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-simple-force (Product)
Do the two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-complex-free (Bibliographic)
Do the two publications refer to the same real-world product?
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-simple-free (Bibliographic)
Do the two publications match?
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-complex-force (Bibliographic)
Do the two publications refer to the same real-world product? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
domain-simple-force (Bibliographic)
Do the two publications match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
Narayan-complex
Product A is 'DYMO D1 - Roll (1.9cm x 7m)'. Product B is 'DYMO D1 Tape 12mm x 7m'.
Are Product A and Product B equivalent?
Narayan-simple
Product A is 'DYMO D1 - Roll (1.9cm x 7m)'. Product B is 'DYMO D1 Tape 12mm x 7m'.
Are Product A and Product B the same?
Fewshot-random/handpicked/related (up to 5 positive and negative examples at once)
User: Do the following two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: ‘DYMO D1 19 mm x 7 m’
Product 2: ‘Dymo D1 (19mm x 7m – BoW)’
AI: Yes.
User: Do the following two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: ‘DYMO D1 Tape 24mm’
Product 2: ‘Dymo D1 19mm x 7m’
AI: No.
User: Do the following two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
Hand-written rules (Product)
The following rules regarding product features need to be observed:
1. The brand of matching products must be the same if available.
2. Model names of matching products must be the same if available.
3. Model numbers of matching products must be the same if available.
4. Additional features of matching products must be the same if available.
5. Matching attributes may not have the exact same surface form due to different case, typos, value formats.
6. If an attribute is missing for one description, it is likely still a match if the existing attributes match.
Do the following two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
Hand-written rules (Bibliographic)
The following rules regarding publication features need to be observed:
1. The authors must be the same.
2. The title must be the same.
3. The venue should be the same but could differ.
4. The year should be the same but could differ.
5. Matching attributes may not have the exact same surface form due to different case, typos, value formats, etc.
6. If an attribute is missing for one publication, it is likely still a match if the existing attributes match.
Do the following two publications match? Answer with 'Yes' if they do and 'No' if they do not.
Publication 1: 's lee , t ling further improvements on integrity constraint checking for stratifiable deductive databases vldb 1996'
Publication 2: 'sy lee , tw ling further improvements on integrity constraint checking for stratifiable deductive databases vldb , 1996.0'
Learned Rules
The following rules regarding product features need to be observed:
1. The product titles match if they both refer to the same core product, which means they have the same primary brand and product model. For example, if two products both mention 'SANDISK EXTREME PRO SDHC 32GB 300MB/S UHS-II U3', they are considered as a match.
2. Order of words and usage of separators (like hyphen, comma, slash, parentheses) does not matter. For example, 'Dymo 53718 Black On Yellow - 24mm' and 'Dymo 24mm Black On Yellow D1 Tape (53718)' are considered as a match.
3. The product titles match if they both contain the same key information, even if one product title contains additional details. For example, 'Intel Core i7-8700K 6/12 3.7/4.7GHz' and 'Intel Core i7 8700K (Base:3.70GHz, Turbo:4.70GHz / 12MB / LGA1151 / 6 Core / 95W / Fully Unlocked / Without Heatsink/Fan / Coffee Lake)' are a match, because the key information (brand, product line, model, and key specs) are the same.
4. Exact matching of all specifications is not necessary. Titles match as long as the critical specs like model number, series, and brand are the same. For example, 'Gigabyte NVIDIA GeForce GTX 1650 4GB D6 OC Turing Graphics Card' and 'GigaByte GeForce GTX 1650 D6 Windforce OC 4G' are a match, despite the difference in some details.
5. The product titles do not match if they refer to different models of the same brand. For example, 'RAM Corsair ValueSelect DDR4 2133MHz 1x8Go' and 'CORSAIR New 8gb (1x8gb) Ddr4 2666mhz Vengeance CMK8GX4M1A2666C16' are not a match because although they are both Corsair RAMs, they belong to different product lines and have different specifications.
6. The product titles do not match if they refer to different capacity or performance levels of the same product line. For example, 'Samsung SSD 970 EVO Plus 250GB' and 'Samsung 970 EVO SSD M.2 2280 - 500GB' are not a match, because the storage capacity differs.
7. The product titles do not match if they refer to different types of a product from the same brand. For example, '128GB Pendrive SanDisk Extreme PRO SSD USB 3.1 420MB/s' and 'SanDisk 128GB Extreme Pro SDXC 150MB/s V-30 UHS-1 U3 Memory Card' are not a match, even though they are both storage devices by SanDisk, because one is a USB flash drive and the other is an SD card.
8. The product titles do not match if they refer to different models of the same type of product, even from the same manufacturer. For example, 'Cooler Master Masterliquid ML120R RGB All-in-one Watercooler' and 'Cooler Master MasterLiquid ML120L V2 RGB All In One Liquid CPU Cooler' are not a match, because the model numbers (ML120R and ML120L V2) are different.
Do the following two product descriptions match? Answer with 'Yes' if they do and 'No' if they do not.
Product 1: 'DYMO D1 - Roll (1.9cm x 7m)'
Product 2: 'DYMO D1 Tape 12mm x 7m'
- Examples of Matches and Non-Matches
Below, we provide the 20 handpicked examples of matches and non-matches that were used for the experiments and learning the rules. You can use them for simple copy/paste experimentation with your favorite LLM.
Matches:
Product 1: 'SANDISK EXTREME PRO SDHC 32GB 300MB/S UHS-II U3' Product 2: 'Sandisk SDXC card Extreme Pro UHS-II, 32gb, 300mbps'
Product 1: 'Dymo 53718 Black On Yellow - 24mm' Product 2: 'Dymo 24mm Black On Yellow D1 Tape (53718)'
Product 1: 'DS-7216HQHI-K1 Hikvision 16 cs. TurboHD DVR' Product 2: 'Hikvision DS-7216HQHI-K1 Turbo HD DVR'
Product 1: 'APCRBC133APC Replacement Battery Cartridge #133' Product 2: 'APC RBC133 Replacement Battery Cartridge'
Product 1: 'Gigabyte NVIDIA GeForce GTX 1650 4GB D6 OC Turing Graphics Card' Product 2: 'GigaByte GeForce GTX 1650 D6 Windforce OC 4G'
Product 1: '1TB 970 EVO 2280, 3400 / 2500 MB/s, V-NAND 3-bit MLC, PCIe 3.0 x4 NVMe, M.2 SSD' Product 2: 'Samsung NVMe SSD 970 Evo 1TB, M.2'
Product 1: 'Intel Core i7-8700K 6/12 3.7/4.7GHz' Product 2: 'Intel Core i7 8700K (Base:3.70GHz, Turbo:4.70GHz / 12MB / LGA1151 / 6 Core / 95W / Fully Unlocked / Without Heatsink/Fan / Coffee Lake)'
Product 1: 'Cooler Master MasterLiquid ML120L RGB V2 AIO Liquid CPU Cooler MLW-D12M-A18PC-R2' Product 2: 'Cooler Master MasterLiquid ML120L V2 RGB All In One Liquid CPU Cooler'
Product 1: 'Sony - FE 50mm F1.8 Standard Lens (SEL50F18F)' Product 2: 'Sony FE 50mm f/1.8 Lens E-Mount Lens/Full-Frame Format, Aperture Range: f/1.8 to f/22'
Product 1: 'Zebra 800015-301 (Eltron) Black (K) Resin Ribbon - 1500 Prints' Product 2: 'Zebra 800015-301 Monochrome Ribbon 1500 Images (Standard Black)'
Non-matches:
Product 1: 'RAM Corsair ValueSelect DDR4 2133MHz 1x8Go' Product 2: 'CORSAIR New 8gb (1x8gb) Ddr4 2666mhz Vengeance CMK8GX4M1A2666C16'
Product 1: 'Evolis Zenius/Primacy Black Monochrome Ribbon 2000 image RCT023NAA' Product 2: 'Zebra 800015-101 Black Monochrome Ribbon - 1000 Prints'
Product 1: '128GB Pendrive SanDisk Extreme PRO SSD USB 3.1 420MB/s' Product 2: 'SanDisk 128GB Extreme Pro SDXC 150MB/s V-30 UHS-1 U3 Memory Card'
Product 1: 'Samsung SSD 970 EVO Plus 250GB' Product 2: 'Samsung 970 EVO SSD M.2 2280 - 500GB'
Product 1: 'Akumulator APC Replacement Battery Cartridge #110' Product 2: 'APC - Replacement Battery Cartridge #117'
Product 1: 'Logitech H151 Wired Headset, Stereo Headphones with Rotating Noise-Cancelling Microphone, 3.5 mm Audio Jack' Product 2: 'Logitech H390 USB Headset with Microphone'
Product 1: 'Cooler Master Masterliquid ML120R RGB All-in-one Watercooler' Product 2: 'Cooler Master MasterLiquid ML120L V2 RGB All In One Liquid CPU Cooler'
Product 1: 'HD SSD M.2 500 GB Crucial MX500' Product 2: 'Samsung 970 EVO 500GB - NVMe PCIe M.2 2280 SSD | MZ-V7E500BW'
Product 1: 'ASUS GeForce RTX 2060 DUAL EVO OC' Product 2: 'Gigabyte AORUS GeForce RTX\u2122 2080 Ti XTREME 11G'
Product 1: 'ASUS GeForce GTX 1660 SUPER DUAL EVO 6GB OC Graphics Card' Product 2: 'Gigabyte GeForce GTX 1650 4GB D6 OC Graphics Card'
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent