patentsview / patentsview-evaluation Goto Github PK

View Code? Open in Web Editor NEW

Evaluation and benchmarking of PatentsView disambiguation algorithms

License: GNU General Public License v3.0

Makefile 1.81% Python 51.08% Jupyter Notebook 6.56% HTML 40.55%

patentsview

Placeholder Repo for Issues List

patentsview-evaluation's People

Contributors

Stargazers

patentsview-evaluation's Issues

Add generated time and date to html report

Check the format of design patent numbers in the ENS inventors benchmark

PV is updating is formatting of patent numbers for non-utility patents to conform to https://www.uspto.gov/patents/apply/applying-online/patent-number

Add license, citation file, and citation to arxiv paper

Add examples of how to use benchmark datasets for estimation in their docstrings.

Add representative estimates section to inventor html report

Add handling of missing values in metrics

NA predicted ids or NA reference ids should be handled correctly and conservatively.

Disaggregate summary statistics by region

Provide disaggregated statistics and visualizations by an inventor's modal region. This would potentially help highlight regional biases and other issues related to name spelling conventions.

Create processing scripts for Png LinkedIn benchmark and Lai 2011 benchmark

Missing dependency: openpyxl

Pip dependency openpyxl is missing for included scripts.

pv_evaluation public release checklist

TODO

Develop version 2.0.0 compatible with new data downloads

Stated interpretation of Hill numbers is inversed

Interpretation of Hill numbers H for certain values is stated in terms of 1/H instead... should be fixed.

Add inventor name and link to patents to disambiguation clustering error tables in html reports.

Binette's 2022 estimator example has wrong title

Documentation is missing some functions

There are a few missing documentation items

Benchmark datasets not downloaded when installing package

After running pip install git+https://github.com/PatentsView/PatentsView-Evaluation.git@release, the load_patentsview_inventors_benchmark fails to load and gives a "no such file or directory" error. It seems like the csv file is not in the directory the function searches for. Thank you!

Docs/Example Request - Show Full RLData experiment + results in Python

Hello,

As an understanding check while reading the paper I attempted to recreate the experiment from Appendix 1. Appendix 1 says in part:

For this example, we considered the RLdata10000 dataset from Sariyar and Borg (2022). This is a
synthetic dataset containing 10,000 records with first name, last name, and date of birth attributes. There
is noise in these attributes and a 10% duplication rate. Ground truth identity is known for all records.
The disambiguation algorithm we consider matches records if any of the following conditions are met:
• records agree on first name, last name, and birth year,
• records agree on first name, birth day, and birth year, or
• records agree on last name, birth day, and birth year.
Note that this is not at all a good disambiguation algorithm. It has 52% precision and 83% recall.

I've attempted to reproduce those precision-recall metrics using that disambiguation algorithm in Python on the RLdata10000 dataset but haven't been able to. Any chance you'd be willing to share it as an example for future readers?

My (likely erroneous) implementation below in case it's helpful, which on my machine returns a precision of 0.5943 and a recall of 0.832:

import pandas as pd

pd.set_option('display.max_columns', None)

df = pd.read_csv('RLdata10000.csv')
comparisons = pd.merge(df, df, how="cross", suffixes=["_left", "_right"])

comparisons['true_match'] = (comparisons["ent_id_left"] == comparisons["ent_id_right"])

first_names_c1_match = comparisons['fname_c1_left'] == comparisons['fname_c1_right']
first_names_c2_match = (
    (comparisons['fname_c2_left'] == comparisons['fname_c2_right'])
    | (comparisons['fname_c2_left'].isna() & comparisons['fname_c2_right'].isna())
)
first_names_match = first_names_c1_match & first_names_c2_match


last_names_c1_match = comparisons['lname_c1_left'] == comparisons["lname_c1_right"]
last_names_c2_match = (
    (comparisons['lname_c2_left'] == comparisons['lname_c2_right'])
    | (comparisons['lname_c2_left'].isna() & comparisons['lname_c2_right'].isna())
)
last_names_match = last_names_c1_match & last_names_c2_match

birth_days_match = comparisons['bd_left'] == comparisons["bd_right"]
birth_years_match = comparisons['by_left'] == comparisons['by_right']

condition_1 = first_names_match & last_names_match & birth_years_match
condition_2 = first_names_match & birth_days_match & birth_years_match
condition_3 = last_names_match & birth_days_match & birth_years_match

comparisons['predicted_match'] = condition_1 | condition_2 | condition_3

comparisons = comparisons[comparisons['predicted_match'] | comparisons['true_match']]
comparisons = comparisons[comparisons['rec_id_left'] != comparisons['rec_id_right']]

comparisons['true_and_predicted_match'] = comparisons['true_match'] & comparisons['predicted_match']

num_true_matches = comparisons['true_match'].sum()
num_predicted_matches = comparisons['predicted_match'].sum()
num_true_and_predicted_matches = comparisons['true_and_predicted_match'].sum()

true_precision = num_true_and_predicted_matches/num_predicted_matches
true_recall = num_true_and_predicted_matches/num_true_matches
true_f1 = (2 * true_precision * true_recall) / (true_precision + true_recall)

print(f"There is a true precision of {round(true_precision, 4)}")
print(f"There is a true recall of {round(true_recall, 4)}")
print(f"There is a true f1 of {round(true_f1, 4)}")

Recommend Projects