Hello BruceWen!
thanks for your work and making the medal dataset available!
I loaded the dataset and was looking into the most frequently occuring abbreviations from the first 50000 rows and was wondering why the abbreviations all seem rather cryptic:
![grafik](https://private-user-images.githubusercontent.com/16795915/356179329-89304462-c7c5-41f9-99f6-c9d7f4741855.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUxNjA3MDYsIm5iZiI6MTcyNTE2MDQwNiwicGF0aCI6Ii8xNjc5NTkxNS8zNTYxNzkzMjktODkzMDQ0NjItYzdjNS00MWY5LTk5ZjYtYzlkN2Y0NzQxODU1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDAzMTMyNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU3NDQ5MzFmZjcxMTFmZTMxNTFiMmQ5ODI4MDgwOTdlMmMwMmFkZjNjNDEwZjZkZGUwN2IxN2M0ZmVhMDA1NjYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0dKZEXSChkfCTEAE0X3a5zZLkceZlHqZXCkh5M4oFf8)
It seems that occurences of "after" were substituted with "T3" in the articles through the reverse-substitution process, but I don't understand where the abbreviation "T3" stems from.
![grafik](https://private-user-images.githubusercontent.com/16795915/356182104-05858a87-f389-458e-a6af-192507497edb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUxNjA3MDYsIm5iZiI6MTcyNTE2MDQwNiwicGF0aCI6Ii8xNjc5NTkxNS8zNTYxODIxMDQtMDU4NThhODctZjM4OS00NThlLWE2YWYtMTkyNTA3NDk3ZWRiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDAzMTMyNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI5OTBkMGM1MGIwNTMxMGNkYTIyYTFjZDFkYWU2OGQzZWE5Mjk0NDlhMjRiNDA1ZTVlYmQxMjQ3NTBlM2E2ZGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XFhkF0if_lulRtn6jRRrcGoT08zs6gh109xQaCqreAI)
I found the original publication of the first article in the dataset with the "T3" abbreviation and it does not mention T3 anywhere but just "after", which was then replaced through your procedure.
Publication: https://pubmed.ncbi.nlm.nih.gov/22/
I also noticed that the text in the medal dataset does not contain and punctuation. Why was this removed? The part in your publication where the creation of the medal dataset is presented, does not go into details here.
And you modified abbreviations from prior research and ended up with 5,886 abbreviations in total with corresponding expansions. Is this dataframe/dictionary also available for download?
This is the code I used to reconstruct such abbreviation-expansion dictionary based on the medal dataset now:
from tqdm.auto import tqdm
import pandas as pd
medal_df = pd.read_csv(path_to_full_data_csv)
medal_df.head()
loop_df = medal_df[:50000]
overall_article_abrbeviations_df = pd.DataFrame()
for index, row in tqdm(loop_df.iterrows(), total=len(loop_df)):
start_positions = [int(x) for x in row["LOCATION"].split("|")]
resolutions = row["LABEL"].split("|")
abbreviations = []
for start in range(len(start_positions)):
abbreviations.append(row["TEXT"].split(" ")[start_positions[start]])
this_article_abbreviations_df = pd.DataFrame()
this_article_abbreviations_df["Abbreviations"]=abbreviations
this_article_abbreviations_df["Resolutions"]=resolutions
overall_article_abrbeviations_df = pd.concat([overall_article_abrbeviations_df,this_article_abbreviations_df])
abbreviation_df = overall_article_abrbeviations_df.groupby(['Abbreviations', 'Resolutions']).size().sort_values(ascending=False).reset_index(name='count')
I inspected the most frequently occuring and least frequently occuring abbreviations and I suspect that some error happened during the reverse-substitution process for most of the frequently occuring abbreviations:
![grafik](https://private-user-images.githubusercontent.com/16795915/356183186-2fcd15e2-4911-492b-849f-3beed21581bb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUxNjA3MDYsIm5iZiI6MTcyNTE2MDQwNiwicGF0aCI6Ii8xNjc5NTkxNS8zNTYxODMxODYtMmZjZDE1ZTItNDkxMS00OTJiLTg0OWYtM2JlZWQyMTU4MWJiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDAzMTMyNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY5ZDU2ZGM3N2NlMTRlMjQxYWUyMzU4Y2Y5NDkyNjc4NWY2OTY0N2E4MjkzZjkzZDYyOWMwNWNkMGIwYTNiZTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.68F-Y04nQqie-H5yYIVzmuHi2EJkThZxkrUkyBRHRW0)
Hopefully you can help me understand these things a little bit better :)