A simple pipeline wrapped around SpaCy and DaCy for anonymizing danish corpora. The pipeline allows for custom functions to be implemented and piped in combination with custom functions.
The DaCy model is built on multilingual RoBERTa which enables cross-lingual transfer for other languagues ultimately providing a robust named entity recognition model for anonymization that is able to handle noisy Danish text data which could include other languages.
Languages used in the multilingual RoBERTa can be found in appendix A of XLM-RoBERTa paper: Unsupervised Cross lingual Representation Learning at Scale
- Free software: Apache-2.0 license
Disclaimer: As the pipeline utilizes predictive models and regex function to identify entities, there is no guarantee that all sensitive information have been remove.
- Regex for CPRs, telephone numbers, emails
- Integration of custom functions as part of the pipeline
- Named Entity Models for Danish language implemented (PER, LOC, ORG, MISC):
- DaCy: https://github.com/KennethEnevoldsen/DaCy
- Default entities to mask: PER, LOC and ORG (MISC can be specified but covers many different entitites)
- Batch mode and multiprocessing
- DaCy is robust to language changes as it is fine tuned from a multilingual RoBERTa model
- Allow anonymizing using suppression
- Allow masking to be aware of prior knowledge about individuals occuring in the texts
- Pseudonymization module (Person 1, Person 2 etc.)
- Logging to masked_corpus function, enabling tracking of warning if no person was found in a text
- Beta version: Masking or noising with laplace epsilon noise of numbers
Install from pip
pip install DaAnonymizationInstall from source
pip install git+https://github.com/martincjespersen/DaAnonymization.gitDaAnonymization's two main components are:
- TextAnonymizer
- TextPseudonymizer
Both components uses their mask_corpus function to anonymize/pseudonymize text by removing person, location, organization, email, telephone number and CPR. The order of these masking methods are by default CPR, telephone number, email and NER (PER,LOC,ORG) as NER will identify names in the emails. The following example shows an example of applying default anonymization and how it also cross-lingual transfer to english.
from textprivacy import TextAnonymizer
# list of texts (example with cross-lingual transfer to english)
corpus = [
"Hej, jeg hedder Martin Jespersen og er fra Danmark og arbejder i "
"Deloitte, mit cpr er 010203-2010, telefon: +4545454545 "
"og email: [email protected]",
"Hi, my name is Martin Jespersen and work in Deloitte. "
"I used to be a PhD. at DTU in Machine Learning and B-cell immunoinformatics "
"at Anker Engelunds Vej 1 Bygning 101A, 2800 Kgs. Lyngby.",
]
Anonymizer = TextAnonymizer(corpus)
# Anonymize person, location, organization, emails, CPR and telephone numbers
anonymized_corpus = Anonymizer.mask_corpus()
for text in anonymized_corpus:
print(text)Running this script outputs the following:
Hej, jeg hedder [PERSON] og er fra [LOKATION] og arbejder i [ORGANISATION], mit cpr er [CPR],
telefon: [TELEFON] og email: [EMAIL]
Hi, my name is [PERSON] and work in [ORGANISATION]. I used to be a PhD. at [ORGANISATION]
in Machine Learning and B-cell immunoinformatics at [LOKATION].As each project can have specific needs, DaAnonymization supports adding custom functions to the pipeline for masking additional features which are not implemented by default.
from textprivacy import TextAnonymizer
import re
# Takes string as input and returns a set of all occurences
example_custom_function = lambda x: set(list(re.findall(r"\d+ år", x)))
# list of texts
corpus = [
"Hej, jeg hedder Martin Jespersen, er 20 år, er fra Danmark og arbejder i "
"Deloitte, mit cpr er 010203-2010, telefon: +4545454545 "
"og email: [email protected]",
]
Anonymizer = TextAnonymizer(corpus)
# update the mapping to include new custom function entity finder and replacement placeholder
Anonymizer.mapping.update({"ALDER": "[ALDER]"})
# add the name to masking_order in the desired order
# add custom function to custom_functions to update pool of possible masking functions
anonymized_corpus = Anonymizer.mask_corpus(
masking_order=["CPR", "TELEFON", "EMAIL", "NER", "ALDER"],
custom_functions={"ALDER": example_custom_function},
)
for text in anonymized_corpus:
print(text)Hej, jeg hedder [PERSON], er [ALDER], er fra [LOKATION] og arbejder i [ORGANISATION],
mit cpr er [CPR], telefon: [TELEFON] og email: [EMAIL]Sometimes it can be useful to maintain some context regarding sensitive information within the text. Pseudonymization allows for maintaining the connection between entities while masking them. Essentially this means adding a unique identifier for each individual and their information in the text.
By using the optional input argument individuals, you can add prior information about known individuals in the text you want to mask. The structure of individuals needs to be as shown below. The first dictionary provides a key for index of the text in the corpus, the next the unique identifier (integer) of the individuals and finally a dictionary of entities known prior for each individual.
from textprivacy import TextPseudonymizer
# prior information about the text
individuals = {1:
{1:
{'PER': set(['Martin Jespersen', 'Martin', 'Jespersen, Martin']),
'CPR': set(['010203-2010']),
'EMAIL': set(['[email protected]']),
'LOC': set(['Danmark']),
'ORG': set(['Deloitte'])
},
2:
{'PER': set(['Kristina']),
'ORG': set(['Novo Nordisk'])
}
}
}
# list of texts
corpus = [
"Første tekst om intet, blot Martin",
"Hej, jeg hedder Martin Jespersen og er fra Danmark og arbejder i "
"Deloitte, mit cpr er 010203-2010, telefon: +4545454545 "
"og email: [email protected]. Martin er en 20 årig mand. "
"Kristina er en person som arbejder i Novo Nordisk. "
"Frank er en mand som bor i Danmark og arbejder i Netto",
]
Pseudonymizer = TextPseudonymizer(corpus, individuals=individuals)
# Pseudonymize person, location, organization, emails, CPR and telephone numbers
pseudonymized_corpus = Pseudonymizer.mask_corpus()
for text in pseudonymized_corpus:
print(text)Første tekst om intet, blot Person 1
Hej, jeg hedder Person 1 og er fra Lokation 1 og arbejder i Organisation 1, mit cpr er CPR 1,
telefon: Telefon 5 og email: Email 1. Person 1 er en 20 årig mand. Person 2 er en person som
arbejder i Organisation 2. Person 3 er en mand som bor i Lokation 1 og arbejder i Organisation 4DaAnonymization is now available with an easy demo website created in streamlit.
pip install streamlit==1.2.0
streamlit run app.pyRunning the code above will result in a website demoing the use of DaAnonymization.
Evaluations on gender and error biases are conducted in DaCy documentation.
- When SpaCy fixed multiprocessing in nlp.pipe, remove current hack
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent