cleanepi is an R package designed for cleaning, curating, and standardizing epidemiological data. It streamlines various data cleaning tasks that are typically expected when working with datasets in epidemiology.
Key functionalities of cleanepi include:
-
Removing irregularities: It efficiently removes duplicated and empty rows and columns, as well as columns with constant values.
-
Handling missing values: It replaces missing values with the standard
NAformat, ensuring consistency and ease of analysis. -
Ensuring data integrity: It ensures the uniqueness of uniquely identified columns, thus maintaining data integrity and preventing duplicates.
-
Date conversion: It offers functionality to convert character columns to Date format under specified conditions, enhancing data uniformity and facilitating temporal analysis.
-
Standardizing entries: It can standardize column entries into specified formats, promoting consistency across the dataset.
-
Age calculation: It calculates age from a given
Datecolumn and a reference date, providing valuable demographic insights for epidemiological analysis.
cleanepi operates on data frames or similar structures like tibbles, as well as linelist objects commonly used in epidemiological research. It returns the processed data in the same format, ensuring seamless integration into existing workflows. Additionally, it generates a comprehensive report detailing the outcomes of each cleaning task.
cleanepi is developed by the Epiverse-TRACE team at the Medical Research Council The Gambia unit at the London School of Hygiene and Tropical Medicine.
The latest development version of cleanepi can be installed from GitHub.
if (!require("pak")) install.packages("pak")
pak::pak("epiverse-trace/cleanepi")
library(cleanepi)The main function in cleanepi is clean_data(), which internally
makes call of almost all standard data cleaning functions, such as
removal of empty and duplicated rows and columns, replacement of missing
values, etc. However, each function can also be called independently to
perform a specific task. This mechanism is explained in details in the
vignette. Below is typical example of how to use the clean_data()
function.
# READING IN THE TEST DATASET
test_data <- readRDS(system.file("extdata", "test_df.RDS",
package = "cleanepi"))| STUDY_ID | EVENT_NAME | COUNTRY_CODE | COUNTRY_NAME | DATE.OF.ADMISSION | DATEOFBIRTH | DATE_FIRST_PCR_POSITIVE_TEST | SEX |
|---|---|---|---|---|---|---|---|
| PS001P2 | day 0 | 2 | Gambia | 01/12/2020 | 06/01/1972 | Dec 01, 2020 | 1 |
| PS002P2 | day 0 | 2 | Gambia | 28/01/2021 | 02/20/1952 | Jan 01, 2021 | 1 |
| PS004P2-1 | day 0 | 2 | Gambia | 15/02/2021 | 06/15/1961 | Feb 11, 2021 | -99 |
| PS003P2 | day 0 | 2 | Gambia | 11/02/2021 | 11/11/1947 | Feb 01, 2021 | 1 |
| P0005P2 | day 0 | 2 | Gambia | 17/02/2021 | 09/26/2000 | Feb 16, 2021 | 2 |
| PS006P2 | day 0 | 2 | Gambia | 17/02/2021 | -99 | May 02, 2021 | 2 |
| PB500P2 | day 0 | 2 | Gambia | 28/02/2021 | 11/03/1989 | Feb 19, 2021 | 1 |
| PS008P2 | day 0 | 2 | Gambia | 22/02/2021 | 10/05/1976 | Sep 20, 2021 | 2 |
| PS010P2 | day 0 | 2 | Gambia | 02/03/2021 | 09/23/1991 | Feb 26, 2021 | 1 |
| PS011P2 | day 0 | 2 | Gambia | 05/03/2021 | 02/08/1991 | Mar 03, 2021 | 2 |
# READING IN THE DATA DICTIONARY
test_dictionary <- readRDS(system.file("extdata", "test_dictionary.RDS",
package = "cleanepi"))| OPTIONS | VALUES | GRP | ORDERS |
|---|---|---|---|
| 1 | male | sex | 1 |
| 2 | female | sex | 2 |
# DEFINING THE CLEANING PARAMETERS
use_na <- list(target_columns = NULL, na_strings = "-99")
remove_duplicates <- list(target_columns = NULL,
rm_empty_rows = TRUE,
rm_empty_cols = TRUE,
rm_constant_cols = TRUE)
standardize_date <- list(target_columns = NULL,
error_tolerance = 0.5,
format = NULL,
timeframe = as.Date(c("1973-05-29",
"2023-05-29")))
standardize_subject_ids <- list(id_col_name = "study_id",
format = NULL,
prefix = "PS",
suffix = "P2",
range = c(1, 100))
params <- list(
keep = NULL,
replace_missing_values = use_na,
remove_duplicates = remove_duplicates,
standardize_date = standardize_date,
standardize_subject_ids = standardize_subject_ids,
to_numeric = "sex",
dictionary = test_dictionary
)# PERFORMING THE DATA CLEANING
cleaned_data <- clean_data(
data = test_data,
params = params
)
#>
#> cleaning column names before after
#> 1 study_id study_id
#> 2 event_name event_name
#> 3 country_code country_code
#> 4 country_name country_name
#> 5 date.of.admission date_of_admission
#> 6 dateOfBirth date_of_birth
#> 7 date_first_pcr_positive_test date_first_pcr_positive_test
#> 8 sex sex
#>
#> replacing missing values with NA
#> removing duplicated rows
#>
#> No duplicates found from the specified columns.
#>
#> standardising date columns
#> checking subject IDs format
#> converting sex into numeric
#> performing dictionary-based cleaning| STUDY_ID | DATE_OF_ADMISSION | DATE_OF_BIRTH | DATE_FIRST_PCR_POSITIVE_TEST | SEX | |
|---|---|---|---|---|---|
| 1 | PS001P2 | 2020-12-01 | NA | 2020-12-01 | male |
| 2 | PS002P2 | 2021-01-28 | NA | 2021-01-01 | male |
| 4 | PS003P2 | 2021-02-11 | NA | 2021-02-01 | male |
| 6 | PS006P2 | 2021-02-17 | NA | 2021-05-02 | female |
| 8 | PS008P2 | 2021-02-22 | 1976-05-10 | 2021-09-20 | female |
| 9 | PS010P2 | 2021-03-02 | 1991-09-23 | 2021-02-26 | male |
| 10 | PS011P2 | 2021-03-05 | 1991-08-02 | 2021-03-03 | female |
# EXTRACT THE DATA CLEANING REPORT
report <- attr(cleaned_data, "report")# DISPLAY THE DATA CLEANING REPORT
print_report(report)browseVignettes("cleanepi")✅ update and create test files
This package is currently an experimental, as defined by the RECON software lifecycle. This means that it is functional, but interfaces and functionalities may change over time, testing and documentation may be lacking.
Contributions are welcome via pull requests.
Please note that the cleanepi project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
citation("cleanepi")
#> To cite package 'cleanepi' in publications use:
#>
#> Mané K (2024). _cleanepi: Clean and standardize epidemiological
#> data_. R package version 0.0.2,
#> https://github.com/epiverse-trace/cleanepi,
#> <https://epiverse-trace.github.io/cleanepi>.
#>
#> A BibTeX entry for LaTeX users is
#>
#> @Manual{,
#> title = {cleanepi: Clean and standardize epidemiological data},
#> author = {Karim Mané},
#> year = {2024},
#> note = {R package version 0.0.2,
#> https://github.com/epiverse-trace/cleanepi},
#> url = {https://epiverse-trace.github.io/cleanepi},
#> }
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent