poseidon-framework / community-archive Goto Github PK
View Code? Open in Web Editor NEWThe Poseidon Community Archive (PCA)
Home Page: https://www.poseidon-adna.org/#/archive_overview
The Poseidon Community Archive (PCA)
Home Page: https://www.poseidon-adna.org/#/archive_overview
Hi!
While extracting some individuals from the prior2018_Boston_Datashare_published package, I noticed that the new plink and eigenstrat genotypes and janno file inherited some mistakes from the original data share.
Group Individual
Vanuatu_2900BP_all_published I1369_al_published (should be I1369_all_published)
Nepal_Samzdong_1500BP.SG (should be Nepal_Samdzong_1500BP.SG) S35.SG
Nepal_Samzdong_1500BP.SG (should be Nepal_Samdzong_1500BP.SG) S41.SG
As they are consistent across all files, this is not urgent, but might be good to fix at some point for consistency within the groups we use for analysis. Thanks!
I believe we should to go back (we had prototypes, once) to a set of checklists that document step-by-step and in unnerving detail how the following operations in our public repository are done:
The complexity of this system is just too big to remember all of it, our automatic tests can not cover everything (for technical and conceptual reasons, e.g. #84) and it is difficult right now to communicate these processes to new and inexperienced users. I think the current write-up we have here is not detailed enough. A very good inspiration might be the extensive guides of the rOpenSci community for package submission and review, e.g. https://devguide.ropensci.org/building.html
Mikkel Nørtoft provided us with a list of Accession numbers for most publications here (see the slack workspace). These should be systematically added to the .janno fiiles (#9).
We could add a simple script to our GitHub Action that creates a list of all files in the repository (excluding the POSEIDON.yml files) and then reads all the POSEIDON.yml files to check if all files are referenced exactly once. That would probably be a fairly easy check to prevent accidentally unattached files (e.g. the .bib files, which are easily forgotten).
The data from 2019_Biagini_Spain are diploid genotypes, and not haploid as the janno file suggests.
$ head 2019_Biagini_Spain.geno ## In EIGENSTRAT
001000020119000110001100000011000011100000101001000011000010110100100010100100011900000100101021201000020002112101110020
000000000011010000100010010010001110001000111010011101010110101002010021011001101000111100000100010110101000011100000011
Heterozygotes (1
s) cannot exist in haploid datasets.
The current version of the package references a bioarxiv preprint. The final paper is out now:
Wang, CC., Yeh, HY., Popov, A.N. et al. Genomic insights into the formation of human populations in East Asia. Nature 591, 413–419 (2021). https://doi.org/10.1038/s41586-021-03336-2
This can be updated.
As far as I can see, the Archaic_Genomes
package doesn't have a bibliography file. We need to compile that.
forge
fails between the SNPs 980000 and 990000 with
Issues in genotype data parsing: SeqFormatException "Error while parsing: not enough input. Error occurred when trying to parse this chunk: \"U\\SOH\""
plink
reports this:
$ plink1.9 --bfile Nakatsuka_SouthPatagonia --recode tab --out test
PLINK v1.90b6.22 64-bit (3 Nov 2020) www.cog-genomics.org/plink/1.9/
(C) 2005-2020 Shaun Purcell, Christopher Chang GNU General Public License v3
Logging to test.log.
Options in effect:
--bfile Nakatsuka_SouthPatagonia
--out test
--recode tab
15814 MB RAM detected; reserving 7907 MB for main workspace.
1233013 variants loaded from .bim file.
20 people (12 males, 8 females) loaded from .fam.
Error: Invalid .bed file size (expected 6165068 bytes).
@AyGhal @93Boy The broken data was added in 125b16b. I think this should be fixed asap, so with the highest priority.
trident validate
did not find the issue, because it only ever looks at the first 100 SNPs (for performance reasons). Probably we should add a full validation to our review process, @stschiff (poseidon-framework/poseidon-hs#223).
This column is defined as follows:
The column Data_Preparation_Pipeline_URL should finally store an URL that links to a complete and human-readable description of the computational pipeline (for example a specific configuration for nf-core/eager) by which the sample data was processed. One solution to document and publish a computational workflow like this might be through protocols.io.
Filling it would be possible for data produced at the MPI-EVA, but it requires a protocols.io pipeline describing the respective data preparation steps. This can probably only be created by the respective author of a paper, so it's nearly impossible to obtain for legacy data.
As demonstrated by #5 we need validation of the genotype data. Maybe the github action could fetch
everything and then attempt a complete forge
. Or we teach validate
some new tricks.
Right now our CHANGELOG files usually look like this:
V X.X.X: What changed there
V X.X.X: What changed here
That is not proper markdown, because there are no explicit line breaks. We should either add line breaks with trailing spaces as defined in markdown (bad idea), make the entries part of a list by prepending -
(slightly better idea) or just rename the files to .txt. I like the last option most, because it's simple to automate.
This issue also requires a change in trident update
, which generated the wrong files in the first place.
The genetic data for Amis and Atayal published in Lazaridis et al (2014) was produced from cell lines, for which the original samples were collected without proper informed consent. Therefore, any derived data is controversial in Taiwan and should not be used for further analyses. We suggest removal of the respective genotypes from the package with further explanation in the README.
I have some questions about the automatic workflow validate.
--ignoreGeno
in the call to trident validate
? Is it possible that is is a historic thing from the time when we didn't have the genotype data checked in?Tissue_Type
fieldsDate_Type
fields (#25)We currently have lots of information about genetic relationships between individuals encoded in the group names (A hack invented in the AADR I believe). We need to move these over to the new Poseidon v2.5 schema columns Relation_To
, Relation_Degree
and Relation_Note
.
When we added the 2022_Gretzinger_AngloSaxons
package, we made a few decisions that need be checked by the first author:
Finally, I noticed that the Supplementary Tables of the paper give the same C14 date ID for samples I0790 (LintonSk352) and I0791 (Linton351), namely SUERC-20250
. They are from the same grave, so the common ID indicates that only one measurement was made, which is fine, but we need the uncalibrated date. I have, for now, switched the Date_Type
to "contextual", as the validator doesn't allow the type to be C14
without uncalibrated dates given.
These two packages are from the same publication. There is a mistake in Publication tag. And correct the Poseidon_IDs in 2019_BraceNatureEcologyEvolution, they are library names.
2013_RaghavanNature and 2014_RaghavanNature have to be merged. 2014_RaghavanNature is the correct name.
Consequence of: poseidon-framework/poseidon-schema#35
snpSet
to all POSEIDON.yml files.Genetic_Source_Accession_IDs
to the janno files.Data_Preparation_Pipeline_URL
to the janno files.Some of the group names that we've imported from the AADR contain symbols like <
or others that trip up our parsing of forge strings.
Ideally we should have group names following alphanumeric symbols and _
and -
and .
or so, but not anything more exotic.
This needs updates. I suggest @93Boy (or @dhananjaya93 ) to give this a go. Just try to find the places on Google Maps, then right click "What is here", then copy the Lat Lon and fill the janno table. Thanks.
We noticed today that Haak et al. 2015 apparently is not in right now. Can you confirm, @AyGhal?
A significant portion of the data in Poseidon was copied and modified from the Reich Lab data release V37.2. This should be pointed out in the respective packages.
One solution would be to add or modify the package description with a sentence like this:
This package was initially constructed based on the Allen Ancient DNA Resource (AADR) (@AADR_37.2).
And then add the following BibTeX entry to the respective .bib file:
@misc{AADR_2019,
title = {Allen Ancient {DNA} Resource ({AADR)}: Downloadable genotypes of present-day and ancient {DNA} data, version 37.2},
year = 2019,
url = {https://reich.hms.harvard.edu/allen-ancient-dna-resource-aadr-downloadable-genotypes-present-day-and-ancient-dna-data},
howpublished = {\url{https://reich.hms.harvard.edu/allen-ancient-dna-resource-aadr-downloadable-genotypes-present-day-and-ancient-dna-data}}
}
(Update on August 14, 2024 by @stschiff)
Special thanks to @mnortoft who compiled much of this list!
The discussion around #83 made me come up with an idea. It would actually be fairly easy to build an HTTP server (deployed on the same cloud-server where the other server currently lives), which does the validation of new branches, PRs and commits for us.
The server would be a simple HTTP server program with a REST API. I can think of the following APIs:
/validate/<branch_name>
-> This would trigger the server to pull the branch and perform a validation. Bandwidth usage would be minimal, because it would only have to pull big files if they changed from master or the latest pulls
/get_validation_results/<branch_name>
-> This would either respond with some kind of "pending" if it's still running, or "done" with the validation results.
We then use a GitHub Webhook to trigger /validate/<branch_name>
whenever a new PR gets opened, or an existing PR gets new commits. A GitHub-action would then query /get_validation_results/<branch_name>
for as long as the result is pending, and show the result when its done.
I'm arguing a but from an armchair here, knowing that the HTTP server itself needs maintenance and code updates, and I don't know when I'll find the time to do it, but conceptually it's easy, and it would solve our bandwidth problems and finally also check genotype data.
Dear Poseidon team,
Thanks for all your great work.
I noticed that one sample, KIN003 (from WangGoldstein2020ScienceAdvances), has a wrong date in the Date_BC_AD_Stop column.
In the original publication, the calibrated date is "1665-present", which may explain the error that the "present" part has become 0 in your Date_BC_AD_Stop column, although this is of course not the calendar year 0 (which does not exist). A calibrated end date for "present" like 1950 would probably make more sense here since it is a C14 date.
When updating the repository with gut pull
I get errors:
Encountered 2 file(s) that should have been pointers, but weren't:
2020_BarqueraCurrentBiology/BarqueraCurrentBiology.bed
2020_BarqueraCurrentBiology/BarqueraCurrentBiology.bim
This is because these two files are actually committed as ordinary files instead of using the git lsf
extension, probably because the committer did not run git lfs install
. Probably an easy fix, but need to play a bit with git rm --cached
and stuff. I'll work in this.
We agreed in our Poseidon meetings that we would soon upgrade our schema to allow for an additional optional file for Poseidon packages, named sequencingSourceFile
. The file will be a tab-separated table, with a number of columns necessary to access and process the raw data behind the genotype data (i.e. fastq or bam files).
@jfy133 kindly provided some help on how to get this this information from the ENA. The easiest way to get started with the ENA data links is simply to use the TSV export feature on the ENA webpage. Example:
Ultimately this will be a joined file with project, sample, experiment (some weird intermediate level) and run level IDs. Run level would then correspond to the actual files you have (corresponding to libraries sequenced on a single run).
There is also an R script which James has written, which provides an R function that takes a Project Accession ID as input and provides a table conforming to https://github.com/SPAAM-community/AncientMetagenomeDir, which might end up being quite similar to what we want for Poseidon.
In a group call today, Matthis thor Straten reported that the package 2022_GnecchiRuscone_CarpathianBasin
seems to have a typo in the column names.
A custom column named Nr_SNPCoverage_on_Target_SNPs
exists, which is likely meant to be Nr_SNPs
and Coverage_on_Target_SNPs
. We should check if and how this has affected the content of other columns in the package.
A critical bug in trident (poseidon-framework/poseidon-hs@df72486), only now fixed in v.0.17.4, caused the janno columns Date_C14_Labnr
to be overwritten by Date_C14_Uncal_BP
. Unfortunately this affects a significant portion of all packages in published_data, because they went through trident when we split up the former Reich dataset.
We will have to go through all packages and reintroduce the respective lab numbers.
The most practical approach for that might be to copy the respective entries from the large Boston_datashare janno file we had before b9d0654 into the new janno files. So yet again boring, manual labour. Alternative ideas are very welcome.
We should also check the other janno files not extracted from the Reich dataset. Maybe some of them went through trident as well. Probably the most easy solution for that will be to make the labnr data type in poseidon-hs more specific, so to validate labnrs. They have a specific structure and can be validated accordingly.
It is sometimes the case that individuals that appear in the janno of a package do not appear in the ENA table for the package (if, say, some of the data was not properly uploaded to the ENA), or vice versa (e.g. when individuals were excluded from analyses and supplementary tables of a paper, but the sequencing data was still uploaded to the ENA).
This will pose a challenge for automatic processing of packages in the future.
A new feature in validate in version 1.4.0.0 checks whether Genotype_Ploidy from the Janno file is consistent with genotype data.
When running this on the community-archive, I noticed that sample HG01804.SG
in the package 2015_1000Genomes_1240K_haploid_pulldown
has indeed heterozygotes, even though it should be haploid pulldown. We need to check whether this is an error already present in the AADR or not.
For now, in order to have this package validate, I marked the Genotype_Ploidy for this one single sample as missing. But we need to set it back to haploid once we have figured out what's going on
The latest server restart failed because of a missing CHANGELOG.md
file in the package 2020_Jeong_EurasiaEasternSteppe
. I fixed it locally on the server for now by removing the line from the POSEIDON.yml
, but I think we should implement a test into our CI for existence of these files.
@dhananjaya93 can you add a CHANGELOG file to that package?
The following four packages have missing Country entries:
trident list --individuals -d . -j Country --raw | awk '$4 == "n/a"' | cut -f1 | sort | uniq -c
4 2014_RaghavanScience
20 2020_Nakatsuka_SouthPatagonia
40 2021_Kilinc_northeastAsia
383 2021_Wang_EastAsia
4 Reference_Genomes
Obviously, the last one should have n/a
, but the others should have proper Countries. Should be easy to fix by checking the original papers. @dhananjaya93 (@93Boy) perhaps you could get to that. Thanks.
We decided to improve the submission processes to be guided by bot-actions or PR-templates, in such a way that submitters will have a bit more hand-holding to submit valid packages to the PCA.
He Yu should be the main contributor for this package. I just fixed some minor issues.
Lots of packages contain missing Date_Type
s in the Janno file. In my, a lot of those we should be able to fill easily:
a. If there are entries in the C14-type columns, put Date_Type
to C14
.
b. If there are entries in the calbrated columns, but not in the C14-columns, put Date_Type
to contextual
.
c. If it's modern samples, put to modern
.
d. If the sample is ancient, but there is no date at all, keep at n/a
for now, but of course those we should anyway also fill soon, at least as a contextual range, which should always be possible from a look into the paper.
published_data % trident list --individuals -d . -j Date_Type --raw | awk '$4 == "n/a"' | cut -f1 | sort | uniq -c
5 2020_Brunel_France
1 2020_Cassidy_IrishDynastic
12 2020_Furtwaengler_Switzerland
20 2020_Nakatsuka_SouthPatagonia
30 2020_Ning_China
1 2020_Wang_subSaharanAfrica
24 2020_Yang_China
40 2021_Kilinc_northeastAsia
826 2021_PattersonNature
18 2021_Saag_EastEuropean
22 2021_SaupeCurrBiol
383 2021_Wang_EastAsia
This package should be updated with the final published version (WangNature2021 in AADR). We added it when it was in bioRxiv.
Shall we introduce an overall Repository versioning? As we are now introducing schema changes for Poseidon 2.6.0, we should package up the old repository as one big tar.gz file and provide it for download somewhere.
Shall we start with 1.0.0
at this point, and put it into some archive, like zenodo?
A lot of samples across packages are lacking spatial latitude and longitude coordinates. Maybe this information can easily be recovered from the respective publications.
With the growing data collection in this repository and the tools we wrote to access it, we could relatively easily prepare some automatic pipelines to construct useful, derived data products. One way to set this up would be to create a github repo, that gets updated automatically with a clever github action, whenever the master branch in published_data changes.
Some ideas:
Theoretically we could also produce figures and interactive toys - then the sky is the limit. I would suggest to stick to the basic necessities, though.
I checked how Date_Type
is filled currently with something equivalent to
trident list --remote --individuals -j Date_Type | cut -f4 | sort | uniq -c
yielding
3181 C14
2356 contextual
7404 modern
534 n/a
Apparently we have 534 missing values in that column, which needs fixing.
I have faced a problem while matching ENA data with Poseidon data. In some packages, I have found multiple Poseidon IDs like I10871.DG, I10871.SG, I10871_published What should be the procedure to map these into ENA data? If I add multiple Poseidon_IDs to such fields will it be harder to read by Trident or other tools?
trident list --individuals -j Publication -d ~/dev/poseidon-framework/published_data --raw | awk '$4=="n/a"'
trident v1.1.3.1 for poseidon v2.5.0
https://poseidon-framework.github.io
[Info] Searching POSEIDON.yml files...
[Info] 168 found
[Info] Checking Poseidon versions...
[Info] Initializing packages...
[Info] Packages loaded: 168
[Info] Preparing output table
[Info] found 15052 individuals/samples
2020_Nakatsuka_SouthPatagonia I8575 Aonikenk_SouthContinent_CerroJohnny_400BP n/a
2020_Nakatsuka_SouthPatagonia I8576 Selknam_FaroMendez_100BP n/a
2020_Nakatsuka_SouthPatagonia I12364 Selknam_NorthTierradelFuego_Grouped_500BP n/a
2020_Nakatsuka_SouthPatagonia I12366 Selknam_NorthTierradelFuego_Grouped_500BP n/a
2020_Nakatsuka_SouthPatagonia I12357 Haush_MitrePeninsula_Grouped_700BP n/a
2020_Nakatsuka_SouthPatagonia I12365 Argentina_Tierra_del_Fuego_brother.I12367 n/a
2020_Nakatsuka_SouthPatagonia I12359 Haush_MitrePeninsula_Grouped_700BP n/a
2020_Nakatsuka_SouthPatagonia I12361 Haush_MitrePeninsula_Grouped_700BP n/a
2020_Nakatsuka_SouthPatagonia I12355 Yamana_BeagleChannel_Grouped_1900-500BP n/a
2020_Nakatsuka_SouthPatagonia I12362 Argentina_NorthTierradelFuego_LaArcillosa2_6000BP n/a
2020_Nakatsuka_SouthPatagonia I12376 Argentina_LagunaToro_2400BP n/a
2020_Nakatsuka_SouthPatagonia I12360 Haush_MitrePeninsula_Grouped_700BP n/a
2020_Nakatsuka_SouthPatagonia I12363 Selknam_NorthTierradelFuego_Grouped_500BP n/a
2020_Nakatsuka_SouthPatagonia I12367 Selknam_NorthTierradelFuego_Grouped_500BP n/a
2020_Nakatsuka_SouthPatagonia I12941 Yamana_BeagleChannel_Grouped_1900-500BP n/a
2020_Nakatsuka_SouthPatagonia I12943 Yamana_BeagleChannel_Grouped_1900-500BP n/a
2020_Nakatsuka_SouthPatagonia I12942 Yamana_BeagleChannel_Grouped_1900-500BP n/a
2020_Nakatsuka_SouthPatagonia I12356 Haush_MitrePeninsula_Grouped_700BP n/a
2020_Nakatsuka_SouthPatagonia I12354 Selknam_NorthTierradelFuego_Grouped_500BP n/a
2020_Nakatsuka_SouthPatagonia I12358 Haush_MitrePeninsula_Grouped_700BP n/a
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.