Documentation

Description

This issue is in preparation for request for a gold Tripal extension module badge.
https://github.com/UofS-Pulse-Binfo/genotypes_loader

Bronze

Silver

Gold

Attempting to run the genotype_loader results in the following error:

Could not find germplasm type 'accession' with the vocabulary 'GCP germplasm[error]
ontology'

Lacey's initial response point to the following code where it should be creating this:
https://github.com/tripal/tripal/blob/7.x-3.x/tripal_chado/includes/setup/tripal_chado.setup.inc#L646-L656

Currently you describe the "Marker Type" under --help as 'The Sequence Ontology (SO) term name that describes the marker technology used to generate the genotypes in the file (e.g. "Exome Capture", "GBS", "KASPar", etc.).'

However, this is the featureprop.value marker type and as such is a free-text descriptor not an SO term. I also noticed in https://github.com/UofS-Pulse-Binfo/genotypes_loader/blob/master/includes/genotypes_loader.genotype_matrix.inc#L17 that you defined marker type wrong there too though it looks like you used it correctly.

Please clean-up documentation to ensure that you are clear which marker type it is. The mix-up in help caused me to supply the wrong value when loading :-(

This is necessary to ensure we can use JSON for metadata. See: UofS-Pulse-Binfo/nd_genotypes@7ef24ff for how to implement this.

I'm running the example command below after creating the matching organism and project.

drush load-genotypes sample_files/sample.vcf sample_files/samples.list --organism="Lens culinaris" --variant-type="SNP" --marker-type="genetic_marker" --project-name="My SNP Discovery Project" --ndgeolocation="here"

and get the following error:

Begin validating options and samples...
  Loading your data according to the "Genotype Call Custom Table" Method.
  Found organism "lens culinaris" in the database.
  Found project "My SNP Discovery Project" in the database.
  Checking for samples [Insert And Select] and for germplasm [Insert And Select] in the database...
PDOException: SQLSTATE[23502]: Not null violation: 7 ERROR:  null value in column "type_id" violates   [error]
not-null constraint
DETAIL:  Failing row contains (2, null, 1, 1048-8R, KP:GERM7951 Individual  Lens Culinaris, null, null,
f). in /Users/bc/tripal/includes/database/database.inc:2227
Stack trace:
#0 /Users/bc/tripal/includes/database/database.inc(2227): PDOStatement->execute(Array)
#1 /Users/bc/tripal/includes/database/pgsql/database.inc(111): DatabaseStatementBase->execute(Array,
Array)
#2 /Users/bc/tripal/includes/database/database.inc(2406): DatabaseConnection_pgsql->query('INSERT INTO
cha...', Array, Array)
#3 /Users/bc/tripal/sites/all/modules/tripal/tripal_chado/api/tripal_chado.query.api.inc(1769):
db_query('INSERT INTO cha...', Array)
#4 /Users/bc/tripal/sites/all/modules/tripal/tripal_chado/api/tripal_chado.query.api.inc(637):
chado_query('INSERT INTO cha...', Array)
#5 /Users/bc/tripal/sites/all/modules/custom/genotypes_loader/api/genotypes_loader.api.inc(99):
chado_insert_record('stock', Array)
#6 /Users/bc/tripal/sites/all/modules/custom/genotypes_loader/genotypes_loader.drush.inc(343):
genotypes_loader_helper_add_record_with_mode('Germplasm', 'stock', 2, Array, Array)
#7 /Users/bc/tripal/sites/all/modules/custom/genotypes_loader/genotypes_loader.drush.inc(155):
genotypes_loader_load_genotypes('/Users/bc/tripa...', '/Users/bc/tripa...', Array)
#8 /Users/bc/.composer/vendor/drush/drush/includes/command.inc(361):
drush_genotypes_loader_load_genotypes('/Users/bc/tripa...', '/Users/bc/tripa...')
#9 /Users/bc/.composer/vendor/drush/drush/includes/command.inc(212): _drush_invoke_hooks(Array, Array)
#10 /Users/bc/.composer/vendor/drush/drush/includes/command.inc(180):
drush_command('sample_files/sa...', 'sample_files/sa...')
#11 /Users/bc/.composer/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php(64): drush_dispatch(Array)
#12 /Users/bc/.composer/vendor/drush/drush/drush.php(70): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#13 /Users/bc/.composer/vendor/drush/drush/drush.php(11): drush_main()
#14 {main}

I may be making a bad assumption with #33 that the example files are "good to go".

I dump the erroring value:

array(4) {
  'uniquename' =>
  string(38) "KP:GERM7951 Individual  Lens Culinaris"
  'type_id' =>
  NULL
  'organism_id' =>
  int(1)
  'name' =>
  string(7) "1048-8R"
}

so, this stock (germplasm?) needs a type, right? From samples.list... is this an argument i should be supplying with the command? should it be in the example file?

edit- oh i see, its not parsking the samples.list file correctly. the type should be Individual. Why isnt it parsing correctly?

edit- i manually re-parsed the input file, looks better, but i still have no type_id. i think this is because i dont have a term for "Individual" in my db. Should this be part of #35 ?

final edit- i replaced the "Individual" with a known term (mRNA) and still no loading...

This issue describes an incompatibility between this module and ND Genotypes.

Specifically, when sync'ing genotypes loaded by this module, I am unable to create the variant detail mview. This is because the nd_genotypes module assumes each variant has

• a feature record (type=sequence_variant; configurable)
• at least one featureloc record describing it's location
• a featureprop record (type=additionalType; configurable) where the value descibes the type of variant (e.g. SNP, MNP, Indel).

There is no featureprop record describing the type of variant when data is loaded via your module.

Note: The variant records should follow the same format as the marker records. Your module correctly has two type_ids for markers (the feature.type_id=genetic_marker and featureprop.type_id=marker_type). You need the same for variants.

I get this error

ERROR: Could not find type 'is_marker_of' with vocabulary 'feature_relationship'. [error]

despite running this term insert:

    chado_insert_cvterm([
      'id' => 'local:is_marker_of',
      'name' => 'is_marker_of',
      'cv_name' => 'stock_relationship',
      'definition' => 'Local term from the genotypes_loader module.'
    ]);

looks like the README says it needs it in the wrong CV.

Currently, the settings forms assumes the user will want to pull terms from a particular controlled vocabulary, or save a new term to that vocabulary. This is a huge assumption and because Tripal 3 encourages use of public ontologies, we need to handle the ability to specify vocabularies through this module. Perhaps we can borrow some functionality from the configuration form for analyzed phenotypes?

@laceysanderson please expand on this issue if necessary! (Or tell me in person and I'll make the change)

The fastest method I've found for loading large files is to use PostgreSQL COPY FROM command. However, this query cannot be executed via chado_query() because it would require the database user to have superuser privileges which would in turn be a security issue. In order to safely get around this, we take advantage of the fact that the file is local (must be transferred separately due to size) and use the command line \copy variation of this command.

Here's an example of how to execute the \copy command: https://github.com/UofS-Pulse-Binfo/generate_trpdata/blob/master/api/generate.genotypes.api.inc#L498.

To use the \copy from command you need to have first generated a file with your data. You can do this by modifying the helper function (https://github.com/UofS-Pulse-Binfo/genotypes_loader/blob/master/api/genotypes_loader.api.inc#L25) to check if the mode is insert only and if so write to a file instead of executing the insert command. Then at the end of the function check the number of lines and if it exceeds a set number (e.g. 10,000): (1) create the copy from command, (2) execute it and (3) empty the file. You would need an additional API function to copy in all the partial files at the end.

Certain VCF genotype fields allow for floats (examples: AF, GP) and the module currently discards these values because it skips any presence of a period (assumes a missing value). This should be an easy fix by ensuring preg_match strictly matches a period only (thus, the field is actually missing), not only checking for the presence of one.

We need official documentation beyond just the readme for release of this module. I'm opening this issue as a place for discussion, planning and outlining such documentation. Additionally we will use this issue as a launching point for documentation development but decided that documentation is not subject to the typical PR review.

We should change our sample files to use Germplasm Accession (CO_010:0000044) rather then Individual. This ensures our content maps to default Tripal content types.

Hi,
Firstly, thank you for developing and maintaining this module!
I am trying to load a vcf file using drush load-genotypes however, I am getting the error

ERROR: Could not find type 'DNA' with vocabulary 'stock_type'

It seems that I do not have the required term 'DNA' for in my controlled vocabulary table. May I ask what is the best practice for adding vocabularies and terms? Or do people usually just log in to their postgres db and insert terms as they come up?
Thank you 😄

Hi,
I was trying to load genotypes from vcf file. However, instead of using the raw IDs in the vcf file, the loaded variants were renamed by the position info in the vcf file, such as chr1p29143.

How to name the variant by its real ID while loading from a vcf file?

Thank you.
Best wishes,

Songtao Gui

Documentation

Description

This issue is in preparation for request for a gold Tripal extension module badge.
https://github.com/UofS-Pulse-Binfo/genotypes_loader

Bronze

Silver

Gold

Same problem as reported in UofS-Pulse-Binfo/nd_genotypes#26 the installer uses regexp to check the postgres version but assumes 9.x

When loading genotypes I see the following errors:

$ drush load-genotypes ***.vcf ***samples.txt --marker-type="genetic_marker" --variant-type="sequence_variant" --organism="Lens culinaris" --project-name="AGILE: Application of Genomic Innovation in the Lentil Economy" --ndgeolocation="here"
WD tripal_chado: chado_insert_record; Cannot insert duplicate record into genotype table: Array              [error]
(   
    [name] => GG
    [uniquename] => GG
    [description] => GG
    [type_id] => 1159
)

ERROR (TRIPAL_CHADO): chado_insert_record; Cannot insert duplicate record into genotype table: Array
(   
    [name] => GG
    [uniquename] => GG
    [description] => GG
    [type_id] => 1159
)

[site http://default] [TRIPAL ERROR] [TRIPAL_CHADO] chado_insert_record; Cannot insert duplicate record into genotype table: Array(    [name] => GG    [uniquename] => GG    [description] => GG    [type_id] => 1159)
Tried to insert "Genotype" but the primary key is returned empty (mode=2). Values: Array                     [error]
(   
    [name] => GG
    [uniquename] => GG
    [description] => GG
    [type_id] => 1159
)

Upon checking my database I see that indeed those genotypes do already exist but with a different type (769). Switching the type fixed the problem.

To be made Tripal v3 compatible this module simply needs the dependency in the .info file changed from tripal_core to tripal.

This is due to the main change between Tripal v2 and Tripal v3 being Nodes => Entities and this module does not interact with Tripal Nodes. Instead it simply loads data into Chado directly and requires the admin to choose which content to sync/publish in their respective Tripal site.

Hi @carolyncaron , when I loaded the genotype data into chado table, I noticed there is something wrong with the 'type_id', and maybe it does not insert into table automatically. I am not sure why? and except that please help me to check if I have other error. Thanks~
Here is the output information from console:
###########################################################################################
23
Please enter a free-text title that describes the type of variants in the file (eg. SNP, MNP, indel).: SNP
Please enter a free-text title that describes the marker technology used to generate the genotypes in the file (e.g. "Exome Capture", "GBS", "KASPar", etc.).: GBS
Please provide the name of a project in which all genotypes will be grouped to allow users to specify a particular dataset.: Cucurbita pepo Demonstration Genotypic Data
Begin validating options and samples...
Loading your data according to the "Genotype Call Custom Table" Method.
Found organism "Cucurbita pepo" in the database.
Found project "Cucurbita pepo Demonstration Genotypic Data" in the database.
Checking for samples [Insert And Select] and for germplasm [Insert And Select] in the database...
Successfully mapped stocks and germplasm.
Detected genotype matrix file format.
Proceeding to load genotypes...
Number of lines to process: 4156
PDOException: SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "type_id" violates not-null constraint [error]
DETAIL: Failing row contains (17731266, null, 7, Cp4.1LG01:1198, Cp4.1LG01:1198, null, null, null, null, f, f, 2022-02-21 15:53:39.608159, 2022-02-21 15:53:39.608159). in
/var/www/html/v2/includes/database/database.inc:2278
Stack trace:
#0 /var/www/html/v2/includes/database/database.inc(2278): PDOStatement->execute(Array)
#1 /var/www/html/v2/includes/database/pgsql/database.inc(111): DatabaseStatementBase->execute(Array, Array)
#2 /var/www/html/v2/includes/database/database.inc(2457): DatabaseConnection_pgsql->query('INSERT INTO cha...', Array, Array)
#3 /var/www/html/v2/sites/all/modules/tripal/tripal_chado/api/tripal_chado.query.api.inc(1783): db_query('INSERT INTO cha...', Array, Array)
#4 /var/www/html/v2/sites/all/modules/tripal/tripal_chado/api/tripal_chado.query.api.inc(637): chado_query('INSERT INTO cha...', Array)
#5 /var/www/html/v2/sites/all/modules/genotypes_loader/api/genotypes_loader.api.inc(104): chado_insert_record('feature', Array)
#6 /var/www/html/v2/sites/all/modules/genotypes_loader/includes/genotypes_loader.genotype_matrix.inc(130): genotypes_loader_helper_add_record_with_mode('Variant', 'feature', 2, Array)
#7 /var/www/html/v2/sites/all/modules/genotypes_loader/genotypes_loader.drush.inc(469): genotypes_loader_load_matrix('/var/www/html/C...', Array, Array)
#8 /var/www/html/v2/sites/all/modules/genotypes_loader/genotypes_loader.drush.inc(172): genotypes_loader_load_genotypes('/var/www/html/C...', '/var/www/html/C...', Array)
#9 phar:///usr/local/bin/drush/includes/command.inc(422): drush_genotypes_loader_load_genotypes('/var/www/html/C...', '/var/www/html/C...')
#10 phar:///usr/local/bin/drush/includes/command.inc(231): _drush_invoke_hooks(Array, Array)
#11 phar:///usr/local/bin/drush/includes/command.inc(199): drush_command('Cpepo.table.tsv', 'samples.list')
#12 phar:///usr/local/bin/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch(Array)
#13 phar:///usr/local/bin/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#14 phar:///usr/local/bin/drush/includes/startup.inc(465): drush_main()
#15 phar:///usr/local/bin/drush/includes/startup.inc(369): drush_run_main(false, '/', 'Phar detected. ...')
#16 phar:///usr/local/bin/drush/drush(114): drush_startup(Array)
#17 /usr/local/bin/drush(10): require('phar:///usr/loc...')
#18 {main}
##################################################################################

It would be wonderful to have a settings page allowing admin to set the variant/marker cvterms to use, as well as, the database mode (ie: insert only, insert & select, select only).

Hi everybody,

Can I use genotypes loader to load somatic data?

Here is a screenshot of the file I want to load in.

Currently only genotypes for a single organism can be loaded from a single file, but we may have diversity panels with mixed species. One way to accomplish this would be to optionally allow the organism to be specified on a per-sample basis in the user-supplied samples list.

it would be nice if the example in the README matched the sample data you provide, and included the relative path.

drush load-genotypes mygenotypes.vcf samples.list --organism="Lens culinaris" --variant-type="SNP" --marker-type="genetic_marker" --project-name="My SNP Discovery Project" --ndgeolocation="here"

could perhaps instead be

drush load-genotypes sample_files/sample.vcf sample_files/samples.list --organism="Lens culinaris" --variant-type="SNP" --marker-type="genetic_marker" --project-name="My SNP Discovery Project" --ndgeolocation="here"

Currently, the copy chunk size is hard-coded in the API as 10 000 records. The efficiency of the COPY functionality is dependent on choosing an appropriate number of records to copy in at once based on the total number of records being loaded. This can be hard to determine, so the user may want to tweak the size on a test site first. They may also choose the size based on available space on their hard drive for the temporary file that gets copied into the database.

Lacey suggested the following in PR #19 (referring to $copy_chunk_size = 10000; in the API):

I would allow this to be set via variable_get/set().

1. Change this line to $copy_chunk_size = variable_get('genotypes_loader_cp_chunk_size', 10000);
2. Set via variable_set() in the first drush function after exposing an option to the user. This way you don't have to worry about trying to pass the value through the chain of functions while still providing a way for your user to change this value per file.

Currently the module stores genotypic calls including the project from which those calls are generated from. However, we are missing the link between markers and project which makes searching for markers by project name difficult. Therefore, the module needs to add an insert into the project_feature table in chado for each marker that is inserted.

Note: This update needs to occur for each of the file types that can be loaded.

Currently, the genotype call is interpreted from the GT field, which is not always reflective of the PL field. GT is determined somewhat based on Hardy-Weinburg, which does not apply to our populations.

Though this is the ideal approach for our own group, I can't help but wonder about other prospective users of the loader. Is using PL a generic approach? Or will other groups want to load genotypes solely based on GT? Should we make it an option?

Hello,
Following the instruction in the link (https://genotypes-loader.readthedocs.io/en/latest/demo.html) and using this command 'sudo drush load-genotypes sample.vcf samples.list --organism="Citrus sinensis" --variant-type="SNP" --marker-type="genetic_marker" --project-name="Citrus Demonstration Genotypic Data"', I got this error:
################
Begin validating options and samples...
Loading your data according to the "Genotype Call Custom Table" Method.
Found organism "Citrullus sinensis" in the database.
Found project "Citrus Demonstration Genotypic Data" in the database.
Checking for samples [Insert And Select] and for germplasm [Insert And Select] in the database...
ERROR: Could not find germplasm type 'accession' with the vocabulary 'GCP germplasm ontology'.
################
I am not sure if it means I should install the GCP germplasm ontology. Thanks~

Since the loader is already saving meta-data for each genotype call in the VCF format, it would be an extremely useful feature to allow the user to set thresholds for what constitutes a good/bad quality call, and then allow that information to be stored directly into the database, also as meta-data. For our purposes, it is more efficient to calculate and store quality metrics than to perform these calculations on the front-end every time user requests for genotype calls through a web interface occurs.

For example,

• The user specifies they want a quality of metric of "NDQR" to be saved as meta-data
• The user also provides thresholds, let's say lower and upper thresholds, to categorize each call as "bad", "acceptable" or "excellent" quality. These thresholds are set based on meta-data that is already present in the genotype file. In this case, the user may specify a lower-threshold as having a read depth (DP) of 5 and allele depth (AD) of 4, whereas an upper threshold requires a read depth of 50 and allele depth of 45.

We would also like to extend this to all file formats and not just VCF, by allowing an optional column within the legacy format, and additional parsing capabilities within the genotype call column of the matrix format for key-value pairs. This provides the user the most flexibility to specify quality in whatever method they want based on their data (for example, if they happen to know read depth or have a percentage-based quality score), regardless of file format.

For reference, this is our (myself and @laceysanderson) thought process about this on the whiteboard:

The current suggested formats are too restrictive.

Suggested Flat-file format:

1. Variant name (required)
2. Marker name (optional)
3. Chromosome (required, unless select only)
4. Position (required, unless select only)
5. Sample name (required)
6. Genotype Call (required)
7. Meta-data (optional) (refer to issue #5 for suggested format)

Suggested Matrix format:

1. Variant name (required)
2. Marker name (optional)
3. Chromosome (required, unless select only)
4. Position (required, unless select only)
5. Genotype call (required) PLUS meta-data* (optional)

*Also see issue #5 for more context regarding how we propose to handle meta-data

Not sure whether this should be in its own issue, but it has also been discussed that users can suggest a custom format for marker name through the use of tokens with chromosome + position.

Instead, we can request from the user what type of file they are loading. This is due to planned increased flexibility in the file formats (see issue #10), and thus this feature is too complex to maintain... sad face

Warning: Illegal string offset 'TAXRANK:genus' in local__genotype_matrix_link_formatter->view() (line 45 of /var/www/html/sites/all/modules/nd_genotypes/includes/TripalFields/local__genotype_matrix_link/local__genotype_matrix_link_formatter.inc).
Notice: Uninitialized string offset: 0 in local__genotype_matrix_link_formatter->view() (line 45 of /var/www/html/sites/all/modules/nd_genotypes/includes/TripalFields/local__genotype_matrix_link/local__genotype_matrix_link_formatter.inc).

Projects need a special genus property set to work with other modules in this suite.

My request/suggestion is that the loader simply associate the property with the project itself, as it should have all it needs (just the genus name), rather than requiring users to know to add the genus property

Basically #8 (for example, is_marker_of). Has your thinking on this changed over time? I think you'll find inserting the terms expected by this module is necessary for the Tripal badge ie #30 . You could keep the customization form, but inserting the default terms on install would be nice (and IMO necessary badge requirement).

worst case, how about just including a PHP script that inserts the expected terms?

• Tripal Version: v3
• Drupal Version: 7.69
• PostgreSQL Version: PostgreSQL 13.0 (Ubuntu 13.0-1.pgdg18.04+1)
• PHP Version: 7.3.23-1+ubuntu18.04.1+deb.sury.org+1)

Hey everybody,

I am trying to load in the sample files that comes with genotype loader, but I get the following error:

This is my Tripal content type:

This is my chromosome Tripal content: