opencb / opencga Goto Github PK

DBObjectToVariantSourceConverter complains when a vcf has a line in the header like:

##source_20120715.1=...

because mongo doesn't allow dots in field names, so they are not loaded. Maybe it's interesting adding a char mapping in the converter.

In order to keep the storage framework independent of OpenCGA Catalog and therefore to be reused in other existing platforms, a mechanism to pass the study configuration with files and samples is needed.

Currently, the MongoDB schema for variants is file oriented. The actual mongo document looks like this:

{
  "_id" : "22_1234_A_T",
  "files" : [ 
    {"sid" : "1000g", "fid" : "f1", "samp": {"def" : "0|0", "0|1" : [254, 623] } , "attrs":{ } ... }, 
    {"sid" : "1000g", "fid" : "f2", "samp": {"def" : "0|0", "0|1" : [54, 78 ] } , "attrs":{ } ... } 
  ]
}

This can be a constraint when the samples of a study are provided separately, and want to query all samples from different files. Files should be merged in a single element in the array and look like this:

{
  "_id" : "22_1234_A_T",
  "studies" : [ 
    { "sid" : "1000g",  "gt": {"0|1" : [54, 78, 254, 623] }, ... } 
  ]
}

Because can happen that in different files, the default genotype is not the same, this default genotype must be the same for the whole study. This won't compromise the size of the collection, because the default value use to be the same in the 99% of the documents. The only consideration to keep in mind is to decide which is this default genotype. It depends on the polyploid of the specie and if the variants are or not in phase. The first approach will require this information from the user. In the future, this default genotype should be inferred by studying the files.

In this scenario, the "attrs" field, witch contains the vcf "info column" will disappear. This is a very unused fields, and they tend to be parsed into stats (#61) or into annot.
For times that this field need to be obtained, a Tabix access must be implemented.

Now, when a file is indexed from the Analysis layer, a new catalog file entry is created with type "index", to represent the indexation of the file in a specific Storage Engine. It works like a virtual index to the data in the storage engine. This file can be confusing for end users, and doesn't provide any interesting feature.

This catalog entry is a good approach with the current file oriented indexation, where you can index independent files, and then, fetch over different files. This is going to change. In the future, the files are going to be indexed in the study, and the queries are going to be done over the whole study, filtering by sample or cohort, not by file.

The indexation information must be stored in a new field on File.java. See Index.java

Create a new VariantVcfMongoDataWriter.

The VariantVcfMongoDataWriter inserts all the data into MongoDB.

Aggregated files are a complicated problem, due to the number of different ways that the aggregated data can be represented. A generic way to parse this data have to be implemented, in order to understand aggregated files.

Currently the variant transformation to the data model is executed in a single thread, this could be highly improved by making a multi-thread implementation.

Filesystem permissions must be implemented for both POSIX and HDFS filesystems. This will provide a more security if someone gains access.

If a PED file is loaded after the associated VCF file, the samples are already stored in catalog and thus rejected with a message like:

[main] ERROR org.opencb.opencga.app.cli.main.OpenCGAMain - Sample { name: 'NA12347'} already exists.

These existing samples should be detected and updated using the information from the PED file.

Separated class with methods for (at least) all CRUD operations for each resource type.
Also, add a CatalogPermissionException class for specific exceptions.

This class should contain all the permissions logic. Will manage Roles and ACLs.

In future versions, this class also will manage permissions at file system level.

Catalog tests are not covering most of the use cases and there are some dependencies between the tests. This must be redone to fully cover Catalog functionality

Current tests have many dependencies between them what makes difficult to test some specific aspects. Also, JUnit test exceptions frameworks should be used.

Stats calculation is done at the transform step. It must be moved to the postLoad step, and work over the loaded data.

CatalogManagerException should be renamed to CatalogDBException to reflect properly that the exception comes from the database

Some third party dependencies such as Guava, SLF4J, SQLlite, ... are quite old

Currently serializing data model with gzip is a bottleneck during the variant transform. Snappy offers a much better performance.

Some changes need to be implemented to allow cohorts and sample annotations. The API needs to be completed with many new methods.

Genotypes are returned from the database always using numeric indexes for the alleles. It would be interesting to allow more flexibility and change the representation to nucleotides on demand.

When running the file creation tool, even when it finishes successfully it display this message about the checksum:

[main] INFO org.opencb.opencga.catalog.CatalogFileManager - Coping file from file:///home/cyenyxe/Templates/chr22_10K.vcf.gz to file:/home/cyenyxe/.opencga/catalog/users/biouser/projects/1/2/chr22_10K.vcf.gz
[main] INFO org.opencb.opencga.catalog.CatalogFileManager - Checksum matches null

OpenCGA lib is more a core library to OpenCGA than a lib for others. This should be renamed to make this clear.

The output of the accessioning tool should be compressed.

Currently only GET methods exists to create single elements. HTTP POST method would allow to get a JSON array of Studies with nested fields (e.g. files in studies).

A HDFS Manager implementing IOManager API is needed to offer the same functionality for Hadoop based installations

Use Bulk Operations and threads in order to speed up the load variants operation.

The current 'master' branch has dependency on

<cellbase.version>3.1.0-rc</cellbase.version>

• maven central shows version 3.1.0 as latest available build.
• latest cellbase repo shows version 3.1-SNAPSHOT as the current version

Having problems building the project / 'master' branch due to this unresolved dependency.

If an empty (not-null) studies list is provided as argument, it is not ignored in the VariantMongoDBAdaptor code, so it parses it as a study with name "" and nothing is returned as result.

OpenCGA depends on variant-lib for the VariantRunner. This variant-lib repository is going to be removed so this dependency must be removed.
Current variant-lib in code in use must be either taken to biodata or to OpenCGA

When the study type is not specified in the study creation tool, the following error is shown:

[main] ERROR org.opencb.opencga.app.cli.main.OpenCGAMain - Name is null

Which makes a bit confusing to trace the problem.

Storage CLI lacks many options and need to be redone and completed. Currently it is hardly usable.

Stored variants must accept multiple rs/ss IDs instead of just one, as it currently happens.

Both in SQLite and Monbase

The description of the --aggregated option is not clear enough. The supported values, as well as the default, must be listed when the help shows.

This may also apply to other arguments such as the variant study type.

Concatenation of _partial files should be done by the daemon (or, in the future, by the workers).

When lastChunk arrive, file.status should change from "uploading" to "uploaded".
Then, the daemon should concatenate, calculate checksum, move to the final destination and, then, set status to "ready".

When multiple instances of OpenCGA's variants loading tool are processing the same chromosome from different studies at the same time, inconsistencies may arise. This is probably due to checking only once whether a variant has been already saved. This condition should be checked not only at the beginning of the DBObject generation, but again right before the insertion.

getAllVariantsByRegionAndStudies should return QueryResult<Variant>. Now the function returns QueryResult<DBObject>

https://github.com/opencb/opencga/blob/develop/opencga-storage/src/main/java/org/opencb/opencga/storage/variant/mongodb/VariantMongoDBAdaptor.java#L74-L89

Use SqliteCredentias instead of dbName.
Use SqliteManager for connections. Remove the actual code for connections.

Implement a method that retrieves a Variant object, with optional VariantStats and VariantEffect.

It's time to migrate java

After April 2015, Oracle will no longer post updates of Java SE 7 to its public download sites.

The referenced files are in a user home directory and can't be found in any other machine. They must be loaded using the getResource() method. An example can be found in opencga-storage, in the class VariantMongoDBAdaptorTest.

Also, asserts must be used instead of System.out.println.

Some studies can't be loaded into Mongo because their samples contain dots in their names, but Mongo won't allow them as keys. Dots should be replaced with a rarely used character ($, £, whatever we decide) and shown properly when they are returned as data model objects by the DB Converter.

A variant annotation pipeline and CLI need to be implemented to allow users to annotate variants using CellBase Variant Annotation

Old web services implemented in the serverold package in opencga-server are not longer used and should be removed.

This classes should accept a QueryOptions param with "exclude" or "include" with a List of fields path.

This fields can be relative to the returning element or absolute to the user. e.g:

getStudy( <ID>, QueryOptions{ include: ["alias"]} )

vs

getStudy( <ID>, QueryOptions{ include: ["projects.studies.alias"]} )

To unify the fields name between all the methods, it's recommended to use an absolute field path

Old OpenCGA Account module is not longer used by any live project, this Account module has been superseded by OpenCGA Catalog

The referenced file is in a user home directory and can't be found in other computers. It should be loaded using the getResource() method. An example can be found in opencga-storage, in the VariantMongoWriterTest class.

Also, asserts should be used instead of System.out.println.

The output from the samples/search tool is something like the following: http://pastebin.com/3gYL2X6J

APIs such as Python's complain when trying to parse this file. A valid JSON file with multiple documents must represent them as a comma-separated list surrounded by square brackets.

(There is a workaround which is using PLAIN_JSON instead of PRETTY_JSON as output and then parsing each line separately, but still it is not very convenient)

Add new factory for aggregated VCFs.
This factory will transform aggregated vcfs into Variant objects.

Variant effect is properly saved when indexing using SQLite, but not in HBase.

Stats per variant are calculated over the whole set of samples. It's needed to allow define different sets of samples (cohorts) to calculate other stats.

Slurm is the mos open source resource manager used, we currently support old SGE but not Slurm. We need to provide equivalent functionality for both of them