Currently, we only support variant data derived from VCF, in either vcf.gz or wormtable format. We need to support other types of data as we move to support the full API. I suggest we use a designated file system layout for the data directory to discover the files and their relationships. An initial pass at this might be
dataDir/
datasetId1/
variants/
variantSetId1/
chrom1.vcf.gz
chrom2.vcf.gz
variantSetId2/
variants.wt
reads/
[Some directory structure laying out BAM/SAM files]
references/
[Some directory structure laying out references]
dataSetId2/
variants/
....
reads/
....
references
That is, within the data directory, each directory d
corresponds to a Dataset
with ID d
. Within a given dataset, we have three directories, called variants
, reads
and references
. Within the variants
directory, we then have more directories, one for each VariantSet
. Within a VariantSet
directory, we have a bunch of files that define these variants in what ever formats we support.
For data within a given variant set, we could follow the existing convention used for the TabixBackend: we can split the variant set over as many files as we like, so long as we don't have data for the same chromosome in two files. I've deliberately mixed up wormtable and VCF format files in here. The input should detect the file type and make an instance of the appropriate handler based on the extension. If we don't support a given file type then we complain and quit. This would get rid of the currently very clumsy model of specifying wormtable
or tabix
as command line arguments to determine the type of backend to allocate.
We'd then just call the server using ga4gh_server dataDir
for development and testing purposes. For deployment an absolute path to the required data directory in a config file would be all that is needed.
The idea is that we use the file system hierarchy to explicitly set out the structure of the data to be served. This keeps the number of configuration variables we need as we deal with more complex datasets down. The downside is that we do require a pretty complex file system hierarchy to specify the data to be served. One way we could alleviate this problem is by providing a tool to check the layout of the hierarchy and to raise errors if it's not well formed. This could also print some summaries about the data that's in there too, so that people can get an overview of what they're about to serve.
I've not filled out the reads and references section, as I don't understand this side of things all that well. Will this sort of decomposition make sense here too? Does this hierarchy capture the structure of the API, or is it too restrictive? Any feedback would be much appreciated!