We've ran into an issue when registering large VCF files from Ensemble. Files sizes varies from 500Mb to 4Gb.
At some point NGB crashes with OutOfMemory exception.

Here are example files from Ensemble FTP:

• Homo_sapiens.vcf.gz ~3.0Gb: ftp://ftp.ensembl.org/pub/release-82/variation/vcf/homo_sapiens/Homo_sapiens_incl_consequences.vcf.gz
• Mus_musculus ~1.9Gb: ftp://ftp.ensembl.org/pub/release-82/variation/vcf/mus_musculus/Mus_musculus_incl_consequences.vcf.gz

I'm running NGB from docker container on laptop to work with my own data. So, I've mounted my data folder to docker as :/ngs
But it looks complicated to register all files I want to view in NGB. In addition, running NGB cli inside docker container was tricky to me first time.
I suggest changing default behavior of NGB server in docker container to access all data in mounted folder without registering.

1. select dataset that have indexed GTF/GFF3
2. find OR6K1P gene
3. click gene and click "Show info" point in appeared menu
4. go to "UNIPROT" tab
   exp.: handlen and userfriendly error message
   act.: see screenshot
   

Initial situation:
• CentOS Linux release 7.3.1611 (Core), Total RAM: 3780884
• There is running /opt/ngb/catgenome-2.5.jar (loaded from http://ngb.opensource.epam.com/distr/2.5/, file size – 93566412, file date - Jul 5 12:07) with some registered datasets.

Version upgrade actions:
• Run /opt/ngb/catgenome-latest.jar (loaded from http://ngb.opensource.epam.com/distr/latest/, file size – 93531271, file date - Sep 28 01:09). I.e. running new NGB version in the same folder, over existing H2 and contens folders/files

Result:
• ERROR [28/09/2017 18:18:33][main][org.springframework.boot.SpringApplication] Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [conf/catgenome/applicationContext-flyway.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Validate failed. Migration Checksum mismatch for migration 2016.11.21.17.58

Output and log files attached.
2017-09-28 NGB run fail output.txt
catgenome-error.log

If partucular sample has field ExAC_AF, in variation table it is displayed somewhat incorrectly
Also sorting by this field doesn't work.

run_CM.sample_22.zip

Background

Currently BAM coverage track can show colored bars with mismatches distribution at a specific locus

These bars get colored only if a certain threshold is exceeded. This threshold is currently hardcoded. Sometimes it is necessary to view mismatches at lower thresholds.

It will be great to allow user to configure this threshold.

Requirements

1. Add "Allele frequency threshold" setting to NGB global settings: "Settings" -> "Alignments" -> "Coverage Options"
   

2. This setting shall be always initilized with default value (current default)

3. The value of this setting shall be used to decide - whether to color coverage track bar

4. Once a value is changed - BAM tracks shall be redrawn to reflect the changes (same as other BAM specific settings)

Personal communication with Scott P Myrand (ThermoFisher)
1 Add ENSEMBL or REFSEQ ID info to the transcript viewer
2 Add the reference build info (GRCh37/hg19 or GRCh38/hg38)
3 Fix the bug that mixes up the domains (“forget” to reverse AA numbering on the reverse strand). ROS1 domain reversed in example
4 Exon Deletions, skipping, or non-canonical Exon inclusion

NGB produce very good paper-quality pictures, but can export in only to raster format, which is not favorable for preparation of papers.
I suggest implementing image export to some widely-used vector format (preferably pdf or svg).

On the BAM coverage window NGB displays coverage in gray only if all letters at this position are the same as in reference. If there are mismathes, NGB displays each letter in their own color.
On the example below there are no variation at position marked by arrow (low frequency of alternative letters, red is actually reference letter "C"). But in coverage graph it looks wery similar to real variation.

I suggest that reference letters on coverage grach shoul be displayed by gray color (as on the picture below)

At the moment NGB displays variation parameters only form INFO column of VCF file.
Data form FORMAT column (set of values separated by ":" symbol ) are not displayed.
I suggest that NGB should parse these data and display it in variations table and variation info window in addition to data from INFO column.

Here is one variation form VCF file:
chr1 78392446 rs1166698 G A 56770.20 PASS AC=1;AF=0.500;AN=2;ESP6500_AF=0.15781;ExAC_AF=0.184;FS=0.000;GENEINFO=Nexilin;MQ=59.99;PM;QD=14.59;SNP;VARTYPE=SNP;VQSLOD=5.84;X1000Gp3_AF=0.150759;dbNSFP_PROVEAN_pred=N,N,N,N,N;dbNSFP_Polyphen2_HVAR_pred=D,D,D,D;dbNSFP_SIFT_pred=D,D,D,D,D;set=foo GT:AD:DP:GQ:PL:PP 0/1:240,239:479:99:7418,0,6904:7414,0,6914

FORMAT column describes the order of format fields (GT:AD:DP ...) and next column contains values for particular sample.
GT:AD:DP:GQ:PL:PP 0/1:240,239:479:99:7418,0,6904:7414,0,6914
So, this data should be read as GT=0/1; AD=240,239; DP=479, ...

In the following section in ngb-cli man page:

FILE commands:
rf	ref_file	: registers a feature file for a specified reference	{rf grch38 \path\to\file.bam?\path\to\file.bam.bai -n my_vcf}
df	del_file	: deletes a feature file one	{df my_vcf}
if	index_file	: creates a feature index for a file. 	 {if genes.gtf}

shouldn't ref_file be reg_file ?

WIG track won't show anything if names of chromosomes into the fasta and the bedGraph files don't match.
So instead of current behavior new logic should be implemented:
According to new logic we have to assume that names of chromosome could be presented in two ways:
names: 1 2 3 4 5 6 ...
names: chr1 chr2 chr3 chr4 ....

Having the same configurations as in #5, we're unable to retieve list of loaded gff files via ngb s -l .gff, instance just stops responding to any kind of requests.

1. register .bw file, in this file should be absent data for one or more chromosome
2. add this file in dataset
3. open dataset
4. open browser console
5. select chromosome which absent in .bw
   expected: no errors, load indicator is disappearing during seconds
   actual: load indicator isn't disappearing.
   error in console
   {"message":"Parameter specified as non-null is null: method org.jetbrains.bio.big.BigFile.summarize, parameter name","status":"ERROR"}
   see server log file
   ngb_server_log.txt

Add a vertical line which illustrates the centre of the visualized region on each track (not only on BAM).
It will be useful in cases of compare some tracks. Its very helpful to see if peak summits line up on WIGs tracks.

When building NGB from master branch - build is errored. See details below:

Build command:
./gradlew buildJar

Output:

What went wrong:
A problem occurred configuring root project 'ngb'.
> Could not resolve all files for configuration ':classpath'.
   > Could not find trilead-ssh2.jar (com.trilead:trilead-ssh2:1.0.0-build221).
     Searched in the following locations:
         https://plugins.gradle.org/m2/com/trilead/trilead-ssh2/1.0.0-build221/trilead-ssh2-1.0.0-build221.jar

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

We still observe low performance while loading large vcf files (same files are used as in #16)

It takes up to 10 seconds to load variations tables without any filters.
3+ simulteneous requests cause NGB to hang.

Currently "Open from URL" menu allows to open only VCF and BAM files.

If we try to open BED/GFF/GTF files - LOAD button is grayed out and does not allow to load that file.

Seems that this bug is just a client-side check for vcf/bam files. So the list of supported files shall be extended to all other files extensions, that are supported by server. @mzueva could you please provide such a list?

Background

Currently reference track can show only forward strand.

Sometimes it is necessary to view reverse strand and all variants of sequence translation.

It will be great to allow user to configure reference track displaying.

Requirements

1. Add "General" hyperlink on reference track header. In case when user click it menu is appeared.

2. "Show forward strand" is checked by default. "Show translation" is unchecked by default.

3. In case when is checked "Show translation" point shall be displayed all variants of translation. Aminoacids are counting from first, second and third letter of chromosome.

4. User shall have possibility to check "Show forward strand", "Show reverse strand" or both of them. User shall not have possibility to uncheck both of them. Reverse strand sequence is displayed above the forward strand.

5. In case when user is checking "Show reverse strand" on reference track is appearing second sequence, that's mean there is displaying reverse strand that complimentary to forward strand.

6. In the header of reference track should be displayed label that indicate which mode is enabled. For example if enabled "Show reverse strand" and "Show forward strand" mode in the header of reference track is displayed - General (Forward strand, Reverse strand), if enabled "Show translation" and "Show forward strand" mode in the header of reference track is displayed - General (Forward strand, Translation)

7. Once a settings is changed - reference tracks shall be redrawn to reflect the changes.

8. All modes for reference track should be configurable from URL. This param should be stored to a local storage.

Legend:

- A (Adenin)
- C (Cytosine)
- T (Timin)
- G (Guanin)
- STOP codon
- Methionine (abbreviated as Met or M) START codon
- Amino acid, letter is 1-letter name

Algorithm of amino acids calculation

For firs amino acids track

1. divide first coordinate on 3, if it is more or equal 3
2. if value of first coordinate mod 3 = 0, then start translating triplets to amino acids from first coordinate
3. if value of first coordinate mod 3 = 1, then start translating triplets to amino acids from value of first coordinate - 1
4. if value of first coordinate mod 3 = 2, then start translating triplets to amino acids from value of first coordinate - 2

For second and third tracks, translate triplets to amino acids, with shift on one and two letters respectively.

In VCF info table fields with too long names overlaps with data.
I suggest adding hyphenation or some adaptive scaling both for fields captions and for fields data.

At the moment on the variation info windows ANN field is collapsed by default and can be expanded by click. But visually this ability is not emphasized.
I suggest adding some symbol that indicates that ANN field is expandable. The '+' or arrow down should be fine.

Steps for reproducing:

1. Click the "Open" link on main toolbar and select "from URL"
2. Enter the valid link to the "File path" field
3. Enter the same link to "Index file path"

Expected: Checking the index file for validity and throwing the exception "The index file is invalid"
Actual result: NGB tries to display the file

NGB allows to use specified value for reference sequence name through-n (--name) [value] option. But if one registers file using number as name, file registration shall not work with number name reference.

A the moment variaion info window displays all avaliable data from VCF record, which is often overkill.
I suggest that VCF info window shold have ability to select displayable fields, similar to variation table.

When folder with vcf file to be registered contains GATK index (file with the same name and "idx" extension), NGB doesn't register this file and reports Vcf.idx does not match its file format [VCF]. After deletion of GATK idx everything works fine.
I suggest that NGB just ignore index with illegal format if it is not necessary to register file.

Here is archive with sample files:
run_CM.sample_22.zip

Is it possible to load BAM files directly from AWS s3

If the variation has rsID we can get information about variation from dbSNP
(http://www.ncbi.nlm.nih.gov/projects/SNP/):

Here is a response example for variation rs7412:
Human-readable: http://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=7412
XML: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&id=7412
Detailed XML: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=snp&id=7412&report=XML
JSON: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&id=7412&retmode=json
Clinvar JSON: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=clinvar&id=7412&retmode=json

I suggest that for variation with rsID NGB should have Variation ID as a web-link to dbSNP and a separate info tab "dbSNP" (similar to "NCBI" and "Ensembl" tabs for genes. On clicking on this tab NGB should download and display the following information:

1. Organism
2. Map to Genome Build
3. Variation Class: <Item Name="SNP_CLASS"
4. RefSNP Alleles:
5. Clinical Significance: <Item Name="CLINICAL_SIGNIFICANCE"
6. MAF/MinorAlleleCount
7. MAF Source:
8. HGVS Names: <Item Name="DOCSUM"
9. From table with variation location (Primary Assembly Mapping ):
   a) Assembly (reference name)
   b) Chr
   c) Chr Pos
   d) Contig
   e) Contig Pos
10. From table RefSeq Gene Mapping:
    a) RefSeqGene
    b) Gene (ID)

At the moment two rulers - local and global can confuse user since they are rather small and not contrast enougn. So user can't easily decide where to drag to select region.

To make local ruler move visible and simultaneously avoid wasting of screen space I suggest merging local ruler with reference track. Mouse gragging on this track should select region to zoom, as for rulers.
On all other tracks mouse behavior should remain the same -- dragging moves tracks.

Also it would be useful to shmewhat separate local and global rulers

Something like this:

When user investigates variations from some target panel he usually analyses only specific set of genes.
Filtering variations by gene in variation table is a great feature but it's somewhat annoyng to manually select 25+ genes every time when starting NGB session.
It would be fine to allow user to create custom list of genes to select/deselect it all by single click.

I suggest that we should have two sections in gene filtering dialog: genes and custom gene lists.
Something similat ro this:

General NGB interferface has low contrast, too few vivid elements, which cause issues with tracks operation.
On screen zooming and screenshot buttons are barely visible.
Tracks areas are too close to each other and tracks borders has low contrast.
I believe NGB should significantly improve contrast and highlighting of visual elements.

Some mockups:

Variant quality QUAL could not be found in the filters nor built-in columns, it would be handy to have it as filter for prioritization.
Attached is the screenshot of the demo site http://ngb.opensource.epam.com/catgenome (sv_sample1 dataset).

Also, many columns were sorted as String (rather than numeric, despite the type definition in the .vcf file - taking example the Svlen field from the demo site (sv_sample1 dataset). Other fields also seems to be treated as String; that will make quality filtering and sorting problematic to user.

Found this issue locally, on the latest tag v2.5 (built from source code)

Complex analysis (long-path connectivity , etc) of data which aggregated from some different databases (UniProt, KEGG, SMPDB, etc) can help to find new nontrivial dependencies between different concepts, and provide information for new hypothesis. It may be useful for end users and for researchers.

Examples of projects in this area:

• Grinn project
• Biomine project
  • Paper on graph analysis: Sevon P., Eronen L. Subgraph queries by context-free grammars //Journal of Integrative Bioinformatics (JIB). – 2008. – Т. 5. – №. 2. – С. 157-172.

Great part of all of this projects is creation of graph data base which aggregate data from many other data bases. Seems, that creation of access point to actual aggregated data is a complex technical task (different data format parsing, optimal storage providing, etc) which can be done once.
May NGB be a base for unified access to different existing DBs, or host for data aggregation?

i have bam file and vcf file, i want to visualize vcf , is it possible in NGB?

It seems that "screenshot" feature is broken in 2.4 LTS version.
Screenshot is being saved, but contains only tracks names without actual visualization of track data

Here is an example

Currently ngb index rebuilds only lucene index.
But in some cases it is useful to regenerate all other index stuff. E.g. histograms, bound for genes files

This is something that was previously done as a PoC in 2.4-LTS_full_reindex_for_gene_files. But it shall be refactored and merged into 2.4-LTS and develop

In addition - ngb index command shall accept --no-tabix option, when this option is set - .tbi index shall not be rebuilt. (This shall be added to CLI reference)

By default NGB works using http connection.
After manually switching to https, NGB cli utilities fails to connect to the server. There are no ability to set up authentication in NGB cli.
Please, implement and document NGB usage with https.

NGB allows to configure a path where all caches and indeces are stored using files.base.directory.path in a configuration files

But when files are registered - absolute path is written to the index DB, thus changing files.base.directory.path value - results into FileNotFound errors

It will be great to:

• Store files paths in a DB using relative paths (relative to files.base.directory.path)
• Full path shall be resolved when a file is requested
• Backward compatability shall be provided: if a file in a DB is absolute (e.g. starting with /) - it shall not be resolved using files.base.directory.path

BLAT is an annotation utility which allow user to efficiently search for a potential location of particular DNA sequence. UCSC provides web service for BLAT search.
IGV has ability to submit a prticular read sequence to BLAT and view results.

NGB should have the same functionality
By clicking on each read and selecting "BLAT sequence "user shold be able to send read sequence to BLAT. The search results should be displayed in additional popup window or tab, similar to IGV.
By clicking on the each line user can navigate to specified location (chromosome and position from start to end).

The results of each BLAT search are represented in BLAT as a BED track, where each region is a potential location reported by BLAT.
I suggest that user should have and ability to hide BLAT results window/tab and show it again from results track.

Hi, I noticed that all variants in the Variants panel doesn't display any value under "Gene" column, however in the filtering popup it shows Ensembl gene id. Also all my variants cannot be visualized.
It seems there is problem on looking up the gene information (or gene name?) but it seems it associates with gene ID fine underneath (as shown in the INFO).

What would be the possible causes? I used vcf from Gridss and Manta, both having the same problem; is the vcf need to have specific field to get this gene annotation appear? (eg. does it relies on the ANN field in vcf to get the gene name?)

I'm using GRCh37 reference, using with gtf from GENCODE. I ran vcf through snpeff 4.3 to get ANN annotation with Gene Id and Name, however the gene name and sv visualization still not working.

Thank you

I just want to say that NGB is a great genomics browser. I was able to quickly deploy the application as a docker container.

This is not so much of an unresolved issue but rather a proposed solution to an issue I encountered when deploying NGB in a docker container. One thing that was not clear from the documentation is how to persist data (reference, gene, bed, etc. registrations and datasets) between container sessions.

I was able to come up with a solution that seems to work. Essentially, both the H2 and contents folder under /opt/catgenome have to be locally stored on the host and mounted as volumes when initally using "docker run." Here is the command I used:

docker run --privileged -p 8080:8080 -d --name ngb -v /host/ngs:/ngs -v /host/H2:/opt/catgenome/H2 -v /host/contents:/opt/catgenome/contents lifescience/ngb

So in our setup:

/host/ngs is where the bam and bai files live. We have it mounted via sshfs which is why "--privileged" is needed in the docker command.

/host/H2 is where the NGB database will be persisted on the host.

/host/contents is where the NGB registered files will be persisted on the host.

After stopping the docker container or a host reboot, launching "docker start ngb" will load up from where it left off with persistent contents and datasets.

Hopefully this helps others who want to deploy as a docker container.

Some of the users suffer npm install failures. Almost all of the reference phantomjs dependency problem. As we do not use phantomjs for tests automation, seems it comes as nested dependency. So let's try to get rid of it

Hi there,
If I load a structural variant vcf file using the GUI it seems like the fusion genes do not get recognised. What is shown is no affected gene for both.

Hello,

Great tool..... Is it possible to add reverse strand to REFERENCE sequence? I would also like to add translation in first frame on forward sequence.

Many thanks.

Let's add this project to openhub's EPAM org to increase the visibility and provide some key metrics about it.

When NGB receives too many "huge" requests, it tries to fulfil them all. This results into loss of reponsiveness for minutes/hours for all users.

We shall introduce timeouts for the API methods. If a request can not be fulfilled in a certain amount of time - HTTP 503 shall be returned.

Also a number of APIs shall not be wrapped with timeouts, such as registration API.

Let's begin from the most computation intensive methods, that are listed below. Once those are done and working, we can proceed with others.

• /filter
• /filter/group
• /reference/loadAll
• /project/tree
• /bam/track/get
• /gene/{id}/track/get
• /wig/track/get
• /vcf/track/get

Timeout setting shall be defined in .properties with default value 10 sec

This feature shall be introduced in both 2.4-LTS and current development branch develop

File name is not fully displayed on a track header. It would be nice if users could see the full file name when they mouse over the bam file name.

#17 #18

1. open NGB
2. click "SETTINGS" icon
3. click "VCF" tab
4. enter negative number in "maximum variants range(bp)" field
5. click "SAVE" button
   expected: warning message
   actual: see screenshot
   

Having the following configuration:

1. ~50k genomes registered
2. ~100k datasets/files registered for the genomes above

Each NGB load takes unacceptable amount of time to load datasets tree, stats are below:

1. Versions used:
   1.1 2.4.2, 2.5.0 (pre-release)
   1.2 Distribution used: Spring boot jar
2. Hardware:
   2.1 CPU: 1 core
   2.2 RAM: 2 Gb
3. Request used: load 2 tracks from a dataset, with a filtered datasets tree (&filterByGenome=)
4. Timings of the network requests series is below:

5. Having these stats, the following conclusions can be made:
   5.1. (HIGH) Double loadAll for the large datasets lists - fix: shall be only one call
   5.2. (HIGH) Do we really need to load all genomes? Shall fix to load only needed genomes
   5.3. (HIGH) Fine-tune springboot configuration for better static content serving
   5.4. (MED) Single dataset filter takes 2 seconds, which does not look reasonable - shall be optimized

6. Datasets - related issues shall be fixed in hotfix/datasets-performance

7. TODO:

• @rodichenko , 5.1 and 5.2 are for you to check
• @mzueva , please look into 5.3, 5.4

Implement sevaral possible coordinate formats for location:
Implemented ones:
chromosome:start-end
chromosome:location - place center of the screen to location

Suggested ones:
chromosome: - navidate to the whole chromosome (minimum )
start-end - navigate to region on current chromosome (only if chromosome already selected)
location - - place center of the screen to location on current chromosome

CLI tests currently fail on Windows machines due to problems in relative path resolving. It seems that this issue comes from changes in CLI in July 2017.