genome-spy / genome-spy Goto Github PK

A visualization grammar and GPU-accelerated toolkit for genomic data

License: MIT License

JavaScript 74.07% GLSL 2.47% Python 0.63% HTML 20.74% TypeScript 0.02% SCSS 2.05% Shell 0.02%

bioinformatics cancer-genomics data-visualization genome genome-visualization graphics visual-analytics visualization webgl

genome-spy's Introduction

GenomeSpy

GenomeSpy is a visualization toolkit for genomic (and other) data. It has a Vega-Lite inspired visualization grammar and a high-performance, WebGL-powered graphics renderer.

The software is still work in progress. Documentation and examples for the current version can be found at https://genomespy.app/

Monorepo

GenomeSpy is split into several packages, two (core and app) of which are the most important:

Core

The core library provides the visualization grammar and a WebGL-powered rendering engine.

Cohort App

The app builds upon the core, extending the visualization grammar with support for faceting multiple (up to thousands of) patient samples. It provides a user interface for interactive analysis of the samples, which can be filtered, sorted, and grouped flexibly. A session handling with provenance, url hashes, and bookmarks is included.

Embed Examples

The embed-examples package contains examples of how to embed GenomeSpy in web applications and use the API for advanced use cases.

Contributing

Bootstrapping and running

git clone [email protected]:genome-spy/genome-spy.git
cd genome-spy
npm install (use npm7!)
npm start (starts a development server with the app package)

The packages/core/examples directory contains some random view specification that can be accessed through urls like http://localhost:8080/?spec=examples/first.json.

The packages/core/private/ directory is in .gitignore and served by the development server: http://localhost:8080/?spec=private/foo.json. Use it for experiments that should not go into version control.

If you want to use or develop the core library, launch a single-page app using: npm -w @genome-spy/core run dev

Contributing guidelines

Please see the CONTRIBUTING.md file for more information.

About

GenomeSpy is developed in The Systems Biology of Drug Resistance in Cancer group at the University of Helsinki.

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 965193 (DECIDER) and No. 847912 (RESCUER), the Sigrid Jusélius Foundation and the Cancer Foundation Finland.

Contains some code copied and adapted from the following projects:

Vega and Vega-Lite (LICENSE)
TWGL (LICENSE)

genome-spy's People

Contributors

Stargazers

Watchers

Forkers

alexpreynolds firemax19 cqh189 khurrammaqbool khurram-maqbool theitalianjob eddygenome trentfridey razsultana

genome-spy's Issues

Feature: observable scales

The GenomeSpy API should provide means for accessing scales and observing their domains. This could be used for things like synchronizing the viewports with other components on the web page or triggering some dynamic data loading.

GenomeSpy uses the scale resolution logic introduced in Vega-Lite. Thus, the scales are created implicitly and possibly shared between the views. The ScaleResolution objects keep track of the participating views and provide some convenience methods for working with the underlying d3 scales, e.g., zooming and panning. The ScaleResolution class already has an addScaleObserver method, which GenomeSpy uses internally. That together with the removeScaleObserver could be included in the public API.

Accessing the ScaleResolutions

Because the scales (and ScaleResolutions) are created implicitly, they have no identifiers that could be used to access them. Some possible solutions:

Access the resolution by the view path

Because views form a hierarchy, they could be accessed using the view path and the channel:

getScaleResolutionByPath(["root", "first_track"], "x")

However, GenomeSpy sometimes modifies the view hierarchy by adding decorators, etc., and figuring out the actual path may be tricky for the user.

Access the resolution by the view identifier

All the views have names, which are either explicitly specified or implicitly generated. However, the names are not required to be unique. A new property, namely an id could be added, which should be unique within the view specification:

{
  "vconcat": [
    {
      "id": "myView",
      "mark": "rect",
      ...
    },
    ...
}

Then the resolution could be accessed using a new method: getScaleResolutionByViewId("myView", "x")

Or alternatively: getView("myView").getScaleResolution("x")

However, I'd like to keep the public API quite flat and concise for now, and thus, prefer the former option.

Specify a name for the scale

Something like this:

"x": {
  "field": "foo",
  "scale": {
    "domain": [0, 1],
    "name": "myScale"
  }
}

Now the API could be used like:

getScaleResolution("myScale").addScaleObserver(scale => { ... })

Properties of all the merged scales must have the same name (or an undefined name). This could be a problem if visualizations are composed by including views (tracks) from external specification files.

Scales with `scheme` are sometimes discordant on GPU and d3

For example, the color in a plot (GPU) and a tooltip (d3) disagree.

Firefox 99.0 (64-bit) Ubuntu webGL error.

Encountered an error when opening up a visualization with firefox browser..

At "viewRoot/concat0_decorator/concat0/layer0": Error: Cannot create shader program: 0(26) : error C7532: global function uintBitsToFloat requires "#version 330" or later 0(26) : error C0000: ... or #extension GL_ARB_shader_bit_encoding : enable 0(26) : error C0000: ... or #extension GL_ARB_gpu_shader5 : enable

Stroked rect marks

Implement stroked rect marks and the new properties: filled, fill, stroke, fillOpacity, and strokeOpacity.

Stroke dashes need not be supported now because they are a bit tricky (albeit possible) to implement for stretchable rectangles using GPU.

The strokes can be implemented either using a vertex shader (new vertices for the strokes) or a fragment shader (something like distance fields). The former approach is more efficient for large fully transparent rectangles, the latter one is better for (a high number of) small filled rectangles.

Accept single-base coordinates in the location/search field

Not only chrom:startpos-endpos but also chrom:pos.

Implement `zindex` for stuff handled by GridView

Would allow ordering axes, titles, and the actual views

Migrate from Jest to Vitest

https://vitest.dev/

Jest needs Babel to properly support ESM. I'd like to get rid of it.

Add "Harmonize groups using the yyy attribute" action to nominal attributes

Use case: The user has grouped the samples into n groups based on some criteria. However, they would like to have exactly the same set of patients in all groups.

This could be used in comparison of different treatment phases, tissue sites, etc.

Naive dynamic data loading

Because a proper implementation of #109 needs quite a bit of work, a simpler solution could be implemented for urgent needs.

GenomeSpy already internally supports dynamically updated data. For instance, axis ticks and the prioritised gene symbols are updated dynamically.

Thus, the naive solution would work as follows:

Observe zoom/pan events by registering a listener to a scale resolution: https://genomespy.app/docs/api/#api_getScaleResolutionByName
Write an application-specific listener function that fetches data. Use whatever means to do that.
Feed the new data to GenomeSpy through named data: https://vega.github.io/vega-lite/docs/data.html#named

This feature mainly needs finishing the implementation of named data. GenomeSpy alredy has some internal named data support, but it's undocumented and no API is exposed.

Enable GENCODE annotations

I'll write more detailed issue at a later time. But briefly...

Implement some form of lazy loading: #109 or #111
Implement a transform that splits GTF's key-value column into objects. Or alternatively, a GTF3 loader could do it.
Implement support for more complex data flows. Currently, each flow node has a maximum of one parent and each data flow has only a single data source.
Implement a "lookup" transform that joins data from two flow nodes using a key. Something similar to: https://vega.github.io/vega-lite/docs/lookup.html

The required data flow looks roughly like this:

Filter the annotation and retain transcripts.
1. Compute a layout using the "pileup" transformation
2. Publish it as piledTranscripts
3. Display using "rule" mark
Filter the annotation and retain exons, UTRs, etc.
1. Join the data flow to piledTranscripts using transcript_id
2. Display using a suitable mark, probably "rect"

Fix remaining typings and check them using `tsc --no-emit`

Do it for:

core
app

Incorrect canvas height on ObservableHQ

The height of the canvas seems to fluctuate randomly: https://observablehq.com/@tuner/ascat-copy-number-segmentation?collection=@tuner/genomespy

Not sure what's going on.

Split GenomeSpy into two packages: the core vis. library and the App for exploring large cohorts

This is getting a bit fat and could be modularized.

The proposed packages

The visualization library (`genome-spy`)

Provides the visualization grammar and WebGL-based rendering
Basic support for genomic coordinates
Provides an embed method, which allows for embedding on web pages, Observable, etc.

The App (`genome-spy-app`)

Uses the genome-spy package
Provides sample faceting, i.e., the fancy stuff that allows viewing, filtering, sorting, and grouping of up to thousands of samples
- This is in practice implemented in SampleView
Provenance tracking for the `SampleView``
Bookmarking the visualization state
Updating the state to the url hash
The toolbar with a location/search field and buttons for the above
Introduces dependencies that are unnecessary in the core lib. For example: Redux, lit, idb, fontawesome, and lzstring.

Next steps

The App (https://github.com/tuner/genome-spy/tree/master/packages/genome-spy/src/app) is already decoupled from the core. They just need to be put into separate packages. The tricky thing is to figure out how to have a nice development experience in the monorepo, i.e., how to properly import stuff from the other packages without any build steps, etc., and how to ensure that debugging, "source maps", typings, and stuff others work.

Use WebGL 2 uniform buffer objects (UBOs) extensively

It's a great way to prepare for WebGPU: https://twitter.com/tojiro/status/1340503959321018368

... and also reduces the number of WebGLRenderingContext.uniform[1234][fi][v]() calls.

TODO:

Mark-specific props, such as those in "link" mark.
datums and values

Text effects such as outline and drop shadow

They can be used to enhance contrast on uneven backgrounds.

An example from Google Maps:

New marks: tick, bar, circle, and square

These are actually just new parametrizations of existing marks.

rect: bar
rule: tick
point: circle and square

Some refactoring needs to be done. Possible solutions:

Subclass the existing Mark classes
Implement support for composite marks. tick would be a composition of a single mark: 'rule', but with different (but dynamic) config

Calculate bounds for view titles and axes

I implemented view titles in #101, but bounds calculation is still missing.
Also, vertical axes don't take tick label lengths into account.

De-emphasize invisible sample-metadata attributes in tooltip

Attributes can be hidden, but it's probably good to have them in the tooltip anyway. However, they should be de-emphasized (shown in gray) in the tooltip, because otherwise it may be a bit puzzling for the user why some attributes are not in the heatmap.

Sticky tooltips

Use case: The user would like to copy something from the tooltip. Copying is currently impossible because text cannot be selected from the tooltip.

An alternative behaviour should be provided: the user clicks a mark instance and the tooltip becomes sticky, i.e., it doesn't move, update, or disappear before the user either: clicks somewhere outside the tooltip or clicks an X icon that appeared to the upper right corner of the tooltip. However, this behavior may be undesirable when a custom click handler has been registered and thus, an option for disabling the behavior should be provided. The default behavior could depend on whether a custom click handler has been registered or not.

API for registering custom tooltip handlers

GenomeSpy has an internal API for custom tooltip handlers. The API should be exposed to users.

Current handlers

Generic

Displays the fields and values in a table, formats numbers nicely, skips field names that start with underscore.

https://github.com/tuner/genome-spy/blob/master/packages/genome-spy/src/utils/tooltip/dataTooltipHandler.js

RefSeq gene summary

Fetches gene summary from Entrez.

https://github.com/tuner/genome-spy/blob/master/packages/genome-spy/src/utils/tooltip/refseqGeneTooltipHandler.js

Specifying the handler to use

The generic handler is used by default. The tooltip behavior can be adjusted using the "tooltip" mark property. null disables the tooltip, an object with a handler property allows for changing the handler.

An example:

{
  "mark": {
    "type": "text",
    "size": 11,
    "tooltip": {
      "handler": "refseqgene"
    },
    ...
  },
  ...
}

A full example is available on Observable: https://observablehq.com/@tuner/annotation-tracks

TODO

Design and implement the public API
Refactor the internals a bit. It's currently a quick hack, implemented in the _handlePicking() method of genomeSpy.js
Write documentation (about tooltips in general)

Group samples by threshold(s) on quantitative variables

Use case: The user would like to split the samples into two groups based on a specific threshold and see the exact sizes of the groups.

Currently there's group to quartiles. That's quick and practical, but not specific enough.

It would be nice to support multiple breakpoints. However, it quickly becomes a bit complicated, because it should be possible to define open/closed ranges, etc.

Point marks inside faceted samples are not properly squeezed on Nvidia + Linux

The points should be barely visible in this view:

There's something wrong in setting the uniform variables that the shader uses to scale the points.

Some places where to look

The following sets the y coordinate and height for the sample. It's unlikely be the culprit because the points are centered correctly:

genome-spy/packages/core/src/marks/mark.js

Line 656 in 0f2237a

prepareSampleFacetRendering(options) {

Here the height of the facet is calculated. The coordinates are normalized, [0, 1] spans the whole view. The "transit" stuff is related to the transition animations, which are not enabled (working) at the moment:

genome-spy/packages/core/src/gl/includes/sampleFacet.glsl

Line 94 in 0f2237a

float getSampleFacetHeight(vec2 pos) {

This function calculates a factor for reducing the point size so that it fits the sample nicely:

genome-spy/packages/core/src/gl/point.vertex.glsl

Line 39 in 0f2237a

float getDownscaleFactor(vec2 pos) {

The sampleFacetPadding property specifies how much padding to have between the point mark and the sample (facet) bounds:

genome-spy/packages/core/src/spec/mark.d.ts

Line 306 in 0f2237a

sampleFacetPadding?: number;

Precision problems on M1 Mac GPU

It appears that LUMA_FP64_CODE_ELIMINATION_WORKAROUND is not effective enough in:
https://github.com/genome-spy/genome-spy/blob/master/packages/core/src/gl/includes/fp64-arithmetic.glsl

This affects both Safari and Chrome.

Have to either:

Do something for the workaround
Replace fp64 with something completely diffferent. We don't need full 64 bit precision. It's enough to be able to zoom and translate the genomic axis.

Implement a minimum width/size for strokes/lines in the picking buffer

Hovering rules or strokes that are just 1px wide is currently quite difficult. See: https://genomespy.app/examples/?spec=SV/sv.json

A minimum width of, say, 2px could be enforced in the shaders when rendering to the picking buffer.

Make chromosomes in axes clickable

Motivation: users would like to zoom into chromosomes with (dbl)click. Read more at: #122

Steps:

Refactor picking

The chromosome labels should be pickable. Now the AxisView blocks picking of all its children:

genome-spy/packages/core/src/view/axisView.js

Line 207 in 81074d2

isPickingSupported() {

Make picking configurable. It could be a property of Views, similarly to:

genome-spy/packages/core/src/spec/view.d.ts

Line 106 in 81074d2

visible?: boolean;

It should have two states, which have a bit different meanings depending on the view type:

state	UnitView	Other View
`true` default	Mark instances are pickable	Child views may be pickable
`false`	Mark instances are not pickable	Children are never pickable

Configure picking of Axis components

AxisView should be pickable but all of it's current children should be declared non-pickable.
Add an invisible rectangles behind the chromosome labels and make them pickable.

...

@grsr, I might have some time for this next week, but this is also a "good first issue" 😊

Axis labels for sample groups

Ordinal scale lacks some functionality at the moment. The cleanest solution would be based on sorting by another field: https://vega.github.io/vega-lite/docs/sort.html#sort-by-a-different-field

⇧ ... They should appear here

Compile and publish JSON schema

Allows validation of the spec and autocomplete in an editor

Compile from *.d.ts files. Depends on vega/ts-json-schema-generator#1046
Automatically publish upon release
Use in the Playground app

Feature: constraining scale domains on locus datatype

Hello, I have been playing with a fork of your ideogram track code from here: https://observablehq.com/@tuner/annotation-tracks

My goal is to be able to generate a specification object that constrains the view to a genomic start and end positions, e.g. such as chr1:100000 to chr3:123456, using the chromosomal ordering and sizes provided in an assembly's cytoBand track file, for instance.

The reason for this is that I have a separate Observable notebook with code that I can readily use to lookup intervals, such as those coming from gene or other custom annotation tables. It would be useful to provide an interval to the specification to "zoom in" to a given region of interest.

I have tried a couple things and am unsure if this is an issue, or if I am specifying things incorrectly. Probably the latter.

The starting view for the hg38 assembly looks like this:

The axis is not shown, but the default ideogram renders all chromosomes in natural sort order (chr1, chr2, etc.).

The first attempt was to simply view all of chr1, by applying a filter transform on the chrom field:

  transform: [
    // Remove unlocalized/unplaced scaffolds etc.
    { type: "filter", expr: "!test(/_/, datum.chrom)" },
    { type: "filter", expr: "datum.chrom === 'chr1'" }
  ],

The code for this version is at: https://observablehq.com/d/e085ea04064331ed

While this only renders bands for chromosome 1, the larger view reserves horizontal space for all chromosomes (I removed the axis property to show the chromosome boundaries):

The second attempt was to specify a scale.domain property to clip the loci by the chrom field to chr1, similar to how I would clip a bar chart with a nominal axis in the Vega test editor:

  encoding: {
    x: {
      chrom: "chrom",
      pos: "chromStart",
      type: "locus",
      scale: {
        domain: [{ chrom: "chr1" }]
      }
    },
    ...
  },

The code for this version is at: https://observablehq.com/d/999e38b6bb6d08f8

This did not clip the dataset:

Do I need to constrain the genomic range via values from the linearizeGenomicCoordinate function, or use a different approach? https://github.com/tuner/genome-spy/blob/c126dcdeb45294a1747dcc85d03ec7274f0be68c/docs/grammar/scale.md#locus-scale

Thanks for making this project and for any guidance you can provide.

Implement view config, i.e., themes and more structured defaults

Could be pretty much identical to https://vega.github.io/vega-lite/docs/config.html

Now defaults are scattered all over the code.

Support `datum` with nominal/ordinal data types

This doesn't currently work:

{
  "data": {
    "values": [0.73]
  },
  "mark": "rect",
  "encoding": {
    "x": {"datum": "Hello", "type": "nominal"},
    "y": {"field": "data", "type": "quantitative"}
  }
}

Datums could be passed to shaders as uniforms or compiled-in constants.

Background: Need titles for sample metadata. Each metadata attribute has its own column in a hconcat. Titles could be implemented as axes.

Locus Y-Axis, labels not appearing

Attempting to duplicate the plot on the left with GenomeSpy, the contig labels are not appearing for the Y axis.

Here is the code I am using...

<!DOCTYPE html>
<html>
    <head>
        <title>GenomeSpy Test</title>
        <link 
            rel="stylesheet" 
            type="text/css"
            href="https://unpkg.com/[email protected]/dist/style.css"/>
    </head>
    <body>
        <script 
            type="text/javascript"
            src="https://unpkg.com/[email protected]/dist/index.js">
        </script>


        <div id="genome-spy-example" style="height:400px;width:400px"></div>

    <script>

// Generate a random number between a and b, including both a and b
function generateRandomIntegerInRange(a, b) {
    return Math.floor(Math.random() * (b - a + 1)) + a;
}
let contigs = [
    {name: "1", size: 195471971},
    {name: "2", size: 182113224},
    {name: "3", size: 160039680},
    {name: "4", size: 156508116},
    {name: "5", size: 151834684},
    {name: "6", size: 149736546},
    {name: "7", size: 145441459},
    {name: "8", size: 129401213},
    {name: "9", size: 124595110},
    {name: "10", size: 130694993},
    {name: "11", size: 122082543},
    {name: "12", size: 120129022},
    {name: "13", size: 120421639},
    {name: "14", size: 124902244},
    {name: "15", size: 104043685},
    {name: "16", size: 98207768},
    {name: "17", size: 94987271},
    {name: "18", size: 90702639},
    {name: "19", size: 61431566},
    {name: "X", size: 171031299}
];

let LODS = [];

for (let x = 0; x < 1000; x++) {
    let gene_contig = contigs[generateRandomIntegerInRange(0, 19)];
    let gene_pos = generateRandomIntegerInRange(1, gene_contig['size'] - 1);

    let marker_contig = contigs[generateRandomIntegerInRange(0, 19)];
    let marker_pos = generateRandomIntegerInRange(1, marker_contig['size'] - 1);

    let score = generateRandomIntegerInRange(10, 50);

    LODS.push({
        'genechr': gene_contig['name'],
        'genepos': gene_pos,
        'markerchr': marker_contig['name'],
        'markerpos': marker_pos,
        'lod': score
    });
}

let trackPeakLOD = {
    "name": "lod",
    "layer": [{
            "title": "Xbound",
            "data": {
                "values": contigs
            },
            "mark": {
                "type": "rule",
            },
            "encoding": {
                "x": {
                    "chrom": "name",
                    "pos": "size",
                    "type": "locus",
                },
                "color": {
                    "value": "lightgray"
                }


            },
        },
        {
            "title": "Ybound",
            "data": {
                "values": contigs
            },
            "mark": {
                "type": "rule"
            },
            "encoding": {
                "y": {
                    "chrom": "name",
                    "pos": "size",
                    "type": "locus",
                },
                "color": {
                    "value": "lightgray"
                }


            },
        },
        {
            "data": {
                "values": LODS
            },
            "title": "LODS",
            "mark": {
                "type": "point",
                "geometricZoomBound": 4
            },

            "encoding": {
                "x": {
                    "chrom": "genechr",
                    "pos": "genepos",
                    "type": "locus",
                },
                "y": {
                    "chrom": "markerchr",
                    "pos": "markerpos",
                    "type": "locus",
                },
                "color": {
                    "value": "#7090c0"
                },
                "size": {
                    "value": 30
                },
                "opacity": {
                    "value": 0.5
                },
                "strokeWidth": {
                    "value": 0
                }
            }
        }
    ]
};

const spec = {
    "genome": {
        "contigs": contigs
    },

    "vconcat": [
        trackPeakLOD
        //trackGenes
    ]
};


var genomeSpyObj = genomeSpyEmbed.embed(
    document.querySelector("#genome-spy-example"),
    spec, {
        bare: true
    }
);
        
        
    </script>
  </body>
</html>

examples on https://genomespy.app/ all fail with syntax error

app build throws errors from own directory

I was trying to run the app package from it's own directory and received errors.

However, running npm start from main directory will work. Any idea why the errors:

No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rect.fragment.glsl

Here are the steps to reproduce.

> git clone https://github.com/genome-spy/genome-spy.git
Cloning into 'genome-spy'...
remote: Enumerating objects: 13858, done.
remote: Counting objects: 100% (3278/3278), done.
remote: Compressing objects: 100% (1595/1595), done.
remote: Total 13858 (delta 2499), reused 2364 (delta 1655), pack-reused 10580
Receiving objects: 100% (13858/13858), 5.60 MiB | 12.78 MiB/s, done.
Resolving deltas: 100% (10654/10654), done.

> cd genome-spy/packages/app
> npm i
added 75 packages, and audited 76 packages in 13s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> npm run dev
npm run dev

> @genome-spy/[email protected] dev
> node dev-server.js

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'express'
Require stack:
- /Users/mvincent/work/genome-spy/packages/app/dev-server.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/mvincent/work/genome-spy/packages/app/dev-server.js:6:17)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/mvincent/work/genome-spy/packages/app/dev-server.js'
  ]
}

Install missing dependencies and start server:

> npm i express
> npm i vite
> npm i vite-raw-plugin
> npm i rollup-plugin-minify-html-literals
> npm run dev
npm run dev

> @genome-spy/[email protected] dev
> node dev-server.js

Pre-bundling dependencies:
  vega-util
  vega-loader
  @genome-spy/core/genomeSpy.js
  lit
  @genome-spy/core/genomeSpy
  (...and 51 more)
(this will be run only when your dependencies or config have changed)
✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rect.fragment.glsl

    node_modules/@genome-spy/core/src/marks/rectMark.js:3:28:
      3 │ import FRAGMENT_SHADER from "../gl/rect.fragment.glsl";
        ╵                             ~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rect.vertex.glsl

    node_modules/@genome-spy/core/src/marks/rectMark.js:2:26:
      2 │ import VERTEX_SHADER from "../gl/rect.vertex.glsl";
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.fragment.glsl

    node_modules/@genome-spy/core/src/marks/link.js:3:28:
      3 │ import FRAGMENT_SHADER from "../gl/link.fragment.glsl";
        ╵                             ~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.vertex.glsl

    node_modules/@genome-spy/core/src/marks/link.js:2:26:
      2 │ import VERTEX_SHADER from "../gl/link.vertex.glsl";
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rule.vertex.glsl

    node_modules/@genome-spy/core/src/marks/rule.js:8:26:
      8 │ import VERTEX_SHADER from "../gl/rule.vertex.glsl";
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rule.fragment.glsl

    node_modules/@genome-spy/core/src/marks/rule.js:9:28:
      9 │ import FRAGMENT_SHADER from "../gl/rule.fragment.glsl";
        ╵                             ~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/point.fragment.glsl

    node_modules/@genome-spy/core/src/marks/pointMark.js:6:28:
      6 │ import FRAGMENT_SHADER from "../gl/point.fragment.glsl";
        ╵                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/point.vertex.glsl

    node_modules/@genome-spy/core/src/marks/pointMark.js:5:26:
      5 │ import VERTEX_SHADER from "../gl/point.vertex.glsl";
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/fp64-arithmetic.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:24:17:
      24 │ import FP64 from "../gl/includes/fp64-arithmetic.glsl";
         ╵                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/sampleFacet.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:28:30:
      28 │ import GLSL_SAMPLE_FACET from "../gl/includes/sampleFacet.glsl";
         ╵                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/common.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:25:24:
      25 │ import GLSL_COMMON from "../gl/includes/common.glsl";
         ╵                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/picking.fragment.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:30:34:
      30 │ import GLSL_PICKING_FRAGMENT from "../gl/includes/picking.fragment.glsl";
         ╵                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/scales.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:26:24:
      26 │ import GLSL_SCALES from "../gl/includes/scales.glsl";
         ╵                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/picking.vertex.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:29:32:
      29 │ import GLSL_PICKING_VERTEX from "../gl/includes/picking.vertex.glsl";
         ╵                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/scales_fp64.glsl

    node_modules/@genome-spy/core/src/marks/mark.js:27:29:
      27 │ import GLSL_SCALES_FP64 from "../gl/includes/scales_fp64.glsl";
         ╵                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/text.fragment.glsl

    node_modules/@genome-spy/core/src/marks/text.js:5:28:
      5 │ import FRAGMENT_SHADER from "../gl/text.fragment.glsl";
        ╵                             ~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/text.vertex.glsl

    node_modules/@genome-spy/core/src/marks/text.js:4:26:
      4 │ import VERTEX_SHADER from "../gl/text.vertex.glsl";
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~

/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1605
  let error = new Error(`${text}${summary}`);
              ^

Error: Build failed with 17 errors:
node_modules/@genome-spy/core/src/marks/link.js:2:26: ERROR: No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.vertex.glsl
node_modules/@genome-spy/core/src/marks/link.js:3:28: ERROR: No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.fragment.glsl
node_modules/@genome-spy/core/src/marks/mark.js:24:17: ERROR: No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/fp64-arithmetic.glsl
node_modules/@genome-spy/core/src/marks/mark.js:25:24: ERROR: No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/common.glsl
node_modules/@genome-spy/core/src/marks/mark.js:26:24: ERROR: No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/scales.glsl
...
    at failureErrorWithLog (/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1605:15)
    at /Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1251:28
    at runOnEndCallbacks (/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1036:63)
    at buildResponseToResult (/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1249:7)
    at /Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:1358:14
    at /Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:668:9
    at handleIncomingPacket (/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:765:9)
    at Socket.readFromStdout (/Users/mvincent/work/deleteme/deleteme2/genome-spy/packages/app/node_modules/esbuild/lib/main.js:635:7)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: undefined,
      location: {
        column: 26,
        file: 'node_modules/@genome-spy/core/src/marks/link.js',
        length: 24,
        line: 2,
        lineText: 'import VERTEX_SHADER from "../gl/link.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 28,
        file: 'node_modules/@genome-spy/core/src/marks/link.js',
        length: 26,
        line: 3,
        lineText: 'import FRAGMENT_SHADER from "../gl/link.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/link.fragment.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 17,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 37,
        line: 24,
        lineText: 'import FP64 from "../gl/includes/fp64-arithmetic.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/fp64-arithmetic.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 24,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 28,
        line: 25,
        lineText: 'import GLSL_COMMON from "../gl/includes/common.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/common.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 24,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 28,
        line: 26,
        lineText: 'import GLSL_SCALES from "../gl/includes/scales.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/scales.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 29,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 33,
        line: 27,
        lineText: 'import GLSL_SCALES_FP64 from "../gl/includes/scales_fp64.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/scales_fp64.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 30,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 33,
        line: 28,
        lineText: 'import GLSL_SAMPLE_FACET from "../gl/includes/sampleFacet.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/sampleFacet.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 32,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 36,
        line: 29,
        lineText: 'import GLSL_PICKING_VERTEX from "../gl/includes/picking.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/picking.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 34,
        file: 'node_modules/@genome-spy/core/src/marks/mark.js',
        length: 38,
        line: 30,
        lineText: 'import GLSL_PICKING_FRAGMENT from "../gl/includes/picking.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/includes/picking.fragment.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 26,
        file: 'node_modules/@genome-spy/core/src/marks/pointMark.js',
        length: 25,
        line: 5,
        lineText: 'import VERTEX_SHADER from "../gl/point.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/point.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 28,
        file: 'node_modules/@genome-spy/core/src/marks/pointMark.js',
        length: 27,
        line: 6,
        lineText: 'import FRAGMENT_SHADER from "../gl/point.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/point.fragment.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 26,
        file: 'node_modules/@genome-spy/core/src/marks/rectMark.js',
        length: 24,
        line: 2,
        lineText: 'import VERTEX_SHADER from "../gl/rect.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rect.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 28,
        file: 'node_modules/@genome-spy/core/src/marks/rectMark.js',
        length: 26,
        line: 3,
        lineText: 'import FRAGMENT_SHADER from "../gl/rect.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rect.fragment.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 26,
        file: 'node_modules/@genome-spy/core/src/marks/rule.js',
        length: 24,
        line: 8,
        lineText: 'import VERTEX_SHADER from "../gl/rule.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rule.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 28,
        file: 'node_modules/@genome-spy/core/src/marks/rule.js',
        length: 26,
        line: 9,
        lineText: 'import FRAGMENT_SHADER from "../gl/rule.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/rule.fragment.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 26,
        file: 'node_modules/@genome-spy/core/src/marks/text.js',
        length: 24,
        line: 4,
        lineText: 'import VERTEX_SHADER from "../gl/text.vertex.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/text.vertex.glsl'
    },
    {
      detail: undefined,
      location: {
        column: 28,
        file: 'node_modules/@genome-spy/core/src/marks/text.js',
        length: 26,
        line: 5,
        lineText: 'import FRAGMENT_SHADER from "../gl/text.fragment.glsl";',
        namespace: '',
        suggestion: ''
      },
      notes: [],
      pluginName: '',
      text: 'No loader is configured for ".glsl" files: node_modules/@genome-spy/core/src/gl/text.fragment.glsl'
    }
  ],
  warnings: []
}

Support customizable view backgrounds

... including background color and stroke around the view.

This depends on #35.

Dynamic data loading and domain-specific file formats

Currently all data must be loaded in one go upon startup, which is infeasible for very large datasets.

Briefly:

Support tabix-indexed sparse data (csv)
- Perhaps implement domain-specific formats (bed, gff, vcf, bam)
Support dense data, needs domain-specific formats (bigwig, fasta (indexed))
Some tiling and caching mechanism is needed
- The challenge is to integrate it seamlessly into the dataflow

Some needs for this have arisen in our lab. Not sure about the implementation schedule...

This issue needs to be split into smaller tasks.

Legends

GenomeSpy visualizations can be quite complex, and thus, legend placement is not that straightforward.

One indea for customization is to use a placeholder view for all legends, similar to patchwork's guide_area().

Add "Retain first n categories of xxx" action to nominal attributes

Implement multi touch to enable zoom/pan on mobile devices

This would be nice to have, but has a low priority. Most GenomeSpy's use cases need a mouse (tooltips on hover, context menus, etc.) anyway.

https://developer.mozilla.org/en-US/docs/Web/API/Touch_events/Multi-touch_interaction

Default placeholder spec in Playground is broken

It's a problem with Vite or something...

View titles

View titles can currently be implemented using layers, but that is quite verbose and cumbersome for the user.

The titles could be specified similarly to Vega-Lite, as it allows configuring the placement flexibly: above the view (in case of ordinary plots) or over the views (track-based visualizations).

The layout and rendering could be managed by ContainerViews such as DecoratorView and ConcatView.

Some themeing support should be considered so that titles would be styled consistently on track-based visualizations.

Ruler

A potential user evaluating GenomeSpy asked about an interactive ruler that would be synchronized between views.

Ruler is somewhat related to selections and brushing and linking. It reacts to mouse events, the ruler is shown as a rule that spans a single view or multiple concatenated views, the screen coordinates are mapped to scale domain, etc. However, in brushing & linking, the selection made with the brush controls a filter or scale domain of another view. The ruler, on the other hand, is just synchronized between the views using a value on the domain. The scale domains may or may not be shared between the views. Later on, the rule could, of course, control a filter on another (or the same) view.

Implementation

Basically, the ruler would be another layer with just a single instance of a rule mark. Its data would be updated dynamically when the users moves the mouse cursor over a view. The coordinate of the cursor should first be inverted onto the domain using the scale that is associated with the x/y channel.

GridView is a class that handles the grid/concatenation layout of views. It also takes care of the view background and axes. Eventually, axis grid lines, brushing, etc., will be implemented there, and it would also be the class that handles the ruler. A view hierarchy of just a single view is a special case of grid view. https://github.com/genome-spy/genome-spy/blob/master/packages/core/src/view/gridView.js

When it comes to dynamic data updates, here’s how AxisView updates the ticks dynamically: https://github.com/genome-spy/genome-spy/blob/master/packages/core/src/view/axisView.js#L180

Configuration / grammar

The rulers should somehow be shared between the views. Obviously, when the views have a shared scale, i.e., their zooming is linked, the rulers should be shared as well. However, when the scales are not shared, we should still be able to somehow link them. Possibly some arbitrary indentifier is needed, i.e., those views that have the same identifier have their rulers synchronised. Analogously, such ids could be used for more fine-grained control of domain sharing – now the are constrained by the hierarchical layout of views.

I'm planning to implement selections (brushing & linking) at some point in future, and the grammar will be based on Vega-Lite's design: https://vega.github.io/vega-lite/docs/parameter.html#data-extents

This is how a brush is specified in Vega-Lite:

{
  ...,
  "params": [{
    "name": "brush",
    "select": {"type": "interval", "encodings": ["x"]}
  }]
}

A ruler could be specified similarly ("point" is desinged for discrete values, however):

{
  ...,
  "params": [{
    "name": "ruler",
    "select": {"type": "point", "encodings": ["x"]}
  }]
}

If the ruler should be shown on multiple views with a shared domain (just an idea):

"select": {"type": "point", "encodings": ["x"], "sharedDomain": "repeat" }

Or if the views are concatenated vertically, maybe a single ruler should span all the views:

"select": {"type": "point", "encodings": ["x"], "sharedDomain": "span" }

If the ruler should be shown on multiple views without a shared domain, a ruler with the same name should be specified on each view.

Some things to be decided:

should all children of concat have a single shared ruler that spans all the views or should they be separate
how to configure its style (strokeWidth, colour, dash pattern, etc.)
should it be enabled by default or not.
maybe there should be an easy shortcut to just enable rulers everywhere

Feature Request: Line Charts

A new feature I would like to see implemented is line charts/plots. We currently are using GenomeSpy to display SNP association plots and would like to see it to display QTL mapping and allele effect plots as shown below.

It would also be nice to have hover and click capabilities on the data points.

Respect the order of ordinal metadata attributes when grouping samples

DecoratorView is a hack, replace with an Axis-aware grid view

... that can be specialized into vconcat, hconcat, concat, repeat, and facet (Vega-Lite style faceting, not samples faceting).

Background: Currently, DecoratorView wraps the actual view and handles its axes, view background, and zoom/pan interactions. However, it's difficult to use with faceting and shared axes, etc. Also, in case of facet, state of multiple decorator instances need to be tracked.

The new view could have responsibilities including but not limited to:

#28.
Axes and #58
Annotations such as
- the threshold of score-based semantic zoom
- citations for the shown data
Panning and zooming of scales that are shared between the child views
Brushing
- An example: We have multiple genomic tracks, all having a shared x scale and channel. The brush should span all the stacked tracks

Massive performance degradation on Mac with Safari

Also seems to affect Chrome when the Metal ANGLE backend is enabled.

Not sure if this is a known ANGLE/Metal issue or not. The majority of WebGL apps seem to work fine on Safari. Anyway, GenomeSpy is barely usable on Safari now.

This is quite critical because once Chrome switches to the Metal backend by default, GenomeSpy will be unusable on Chrome as well.

Possible culprits:

Draw calls
Something in shaders
Some buffer stuff?
What else?

Edit: The canvas size seems to have a large effect on the performance, which suggest that maybe there's something in the fragment shaders.

Layout offset problem in SampleView when metadata is hidden

Shown on the left. Doesn't align with axis and cytobands.

Axis grid lines

A proper datasource or transform from tick generation should be created, i.e., extract

genome-spy/packages/core/src/view/axisView.js

Line 151 in 2abd5a2

_updateAxisData() {

into a datasource.

Grid should be handled in https://github.com/genome-spy/genome-spy/blob/master/packages/core/src/view/gridView.js ... just like axis ticks and labels.

It would be nice to be able to parametrize other thing too. Examples: visibility of certain things, titles, styles, field mappings, whatever.

So, instead of:

{
  "import": { "url": "https://genomespy.app/tracks/cosmic/census_hg38.json"  }
}

... it could be like:

{
  "import": {
    "url": "https://genomespy.app/tracks/cosmic/census.json",
    "params": {
      "genome": "hg38"
    }
  }
}

How to to handle the parametrization in the imported spec need to be figured out.

Templating

One promising option is to use json-e templating. There are some challenges, though:

Templating stuff is incompatible with the schema.
Unique view names should be handled somehow. It could happen through parameters but I'm not sure if it's a good idea...
Defining data fields may be cumbersome if the template is very complex. Have to do some prototyping.

Params

Just use params (#205) and use ExprRefs everywhere.

Use cases

Use the same template for multiple data files to add multiple tracks. Here, the URLs of the data sources could accept parameters in a similar way that Vega does: https://vega.github.io/vega/docs/data/#dynamic-data-loading