- Zero-shot learning refers to the ability of large pre-trained models to perform tasks they were not explicitly trained on
- For example, Bert/GPT-based models were trained on large corpora of text with language modeling objectives, but they can generalize to new, unseen tasks well with either minimal training (few-shot) or no training (zero-shot)
- In the context of our goals, we can make use of large DNA foundation models pre-trained on language modeling objectives
- To create an embedding for a region set we can create a set of vectors from a bed file and a reference genome
- Then we can do mean aggregation to create a single region set vector. ![[zero shot .png]]
- There are various large models out there that we could test to gauge the efficacy of a zero-shot approach
- **Enformer (Oct 2021)
- Conv input stage followed by transformer (used to reduce sequence length before transormer to maximize context length)
- Prediction of various values of genomic assays like ChIP-seq, etc.
- Deep RNA (Sept 2023) (they say they'll release the model upon peer review?)
- Conv input stage -> transformer
- Comes from industry so maybe they wont release it.
- Geneformer (May 2023)
- pre-trained on a large-scale corpus of about 30โ million single-cell transcriptomes
- DNABERT2 (June 2023)
- Transformer trained on human reference genome with language modeling objective.
- Uses flash attention and tokenizer tricks to enable faster inference on longer sequences
- Hyena DNA (June 2023)
- Long-range convolution-based model capable of handling very long sequence lengths
- Can handle even up to 1Mbp (with batch size 1 on 80GB A100)
- Mamba (Dec 2023)
- State space sequence model also able to handle very long sequences
Taken from [1] Zheng, G., Rymuza, J., Gharavi, E., LeRoy, N. J., Zhang, A., & Sheffield, N. C. (2023). Methods for evaluating unsupervised vector representations of genomic regions. bioRxiv, 2023-08.
- How well a set of embeddings can form clusters
- Useful for both individual regions and region sets
- Evaluates information preserved in an embedding by testing whether an embedding can reconstruct the training data.
- Probably easy for individual regions and harder for region sets.
- Calculates how the embedding distance between two regions scales with genome distance.
- Unlikely to use this for region sets -- just single regions.
- Use monte carlo simulation ...
- Calculates how much the neighborhood of a region in the genome space overlaps with the neighborhood of the same region in the embedding space.
- Again, this is mostly useful for single regions.
From [1],
We collected 690 BED files from the ENCODE Uniform TFBS composite track1 as a region set collection to generate and evaluate region embeddings.
We can expand on this later to something larger scale.
- check the giggle paper
- Comparison with giggle index searches
- Retrieval augmented machine learning models for other types of tasks.
- Eg. a sequence model that has a KNN data structure that retrieves nearest neighbors and uses them as input features.
![[feedforward aggregation 1.png]]![[transformer agg 1.png]]