Comments (7)
Hi Dien,
Thanks for your comment. When we developed scmap we did not have access to such large datasets, so it was never tested in such conditions. Hence, I have not come across this issue before. It seems as if you are using scmapCell - do you have the same problem with scmapCluster? Also, it is not clear from your message at what step the error occurs, so could you provide some more context?
Best regards,
Martin
from scmap.
Hi,
Same error here with a dataset of 120,000 cells and when calling selectFeatures
.
> sce.SS
class: SingleCellExperiment
dim: 21087 120583
metadata(0):
assays(2): counts logcounts
rownames(21087): Sox17 Mrpl15 ... 1700109K24Rik Gm10556
rowData names(1): feature_symbol
colnames(120583): AAACCTGAGGCAATTA-1_1_1 AAACCTGGTAGATTAG-1_1_1 ... TTTGTCATCTTGTACT.4_32
TTTGTCATCTTTACGT.4_32
colData names(6): nCount_RNA nFeature_RNA ... Tissue ident
reducedDimNames(3): PCA HARMONY UMAP
mainExpName: RNA
altExpNames(0):
> rowData(sce.SS)$feature_symbol <- rownames(sce.SS)
> sce <- selectFeatures(sce.SS, suppress_plot = T, n_features = 500)
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
from scmap.
Hi,
I came up with the same question when calling selectFeatures
with a dataset of about 150k cells.
It seems like the function tried to convert the sparse matrix into a dense object so that it became too large to handle with.
the error message is as follow:
Error in asMethod(object): Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Traceback:
1. selectFeatures(trainset.sce, suppress_plot = TRUE)
2. selectFeatures(trainset.sce, suppress_plot = TRUE)
3. linearModel(object, n_features)
4. as.matrix(logcounts(object))
5. as.matrix.Matrix(logcounts(object))
6. as(x, "matrix")
7. asMethod(object)
from scmap.
Is there a solution?
from scmap.
A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.
Best,
GW
from scmap.
A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.
Best, GW
Glad you found it works for your case. Actually, I worked on HPC with a memory of 200G, so this should not be a problem to my case. I eventually ended up using Scanpy to move on but still following this case.
from scmap.
A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.
Best, GWGlad you found it works for your case. Actually, I worked on HPC with a memory of 200G, so this should not be a problem to my case. I eventually ended up using Scanpy to move on but still following this case.
I think the cholmod error can result from your matrix exceeding the maximum allowed data dimensions by the conversion tool used by Matrix package, which would be independent of the amount of RAM you have. The possible solution that I linked provides an alternative way to convert a matrix from sparse to dense that bypasses this problem. If you re-tried with this way, you probably wouldn't get the cholmod error again with a 200G HPC.
Best,
GW
from scmap.
Related Issues (20)
- Outdated References HOT 1
- scmapCell error HOT 1
- problem with scmapCluster HOT 3
- broken URL in README HOT 4
- Different number of elements passed to dot() function by scmapCell HOT 1
- Cancer tumor reference dataset HOT 2
- Issue with scmap-cluster function output/ Variable inconsistent assignment HOT 2
- Similarity values are not between 0 and 1 HOT 4
- error in selectFeatures HOT 1
- Encountering problems with selectFeatures HOT 1
- error on selectFeatures HOT 5
- How should I import data from seurat to scmap?
- Heat map missing data HOT 2
- Error using scmapCluster HOT 2
- What kind of reference dataset can be use? HOT 2
- Error on scmapCell HOT 1
- Quite low accuracy using index cluster HOT 2
- Questions about confidence scores
- Add 'codetools' to Suggests:
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scmap.