Comments (3)
Reservation setup as follows:
registerDoParallel( cores=num_cores )
Additional options to multicore setup as:
mcoptions=list( silent=TRUE )
Entering the loop as follows:
## Ugh! Sometimes with preschedule=F. %dopar% will finish but only process the first job.
## Until that gets straightened out, change to %dopar% without preschedule=F,
## but without it %dopar% sometimes has a job die, in which case settle for %do%. :(
## Note to self: Might need to specify a particular number of cores based on the number of files?
#foreach ( seurat_filepath=iter( seurat_files ), options.multicore=mcoptions, preschedule=F ) %dopar% {
foreach ( seurat_filepath=iter( seurat_files ) ) %dopar% {
#foreach ( seurat_filepath=iter( seurat_files ) ) %do% {
from cellranger-snakemake.
This appears to have resolved as quickly as it appeared. Will close, but might have to re-open in the future.
from cellranger-snakemake.
I've somewhat serendipitously discovered that the original issue of %dopar%
failing on a single job is very likely due to insufficient memory on the node the R script is running. When running under %do%
, the script consistently failed on the same step every time (FindAllMarkers for a particular group). I ran it a couple of times from the sinteractive node that didn't request enough memory, and it failed the same way each time. I then started a new sinteractive session with more memory allocated (64GB vs 48GB) and the entire job was able to complete (including a rerun where I switched back to %dopar%
). (This particular process used a peak of just over 58 GB). I'm going to bump up the default requested memory for the R script step in the pipeline, which should reduce the number of times this is seen. It may or may not be worth investigating how to predict the amount of memory needed for this step and request as appropriate.
from cellranger-snakemake.
Related Issues (8)
- Default to using the biomart file found in ./bin, recreate in workdir if asked. HOT 1
- Create SOP describing setup, pipeline execution, etc. HOT 1
- Create tSNE, UMAP plots as pdfs HOT 4
- Add FindAllMarkers and create various heatmaps HOT 1
- Add EnrichR after FindAllMarkers HOT 1
- Add filtration of various 'noisy' genes HOT 2
- Cleanup intermediate files (could be done using native snakemake nomenclature)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cellranger-snakemake.