Comments (6)
There is batchMapResults
. What function are you missing?
from batchtools.
Can you write down an example here, how you would do this, pls.
from batchtools.
(Yes, it's not that I cannot, I just want both of us to see wether that gets complicated and lengthy)
from batchtools.
To directly answer: I am missing something similar as this
https://github.com/tudo-r/BatchExperiments/blob/master/R/reduceResultsExperimentsParallel.R
We can talk about different ways to do this, but the motivation for that function is exactly the same as for my issue here.
from batchtools.
Reducing 35k jobs takes around 40 minutes, thats pretty long...
from batchtools.
Some comments for registries with many jobs:
- The file system or network device is usually the bottle neck. Parallelism helps, but don't expect miracles from something like
reduceResultsParallel
, this does not scale well. - It is usually faster to move the file directory to a local ssd first (i.e., do a sequential read on the slow file system), then reduce the results
- A way to store the results of a complete chunk in a single file would be best. But this requires quite some work.
from batchtools.
Related Issues (20)
- Error in readLog() on Slurm cluster HOT 12
- scoping errors depending on objects in global environment
- Job requeue on Slurm - 'no such file or directory' & workaround HOT 1
- ability to suppress warning on failure to source `source` when calling makeRegistry
- All jobs expired with SGE cluster, job script permission denied HOT 2
- makeClusterFunctions store.job.collection flag documentation
- Yet another `BatchtoolsExpiration: Future ('<none>') expired` error
- `$conditions` blows up rds `results/*.rds` files HOT 2
- Publish 0.9.16 on CRAN HOT 3
- Edit the job template without clearing the entire registry
- Make 'options(error = function(e) traceback(2))` optional in testJob
- Error in (function () : unused argument (x = 3) HOT 3
- Question: How are template files interpretted? HOT 1
- Effectively use batchtools for Targets-based workflow in SLURM HOT 4
- squeue cmd not found when running R targets on singularity docker image HOT 1
- calling `batchMap` + `submitJob` in a loop triggers "Registry must be empty" error HOT 1
- submitJob - can't find qsub function HOT 1
- Slurm resource named `clusters` in docs, and `cluster` in `getClusters`
- Creating jobs without package dependencies installed on head node
- Old version of Slurm, "invalid job state specified: SI" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from batchtools.