Comments (4)
This will depend on machine, file size, and algorithm as @sindresorhus has suggested. However, you're unlikely to see significant benefits beyond require('os').cpus().length
since these algorithms are CPU-intensive. Note that my test machine has 6 physical cores and that the os
module returns a logical processor count instead (hyperthreading). Below you will find a benchmark implementation and results in support of require('os').cpus().length
:
// node_modules zipped
BENCHMARK: 50 FILES @ 10 ITERATIONS FILE SIZE: 27877036
Concurrency: 1 Total: 56791 ms Average: 5679.1 ms Cores: 12
Concurrency: 6 Total: 24740 ms Average: 2474 ms Cores: 12
Concurrency: 12 Total: 24488 ms Average: 2448.8 ms Cores: 12
Concurrency: 24 Total: 21875 ms Average: 2187.5 ms Cores: 12
Concurrency: 36 Total: 21975 ms Average: 2197.5 ms Cores: 12
Concurrency: 48 Total: 21945 ms Average: 2194.5 ms Cores: 12
// random jpg
BENCHMARK: 50 FILES @ 10 ITERATIONS FILE SIZE: 549590
Concurrency: 1 Total: 1292 ms Average: 129.1 ms Cores: 12
Concurrency: 6 Total: 506 ms Average: 50.6 ms Cores: 12
Concurrency: 12 Total: 490 ms Average: 49 ms Cores: 12
Concurrency: 24 Total: 482 ms Average: 48.2 ms Cores: 12
Concurrency: 36 Total: 471 ms Average: 47.1 ms Cores: 12
Concurrency: 48 Total: 473 ms Average: 47.3 ms Cores: 12
const logicalProcessors = require('os').cpus().length;
const fs = require('fs');
const {default: PQueue} = require('p-queue');
const hasha = require('.');
// Update w/ your files
const files = [];
for (let i = 1; i <= 50; i++) {
files.push(`./fixtures/${i}.zip`);
}
const stats = fs.statSync(files[0]);
const count = 10;
const benchmark = async concurrency => {
const queue = new PQueue({concurrency});
let total = 0;
const benchmarkStart = Date.now();
for (let i = 0; i < count; i++) {
const iterationStart = Date.now();
await queue.addAll(files.map(f => () => hasha.fromFile(f)));
const iterationEnd = Date.now();
total += (iterationEnd - iterationStart);
}
const benchmarkEnd = Date.now();
console.log(`Concurrency: ${queue.concurrency}\t\tTotal: ${benchmarkEnd - benchmarkStart} ms\tAverage: ${total / count} ms\tCores: ${logicalProcessors}`);
};
const run = async () => {
console.log(`BENCHMARK: ${files.length} FILES @ ${count} ITERATIONS\tFILE SIZE: ${stats.size}`);
await benchmark(1);
await benchmark(logicalProcessors / 2);
await benchmark(logicalProcessors);
await benchmark(logicalProcessors * 2);
await benchmark(logicalProcessors * 3);
await benchmark(logicalProcessors * 4);
};
run();
from hasha.
That really depends on the machine (disk type/speed and number of cores), Node.js version, and file size though. But maybe there's like a safe number. Something that always saturates all CPU cores, but doesn't overload the system. This will need some manual experimentation. I don't have an answer for this. I would guess something like the number of CPU cores times 2-4.
from hasha.
I would guess something like the number of CPU cores times 2-4.
const concurrency = require("os").cpus().length * 3
from hasha.
@brandon93s Cool, thank you!
from hasha.
Related Issues (20)
- Any chance for unhasha? HOT 3
- Hash multiple files? HOT 5
- Add support for doing the hashing in a worker thread HOT 13
- Typescript import broken for `fromFile`? HOT 3
- Add Salt? HOT 2
- New version (5.1.0) hasha.fromFile makes Electron application crash (unallowed memory allocation for worker threads) HOT 1
- New version (5.1.0) throws webpack errors HOT 2
- [warning] Module not found: Error: Can't resolve 'worker_threads' HOT 1
- Image hash length HOT 2
- `fromFile` broken on Node 14 HOT 8
- hasha.fromFile throws error with message "The V8 platform used by this instance of Node does not support creating Workers" when updated to version 5.2.0 HOT 2
- Browser support HOT 2
- Why does `hasha.fromFile` returns `Promise<string | null>` instead of `Promise<string>`?
- Fatal error with Promise.all() HOT 3
- Support for AbortSignal
- convert to esm HOT 1
- It takes a long time when hasha from a large file, provider a bloom filter to speed up the hasha? HOT 6
- add support to calculate hash of first or last nkb of the file HOT 3
- create-hash instead crypto.createHash HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hasha.