Code Monkey home page Code Monkey logo

Comments (4)

brandon93s avatar brandon93s commented on July 21, 2024 2

This will depend on machine, file size, and algorithm as @sindresorhus has suggested. However, you're unlikely to see significant benefits beyond require('os').cpus().length since these algorithms are CPU-intensive. Note that my test machine has 6 physical cores and that the os module returns a logical processor count instead (hyperthreading). Below you will find a benchmark implementation and results in support of require('os').cpus().length:

// node_modules zipped
BENCHMARK:  50 FILES @ 10 ITERATIONS    FILE SIZE: 27877036
Concurrency: 1          Total: 56791 ms     Average: 5679.1 ms      Cores: 12
Concurrency: 6          Total: 24740 ms     Average: 2474 ms        Cores: 12
Concurrency: 12         Total: 24488 ms     Average: 2448.8 ms      Cores: 12
Concurrency: 24         Total: 21875 ms     Average: 2187.5 ms      Cores: 12
Concurrency: 36         Total: 21975 ms     Average: 2197.5 ms      Cores: 12
Concurrency: 48         Total: 21945 ms     Average: 2194.5 ms      Cores: 12
// random jpg
BENCHMARK:  50 FILES @ 10 ITERATIONS    FILE SIZE: 549590
Concurrency: 1          Total: 1292 ms      Average: 129.1 ms       Cores: 12
Concurrency: 6          Total: 506 ms       Average: 50.6 ms        Cores: 12
Concurrency: 12         Total: 490 ms       Average: 49 ms  Cores: 12
Concurrency: 24         Total: 482 ms       Average: 48.2 ms        Cores: 12
Concurrency: 36         Total: 471 ms       Average: 47.1 ms        Cores: 12
Concurrency: 48         Total: 473 ms       Average: 47.3 ms        Cores: 12
const logicalProcessors = require('os').cpus().length;
const fs = require('fs');
const {default: PQueue} = require('p-queue');
const hasha = require('.');

// Update w/ your files
const files = [];
for (let i = 1; i <= 50; i++) {
	files.push(`./fixtures/${i}.zip`);
}

const stats = fs.statSync(files[0]);

const count = 10;

const benchmark = async concurrency => {
	const queue = new PQueue({concurrency});
	let total = 0;
	const benchmarkStart = Date.now();
	for (let i = 0; i < count; i++) {
		const iterationStart = Date.now();
		await queue.addAll(files.map(f => () => hasha.fromFile(f)));
		const iterationEnd = Date.now();
		total += (iterationEnd - iterationStart);
	}

	const benchmarkEnd = Date.now();
	console.log(`Concurrency: ${queue.concurrency}\t\tTotal: ${benchmarkEnd - benchmarkStart} ms\tAverage: ${total / count} ms\tCores: ${logicalProcessors}`);
};

const run = async () => {
	console.log(`BENCHMARK:  ${files.length} FILES @ ${count} ITERATIONS\tFILE SIZE: ${stats.size}`);
	await benchmark(1);
	await benchmark(logicalProcessors / 2);
	await benchmark(logicalProcessors);
	await benchmark(logicalProcessors * 2);
	await benchmark(logicalProcessors * 3);
	await benchmark(logicalProcessors * 4);
};

run();

from hasha.

sindresorhus avatar sindresorhus commented on July 21, 2024 1

That really depends on the machine (disk type/speed and number of cores), Node.js version, and file size though. But maybe there's like a safe number. Something that always saturates all CPU cores, but doesn't overload the system. This will need some manual experimentation. I don't have an answer for this. I would guess something like the number of CPU cores times 2-4.

from hasha.

Richienb avatar Richienb commented on July 21, 2024 1

I would guess something like the number of CPU cores times 2-4.

const concurrency = require("os").cpus().length * 3

from hasha.

papb avatar papb commented on July 21, 2024

@brandon93s Cool, thank you!

from hasha.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.