sindresorhus / hasha Goto Github PK
View Code? Open in Web Editor NEWHashing made simple. Get the hash of a buffer/string/stream/file.
License: MIT License
Hashing made simple. Get the hash of a buffer/string/stream/file.
License: MIT License
when I hasha a 400mb file, it takes 900ms, if I hasha a mount of 1Gb files,maybe it will take me several seconds. so I have to speed up. And bloom filter resolves it.
Is there any good ideas to hasha a mount of big files?
It would be nice to have in readme a suggestion of what concurrency to use if I want to hash multiple files concurrently. I don't have knowledge on this to even guess.
Thank you!
how about adding support to calculate hash of specific part of the file?
Something like hash of first or last 64 KB of a file, can be useful when using API like SubDB where they require hash of first and last 64 KB of a file.
Could maybe be useful for hashing large files, as the crypto Node.js module is synchronous, so it could block the main thread for some time.
https://nodejs.org/api/worker_threads.html
Thoughts?
IssueHunt has been backed by the following sponsors. Become a sponsor
So i got this piece of code:
console.log('RECEIVED ALL HASHES');
const promises = [];
data.paths.forEach(path => {
console.log('RECEIVED PATH:', path)
promises.push(hasha.fromFile(path, {
algorithm: 'md5'
}));
});
console.log('FINISHED LOOP');
const results = [];
for (let i = 0; i<promises.length; ++i) {
const r = await promises[i];
console.log(data.paths[i], ' got HASH: ', r);
results.push(r);
}
console.log('FINISHED PROMISES');
return results.map((r, i) => ({path: data.paths[i], hash: r}))
Running it like this it produces the right output as expected. It takes a while but creates all the hashes
But when using Promise.all like this:
console.log('RECEIVED ALL HASHES');
const promises = [];
data.paths.forEach(path => {
console.log('RECEIVED PATH:', path)
promises.push(hasha.fromFile(path, {
algorithm: 'md5'
}));
});
console.log('FINISHED LOOP');
const results = await Promise.all(promises);
console.log('FINISHED PROMISES');
return results.map((r, i) => ({path: data.paths[i], hash: r}))
I receive this fatal error:
#
# Fatal error in , line 0
# ignored
#
#
#
#FailureMessage Object: 0000005D5B5FC610
1: 00007FF6D6B7B43F node::Buffer::New+127615
2: 00007FF6D68C544A IsSandboxedProcess+653770
3: 00007FF6D4D2F05B v8::Platform::SystemClockTimeMillis+413467
4: 00007FF6D4A75BE3 v8::ArrayBuffer::GetContents+291
5: 00007FF6D4A75AB0 v8::ArrayBuffer::Externalize+16
6: 00007FF6D798AB74 uv_dlerror+338660
7: 00007FF6D798CE41 uv_dlerror+347569
8: 00007FF6D798D341 uv_dlerror+348849
9: 00007FF6D4ABD62E v8::CFunction::CFunction+233422
10: 00007FF6D4ABCA8B v8::CFunction::CFunction+230443
11: 00007FF6D4ABC130 v8::CFunction::CFunction+228048
12: 00007FF6D4ABBDB7 v8::CFunction::CFunction+227159
13: 00007FF6D54D957C v8_inspector::V8StackTraceId::ToString+3453724
14: 00007FF6D546DE75 v8_inspector::V8StackTraceId::ToString+3013653
15: 00007FF6D549BC51 v8_inspector::V8StackTraceId::ToString+3201521
16: 00007FF6D55234D8 v8_inspector::V8StackTraceId::ToString+3756664
17: 00007FF6D548E4FA v8_inspector::V8StackTraceId::ToString+3146394
18: 00007FF6D546B82C v8_inspector::V8StackTraceId::ToString+3003852
19: 00007FF6D4B7D4D1 v8::Unwinder::PCIsInV8+2705
20: 00007FF6D4B7DF6F v8::Unwinder::PCIsInV8+5423
21: 00007FF6D4B7E076 v8::Unwinder::PCIsInV8+5686
22: 00007FF6D4BA2103 v8::Unwinder::PCIsInV8+153283
23: 00007FF6D4BA1EDF v8::Unwinder::PCIsInV8+152735
24: 00007FF6D546ED2E v8_inspector::V8StackTraceId::ToString+3017422
25: 00000067000D38F2
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! [email protected] electron:serve: `wait-on tcp:4200 && npm run electron:serve-tsc && npx electron . --serve`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the [email protected] electron:serve script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\danie\AppData\Roaming\npm-cache\_logs\2020-12-26T20_09_51_843Z-debug.log
ERROR: "electron:serve" exited with 3.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `npm-run-all -p electron:serve ng:serve`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\danie\AppData\Roaming\npm-cache\_logs\2020-12-26T20_09_51_947Z-debug.log
I have no idea what it's trying to tell me...
I have a 8 core 3700x and 32 GB of ram if this is any help.
I would like to run it in parallel since there are alot of files that get hashed somewhat often.
Because of this bug, hasha
doesn't work with Node 14 right now: nodejs/node#33263
vscode
$ node -v
v10.16.3
$ npm -v
6.9.0
$ vue -V
@vue/cli 4.2.2
package.json
"dependencies": {
"hasha": "^5.2.0",
"vue": "^2.6.11",
"vue-i18n": "^8.15.3",
"vue-router": "^3.1.5",
"vuex": "^3.1.2",
}
Hey, Can I make a PR to hash multiple files? I could use combined-stream
and reuse hasha.fromStream
to add this functionality.
To improve the CLI, it would be nice if the output had a newline at the end of it. That way, the prompt will start on a new line. Instead of behind the hash.
Currently:
PROMPT > hasha foobar
0a50261eb...2425PROMPT>
Improved:
PROMPT > hasha foobar
0a50261eb...2425
PROMPT>
I just tried to use the typescript definitions and they're not working for me:
import hasha from "hasha"
-> the import works, but hasha
is undefined
import { hasha } from "hasha"
-> doesn't compile (as expected, hasha is the default export, not one of many)
import * as hasha from "hasha"
-> works, but can't call fromFile
(hasha
is the hashing function (so input, options = {}) => ...
), but fromFile
isn't a property of it for some reason)
You can reproduce this with a new project and the following commands:
$ yarn init
$ yarn add hasha
$ yarn add @types/hasha
$ yarn add ts-node
$ ./node_modules/.bin/ts-node -e 'import hasha from "hasha"; console.log("hasha?", hasha);' # first case (undefined)
$ ./node_modules/.bin/ts-node -e 'import {hasha} from "hasha"; console.log("hasha?", hasha);' # second case (type error)
$ ./node_modules/.bin/ts-node -e 'import * as hasha from "hasha"; console.log("hasha?", hasha.fromFile);' # third case (`hasha` itself is defined but `hasha.fromFile` is not)
Am I forgetting something fundamental about imports or is this a real bug? The first case import hasha from "hasha"
should have been all I needed.
I am using hasha.Fromfile method for creating and comparing hash of two exe files. Recently I updated hasha to v 5.2.0 and I started getting this error. When I revert back to v 5.0.0 then this error is not there.
Some more details on machine environment:
Node version: 12.14.0
npm version: 6.13.0
OS: windows
However, my node and npm version is been same for quiet long. Is there any change in v 5.2.0 compared to v 5.0.0 which can cause this issue. I could not find any changelog document to verify the same.
Can you add salt on your hash so it's tasty?
Using the Web Crypto API, we can generate sha1, sha256, sha384 and sha512 hashes with little code. For example:
const encoder = new TextEncoder("utf-8")
const algorithmMap = new Map([
['sha1', 'SHA-1'],
['sha256', 'SHA-256'],
['sha384', 'SHA-384'],
['sha512', 'SHA-512']
])
const arrayBufferToHex = (arrayBuffer, base) => [...new Uint8Array(arrayBuffer)].map(value => value.toString("hex").padStart(2, "0"))
module.exports = () => {
throw new Error('Synchronous hashing is not supported in browsers!')
}
exports.async = (input, {algorithm = 'sha512', encoding = 'hex'} = {}) => {
if (typeof input === "string") {
input = encoder.encode(input)
}
// TODO: Support other encodings
return arrayBufferToHex(await crypto.subtle.digest(algorithmMap.get(algorithm), input))
}
if somebody will use this package with browserify bundle size will be awful
you can use createHash (part of crypto-browserify) for decrease bundle size without speed loss
Especially fromFile
would benefit from abort signal.
WARNING Compiled with 1 warnings
warning in ./node_modules/hasha/index.js
Critical dependency: the request of a dependency is an expression
I'm using hasha in an Electron app that doesn't support Node 12 yet (v10.15.3).
Using hasha 5.0.0 there were no problems but since 5.1.0 my application crashes following an error that doesn't get caught in the fromFile promise.
The problem (I think) is the use of worker_threads in an environment below Node 12 resulting in an unallowed memory allocation windows error when built.
Locking my hasha version resolved the issue for me but I'm posting this as I don't know this is expected behaviour.
Just wondered if there would be any possibility of going the opposite direction as well? I love the simplicity of the hasha api and hate requiring yet another module with an inferior api just to get back to a string.
Thanks @sindresorhus! Just for the record, I actually bookmarked https://github.com/sindresorhus?tab=repositories, so whenever I'm looking for something, I can see if you've tackled it yet! 👍
True, CRC is less unique than MD5 - but it has its uses. If you're interested, I created the following Crypto-module-Hash-class-friendly wrapper a while back and could send a PR to add it: https://github.com/DavidAnson/crc-hash
In what situation could it return null?
I know you are converting everything to ESM in your one piece and you have a plan laid out. But i just wish to be pinged when it's done.
Thanks in advance ❤️
So your package is the fastest yet. other image hashing takes up to 500ms while yours is 5ms. Though there is something i can't figure out. i need to have my hashes around 64 characters and your hashing doesn't seem to have variable hashing length. is there a way to do this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.