openhft / zero-allocation-hashing Goto Github PK
View Code? Open in Web Editor NEWZero-allocation hashing for Java
License: Apache License 2.0
Zero-allocation hashing for Java
License: Apache License 2.0
Using the library on some Android versions results in following Error:
java.lang.NoSuchMethodError: No virtual method getByte(Ljava/lang/Object;J)B in class Lsun/misc/Unsafe; or its super classes (declaration of 'sun.misc.Unsafe' appears in /system/framework/core-libart.jar)
at net.openhft.hashing.LongHashFunction.<clinit>(LongHashFunction.java:73)
It does work on Android 7. It seems, older Android versions do not implement full functionality of Unsafe.
Considering that Unsafe is likely to be replaced/killed off in future versions of mainline Java, you should probably move all related static constants into the UnsafeAccess and add some handling for situations when it is not available.
See:
Also, a changelog would be welcome since https://github.com/OpenHFT/Zero-Allocation-Hashing/releases doesn't include the actual releases and the readme contains old refrences to versions:
Hi, there's a branch of this with 128 bit xxhashing in it that I need, and I'm trying to build it in intellij to a jar but I can't find the main class? How should I go about producing a jar that other projects can reference?
Thank you very much.
Now a commit triggers travis testings for eight JDKs, 7-14, and each two endian. I think we don't need to tests for a JDK after half year of EOL, such as JDK 9, 10, 12. That is,
Hey,
I'm trying to avoid using off-heap Map/Set implementations for performance reasons (serialization/deserialization) and a good way of doing that is using Long
instead of String for UUID like keys and using some Map implementation that doesn't autobox primitives...
I'm just wondering what is the most suitable 64bit hash function? I'm mostly going for the lowest collision rate rather than for performance. I'd just like to have a HashMap/HashSet with up to 30-40M records with the least memory footprint possible without GC overhead and test it for presence of 400M distinct UUIDs. I have no idea what collision rate could be there if I use say murmur3 for 64bit hashing of those UUIDs.
I'm very grateful for your library, it is the only source of 64bit hash functions. I'm kind of terrified of using Guava's 128bit hash functions and then calling :
http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/hash/HashCode.html#padToLong--
In standard hash functions in Java we can do something like:
Digest digest = new Blake2bDigest(64);
for (int i = 0; i < bytes[0].length; i++) {
digest.update(bytes[0][i], 0, bytes[0][i].length));
}
byte[] hash = new byte[digest.getDigestSize()];
digest.doFinal(hashResult, 0);
return hash;
I understand that it's not possible to have similar API here since the main goal of this project is to have zero allocations. But what will be the correct way to calculate result hash for an array of arrays? Is there any solution except creating one big byte array and put everything into it?
Thanks.
Hi, is it possible to publish a new release that includes metro hash? Thanks.
Are there any objections to upgrade the project to Java 8? Currently the compiler is set to Java 6.
0.11
doesn't seem to contain murmur 128 yet - any plans to release it?
Currently it falls back to something CharSequence
-generic, but should be optimized.
Add JMH benchmarks as a subproject in this repository. Evaluate MetroHash performance and add the results to the table in README: https://github.com/OpenHFT/Zero-Allocation-Hashing#performance
Is the 128-bit version on the road map maybe ??
Hello! I'd like to be able to hash parts of a long string without having to copy them into a separate StringBuilder / String etc.. For this I'd like to be able to implement CharSequence in order to present the chars I would like to hash.
The public methods for LongHashFunction only accept String and StringBuilder which are final. However there is the package private method: long hashNativeChars(CharSequence input) in LongHashFunction.java which would provide that. Is there a reason why it cannot be made public to allow access from other packages to it? (without ofc resorting to accessing it through the private-package scope). That would be really great. Thanks!
Hi,
I have it only on android device: Samsung Galaxy J5 (SM-J530F), but possibly there are other devices affected.
We have fatal crash that always happens when trying to hash byte array (contents does not seem to matter) using xxHash:
LongHashFunction
.xx().hashBytes(value)
Taken from LogCat:
--------- beginning of crash
2020-03-12 12:07:08.966 16010-16280/? A/libc: Fatal signal 7 (SIGBUS), code 1, fault addr 0x1338520c in tid 16280 (.pl/...), pid 16010 ()
2020-03-12 12:07:09.051 16283-16283/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2020-03-12 12:07:09.051 16283-16283/? A/DEBUG: Build fingerprint: 'samsung/j5y17ltexx/j5y17lte:8.1.0/M1AJQ/J530FXXU3BRJ2:user/release-keys'
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: Revision: '7'
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: ABI: 'arm'
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: pid: 16010, tid: 16280, name: .pl/... >>> com.erfg.music <<<
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: signal 7 (SIGBUS), code 1 (BUS_ADRALN), fault addr 0x1338520c
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: r0 1338520c r1 0000000c r2 ca9e95cc r3 0000000c
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: r4 6f31be58 r5 00000004 r6 00000000 r7 ca9e98c8
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: r8 00000000 r9 cb5f2c00 sl ca9e96c8 fp ca9e9654
2020-03-12 12:07:09.052 16283-16283/? A/DEBUG: ip eae9fced sp ca9e95a8 lr eae9fcf7 pc eae9fcfa cpsr 600d0030
2020-03-12 12:07:09.185 16283-16283/? A/DEBUG: backtrace:
2020-03-12 12:07:09.185 16283-16283/? A/DEBUG: #00 pc 00310cfa /system/lib/libart.so (art::Unsafe_getLong(_JNIEnv*, _jobject*, _jobject*, long long)+13)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #01 pc 005db08f /system/framework/arm/boot.oat (offset 0x1cb000) (sun.misc.Unsafe.getLong [DEDUPED]+110)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #02 pc 0040c575 /system/lib/libart.so (art_quick_invoke_stub_internal+68)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #03 pc 004116e5 /system/lib/libart.so (art_quick_invoke_stub+228)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #04 pc 000b0227 /system/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+138)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #05 pc 00204005 /system/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+224)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #06 pc 001ff54d /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+588)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #07 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #08 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #09 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #10 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #11 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #12 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #13 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #14 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #15 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #16 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #17 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #18 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #19 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #20 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #21 pc 00200159 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb1ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+444)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #22 pc 003f8fa5 /system/lib/libart.so (MterpInvokeVirtualQuickRange+472)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #23 pc 00402794 /system/lib/libart.so (ExecuteMterpImpl+30100)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #24 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #25 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #26 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #27 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #28 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #29 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.186 16283-16283/? A/DEBUG: #30 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #31 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #32 pc 003f77b9 /system/lib/libart.so (MterpInvokeStatic+184)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #33 pc 003feb14 /system/lib/libart.so (ExecuteMterpImpl+14612)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #34 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #35 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #36 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #37 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #38 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #39 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #40 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #41 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #42 pc 003f8c87 /system/lib/libart.so (MterpInvokeVirtualQuick+598)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #43 pc 00402714 /system/lib/libart.so (ExecuteMterpImpl+29972)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #44 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #45 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #46 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #47 pc 003f7391 /system/lib/libart.so (MterpInvokeInterface+1080)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #48 pc 003feb94 /system/lib/libart.so (ExecuteMterpImpl+14740)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #49 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #50 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #51 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #52 pc 003f7391 /system/lib/libart.so (MterpInvokeInterface+1080)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #53 pc 003feb94 /system/lib/libart.so (ExecuteMterpImpl+14740)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #54 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #55 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.187 16283-16283/? A/DEBUG: #56 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #57 pc 003f7391 /system/lib/libart.so (MterpInvokeInterface+1080)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #58 pc 003feb94 /system/lib/libart.so (ExecuteMterpImpl+14740)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #59 pc 001e6bc1 /system/lib/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+340)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #60 pc 001eb36f /system/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+142)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #61 pc 001ff535 /system/lib/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+564)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #62 pc 003f7391 /system/lib/libart.so (MterpInvokeInterface+1080)
2020-03-12 12:07:09.188 16283-16283/? A/DEBUG: #63 pc 003feb94 /system/lib/libart.so (ExecuteMterpImpl+14740)
2020-03-12 12:07:11.258 2748-2748/? E//system/bin/tombstoned: Tombstone written to: /data/tombstones/tombstone_07
2020-03-12 12:07:11.266 2693-2693/? E/audit: type=1701 audit(1584011231.251:1220): auid=4294967295 uid=10219 gid=10219 ses=4294967295 subj=u:r:untrusted_app:s0:c512,c768 pid=16280 comm=".pl/..." exe="/system/bin/app_process32" sig=7
2020-03-12 12:07:11.301 2962-16284/? E/ActivityManager: Found activity ActivityRecord{dc4c01c u0 com.efgd.music/.MainActivity t-1 f} in proc activity list using null instead of expected ProcessRecord{ca03534 16010:com.efgd.music/u0a219}
2020-03-12 12:07:11.398 3338-3338/? E/SKBD: bbw getInstance start
2020-03-12 12:07:11.398 3338-3338/? E/SKBD: bbw sendSIPInformation state: 6 isAbstractKeyboardView : true
2020-03-12 12:07:11.404 3338-16293/? E/SKBD: bbw sending null keyboardInfo as SIP is closed
2020-03-12 12:07:11.419 5224-5254/? E/PBSessionCacheImpl: sessionId[22976978907188413] not persisted.
Thank you for the awesome library! Is it possible to add a 128-bit version for one of the hash functions, such as Murmur3?
I tested my implementation for Java 9+ with VarHandles instead of Unsafe.
In my test case (hashing multiple files with a total size of ~ 1 GB) I noticed a huge speed improvement
Is it possible to publish separate version for modern JDKs without dependency on Unsafe?
Hello.
I'm trying to understand the design decision behind requiring hashing to be stateless.
Each call to hashByte/hashLong etc. starts with the state equal to the original seed and calls finalize at the end. As far as I can tell, this means that you can't chain these hash calls (like you can with Guava, for example) unless you keep passing finalized()
hashes around.
Am I missing something? It seems like we should require the user to call finalize and to have a stateful long hash
variable that can be reset()
in each implementation. This allows the user to replicate the current behavior by calling reset()-hashByte()-finalize()
but also to chain calls to each of the hash functions (reset()-hashByte()-hashLong()...finalize()
).
Sorry for the n00b question, but I'm trying to figure out how to use this a project that is neither using Gradle or Maven.
Is there a suitable guide somewhere? I'm using IntelliJ.
The xxHash implementation doesn't produce predictable results when compared to python for strings (as in the given example.
I believe this is due to java's handling of character arrays: https://codeahoy.com/2016/05/08/the-char-type-in-java-is-broken/
We get predictable results when we dump to a byte array instead:
LongHashFunction.xx().hashBytes("test".getBytes())
gets the same output as
xxhash.xxh64('test').intdigest()
which is the result that we'd expect.
We can't figure out how to have the python implementation find the same key for LongHashFunction.xx().hashChars("test")
If there's a way for other languages to get the same output for a given input string, it'd be nice to have it documented.
zero-allocation-hashing version: 0.12
java version: 14
jvm: openjdk
We are experiencing class loading deadlock when using concurrently LongTupleHashFunction
and LongHashFunction
, here is the thread dump of the app when the deadlock occurs:
"default.work-dispatcher-39" #98 prio=5 os_prio=0 cpu=1399.65ms elapsed=747.54s tid=0x00007f54fc07b000 nid=0x7a in Object.wait() [0x00007f540f4f2000]
java.lang.Thread.State: RUNNABLE
at net.openhft.hashing.LongHashFunction.murmur_3(LongHashFunction.java:223)
- waiting on the Class initialization monitor for net.openhft.hashing.MurmurHash_3
(...)
"default.work-dispatcher-48" #108 prio=5 os_prio=0 cpu=2235.06ms elapsed=747.42s tid=0x00007f5410038000 nid=0x84 in Object.wait() [0x00007f540e4b5000]
java.lang.Thread.State: RUNNABLE
at net.openhft.hashing.LongHashFunction.murmur_3(LongHashFunction.java:223)
- waiting on the Class initialization monitor for net.openhft.hashing.MurmurHash_3
(...)
"default.work-dispatcher-50" #109 prio=5 os_prio=0 cpu=211.69ms elapsed=747.42s tid=0x00007f541003c000 nid=0x85 in Object.wait() [0x00007f540e3b2000]
java.lang.Thread.State: RUNNABLE
at net.openhft.hashing.CompactLatin1CharSequenceAccess.<clinit>(CompactLatin1CharSequenceAccess.java:83)
- waiting on the Class initialization monitor for net.openhft.hashing.UnsafeAccess
at net.openhft.hashing.ModernCompactStringHash.<clinit>(ModernCompactStringHash.java:14)
at net.openhft.hashing.Util.<clinit>(Util.java:41)
at net.openhft.hashing.MurmurHash_3.<clinit>(MurmurHash_3.java:38)
at net.openhft.hashing.LongHashFunction.murmur_3(LongHashFunction.java:223)
(...)
"default.work-dispatcher-103" #195 prio=5 os_prio=0 cpu=378.90ms elapsed=727.08s tid=0x00007f54c8009800 nid=0xe1 in Object.wait() [0x00007f539fdc6000]
java.lang.Thread.State: RUNNABLE
at net.openhft.hashing.UnsafeAccess.<clinit>(UnsafeAccess.java:30)
- waiting on the Class initialization monitor for net.openhft.hashing.Util
at net.openhft.hashing.LongTupleHashFunction.<clinit>(LongTupleHashFunction.java:948)
(...)
NB: I have sanitised the stack traces to remove mention of our own classes.
After Clone the repository I try to build this project in IDEA with JDK8.
I receive a error in file src/main/java-stub/java/lang/Math.java
because JDK8 don't have Math.multiplyHigh
and try-catch can't handle this.
Hi,
I am not sure if I am using murmur3
function correctly but the result seems different for hashChars()
and hashBytes()
String input = "abc123";
System.out.println(LongHashFunction.murmur_3().hashChars(input));
System.out.println(LongHashFunction.murmur_3().hashBytes(input.getBytes()));
Output:
-4878457159164508227
-7432123028918728600
Which function should be used for String
inputs?
Shouldn't both functions produce the same result?
Addition question:
How can I get the equivalent output (128-bit hex string) of this:
Hashing.murmur3_128().newHasher().putString(s, Charsets.UTF_8).hash().toString()
[INFO] --- maven-release-plugin:3.0.0-M1:perform (default-cli) @ zero-allocation-hashing ---
[INFO] phase verify-release-configuration
[INFO] starting perform goal, composed of 3 phases: verify-completed-prepare-phases, checkout-project-from-scm, run-perform-goals
[INFO] [perform] 1/3 verify-completed-prepare-phases
[INFO] [perform] 2/3 checkout-project-from-scm
[INFO] Checking out the project to perform the release ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.289 s
[INFO] Finished at: 2021-06-19T16:27:05+02:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "guarded" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:perform (default-cli) on project zero-allocation-hashing: Execution default-cli of goal org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:perform failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:perform: java.lang.AbstractMethodError: org.apache.maven.scm.command.checkout.AbstractCheckOutCommand.executeCheckOutCommand(Lorg/apache/maven/scm/provider/ScmProviderRepository;Lorg/apache/maven/scm/ScmFileSet;Lorg/apache/maven/scm/ScmVersion;ZZ)Lorg/apache/maven/scm/command/checkout/CheckOutScmResult;
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>org.apache.maven.plugins:maven-release-plugin:3.0.0-M1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/teamcity/.m2/repository/org/apache/maven/plugins/maven-release-plugin/3.0.0-M1/maven-release-plugin-3.0.0-M1.jar
[ERROR] urls[1] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-gitexe/1.8.1/maven-scm-provider-gitexe-1.8.1.jar
[ERROR] urls[2] = file:/home/teamcity/.m2/repository/regexp/regexp/1.3/regexp-1.3.jar
[ERROR] urls[3] = file:/home/teamcity/.m2/repository/commons-io/commons-io/2.2/commons-io-2.2.jar
[ERROR] urls[4] = file:/home/teamcity/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
[ERROR] urls[5] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-git-commons/1.8.1/maven-scm-provider-git-commons-1.8.1.jar
[ERROR] urls[6] = file:/home/teamcity/.m2/repository/org/sonatype/sisu/sisu-inject-bean/1.4.2/sisu-inject-bean-1.4.2.jar
[ERROR] urls[7] = file:/home/teamcity/.m2/repository/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar
[ERROR] urls[8] = file:/home/teamcity/.m2/repository/org/apache/maven/release/maven-release-manager/3.0.0-M1/maven-release-manager-3.0.0-M1.jar
[ERROR] urls[9] = file:/home/teamcity/.m2/repository/org/apache/maven/release/maven-release-api/3.0.0-M1/maven-release-api-3.0.0-M1.jar
[ERROR] urls[10] = file:/home/teamcity/.m2/repository/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-6/plexus-interactivity-api-1.0-alpha-6.jar
[ERROR] urls[11] = file:/home/teamcity/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar
[ERROR] urls[12] = file:/home/teamcity/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[13] = file:/home/teamcity/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] urls[14] = file:/home/teamcity/.m2/repository/org/apache/maven/shared/maven-invoker/2.2/maven-invoker-2.2.jar
[ERROR] urls[15] = file:/home/teamcity/.m2/repository/org/apache/commons/commons-lang3/3.7/commons-lang3-3.7.jar
[ERROR] urls[16] = file:/home/teamcity/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar
[ERROR] urls[17] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-providers-standard/1.11.2/maven-scm-providers-standard-1.11.2.pom
[ERROR] urls[18] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-accurev/1.11.2/maven-scm-provider-accurev-1.11.2.jar
[ERROR] urls[19] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-bazaar/1.11.2/maven-scm-provider-bazaar-1.11.2.jar
[ERROR] urls[20] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-clearcase/1.11.2/maven-scm-provider-clearcase-1.11.2.jar
[ERROR] urls[21] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvsexe/1.11.2/maven-scm-provider-cvsexe-1.11.2.jar
[ERROR] urls[22] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvs-commons/1.11.2/maven-scm-provider-cvs-commons-1.11.2.jar
[ERROR] urls[23] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvsjava/1.11.2/maven-scm-provider-cvsjava-1.11.2.jar
[ERROR] urls[24] = file:/home/teamcity/.m2/repository/org/netbeans/lib/cvsclient/20060125/cvsclient-20060125.jar
[ERROR] urls[25] = file:/home/teamcity/.m2/repository/ch/ethz/ganymed/ganymed-ssh2/build210/ganymed-ssh2-build210.jar
[ERROR] urls[26] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-jgit/1.11.2/maven-scm-provider-jgit-1.11.2.jar
[ERROR] urls[27] = file:/home/teamcity/.m2/repository/org/eclipse/jgit/org.eclipse.jgit/4.5.4.201711221230-r/org.eclipse.jgit-4.5.4.201711221230-r.jar
[ERROR] urls[28] = file:/home/teamcity/.m2/repository/com/jcraft/jsch/0.1.53/jsch-0.1.53.jar
[ERROR] urls[29] = file:/home/teamcity/.m2/repository/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar
[ERROR] urls[30] = file:/home/teamcity/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
[ERROR] urls[31] = file:/home/teamcity/.m2/repository/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar
[ERROR] urls[32] = file:/home/teamcity/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar
[ERROR] urls[33] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-hg/1.11.2/maven-scm-provider-hg-1.11.2.jar
[ERROR] urls[34] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-perforce/1.11.2/maven-scm-provider-perforce-1.11.2.jar
[ERROR] urls[35] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-starteam/1.11.2/maven-scm-provider-starteam-1.11.2.jar
[ERROR] urls[36] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-svnexe/1.11.2/maven-scm-provider-svnexe-1.11.2.jar
[ERROR] urls[37] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-synergy/1.11.2/maven-scm-provider-synergy-1.11.2.jar
[ERROR] urls[38] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-vss/1.11.2/maven-scm-provider-vss-1.11.2.jar
[ERROR] urls[39] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-tfs/1.11.2/maven-scm-provider-tfs-1.11.2.jar
[ERROR] urls[40] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-integrity/1.11.2/maven-scm-provider-integrity-1.11.2.jar
[ERROR] urls[41] = file:/home/teamcity/.m2/repository/com/mks/api/mksapi-jar/4.10.9049/mksapi-jar-4.10.9049.jar
[ERROR] urls[42] = file:/home/teamcity/.m2/repository/org/codehaus/groovy/groovy-all/1.7.6/groovy-all-1.7.6.jar
[ERROR] urls[43] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-jazz/1.11.2/maven-scm-provider-jazz-1.11.2.jar
[ERROR] urls[44] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-manager-plexus/1.8/maven-scm-manager-plexus-1.8.jar
[ERROR] urls[45] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-provider-svn-commons/1.11.2/maven-scm-provider-svn-commons-1.11.2.jar
[ERROR] urls[46] = file:/home/teamcity/.m2/repository/org/apache/maven/shared/maven-artifact-transfer/0.9.1/maven-artifact-transfer-0.9.1.jar
[ERROR] urls[47] = file:/home/teamcity/.m2/repository/org/apache/maven/shared/maven-common-artifact-filters/3.0.1/maven-common-artifact-filters-3.0.1.jar
[ERROR] urls[48] = file:/home/teamcity/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.1.0/maven-shared-utils-3.1.0.jar
[ERROR] urls[49] = file:/home/teamcity/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar
[ERROR] urls[50] = file:/home/teamcity/.m2/repository/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
[ERROR] urls[51] = file:/home/teamcity/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.7.1/plexus-component-annotations-1.7.1.jar
[ERROR] urls[52] = file:/home/teamcity/.m2/repository/org/apache/maven/scm/maven-scm-api/1.11.2/maven-scm-api-1.11.2.jar
[ERROR] urls[53] = file:/home/teamcity/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.jar
[ERROR] urls[54] = file:/home/teamcity/.m2/repository/org/jdom/jdom/1.1/jdom-1.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import from realm ClassRealm[project>net.openhft:zero-allocation-hashing:0.15-SNAPSHOT, parent: ClassRealm[maven.api, parent: null]]]
[ERROR]
[ERROR] -----------------------------------------------------
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException
[teamcity@teamcity Zero-Allocation-Hashing]$
0.16 was tagged and published, but I don't see any release notes for it at https://github.com/OpenHFT/Zero-Allocation-Hashing/releases/ (or anywhere else I looked)?
according to travis notice:
Please be aware travis-ci.org will be shutting down in several weeks, with all accounts migrating to travis-ci.com. Please stay tuned here for more information.
From JDK10, Math.multiplyHigh()
is optimized as hardware instructions on x86 and arm. This can improve the performance of 64x64_fold
mix steps, which is used in many good hash, such as xx3, tiha, wyhash, mum and etc. On my local test, the wy_3()
is improved significantly.
Math.multiplyHigh()
was introduced from JDK9, so we need some tricks for compiling on JDK7&8, and avoid method reflection invocations.
UnsafeAccess class produces SIGBUS on Solaris/SPARC and crashes the JVM. I have originally found this issue with Gatling that ships OpenHFT ZAH as part of Scala's Zinc compiler. It crashes regardless of OS and JDK version, making the platform being the only thing left. Possible issue is alignment given that SPARC and ARM processors enforce it while x86 is permissive (see, for example: https://shipilev.net/jvm/anatomy-quarks/24-object-alignment/)
To reproduce the problem:
0. on a Solaris/SPARC platform (I can assist here if needed) clone the git repository
mvn package
I am attaching the error reports:
error_report.zip
The latest available one seems to be 0.12
Enabled the deployment into an OSGi runtime
Can we have a module-info.java
added so this can be used in modular projects?
Thanks!
It seems that the Murmur3 implementation has the same problem as that in Guava (see google/guava#3493).
I cannot run xxHash on devices running android 6. Stacktrace:
java.lang.NoSuchMethodError: No virtual method getByte(Ljava/lang/Object;J)B in class Lsun/misc/Unsafe; or its super classes (declaration of 'sun.misc.Unsafe' appears in /system/framework/core-libart.jar)
at net.openhft.hashing.UnsafeAccess.getByte(UnsafeAccess.java:87)
at net.openhft.hashing.UnsafeAccess.getUnsignedByte(UnsafeAccess.java:82)
at net.openhft.hashing.XxHash.fetch8(XxHash.java:51)
at net.openhft.hashing.XxHash.xxHash64(XxHash.java:150)
at net.openhft.hashing.XxHash$AsLongHashFunction.hash(XxHash.java:272)
at net.openhft.hashing.LongHashFunction.unsafeHash(LongHashFunction.java:453)
at net.openhft.hashing.LongHashFunction.hashBytes(LongHashFunction.java:494)
Can you add SipHash to your library? It is very popular function, fast, secure and immune to Hash DoS attacks.
I see that hashLong
rotates input on Big Endian platforms with Primitives.nativeToLittleEndian
:
Do I understand correct that hashLong(1L)
will not return the same result on Big and Little Endian platforms?
The project looks very nice but is lacking a LICENCE file.
// on little endian machine
CharSequenceAccess a = CharSequenceAccess.nativeCharSequenceAccess();
assertEquals((short)65, (short)a.getShort("AB", 0)); // correct
assertEquals((short)(66<<8), (short)a.getShort("AB", 1)); // exception
getByte()
is correct via a shift
parameter, but short/int/long seems to assume offset is always even. It is not exploded by Access interface that CharSequenceAccess cannot deal the alignment.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.