Comments (11)
braewoods
commented on June 24, 2024
Can you try forcing the ARM cpu_features.h file to be enabled for your M1 environment? I wonder if that would change anything.
from deflate.
braewoods
commented on June 24, 2024
Since I lack access to any MacOS M1 environment, it's not like I can test anything like this.
from deflate.
ThomasWaldmann
commented on June 24, 2024
That wouldn't help, the code there is linux specific. But I guess best is to continue in libdeflate's issue tracker and point them to the zlib-ng code i found.
from deflate.
braewoods
commented on June 24, 2024
Ok. I can't say I'm surprised. For a long time Linux was the only serious ARM target of note.
from deflate.
ThomasWaldmann
commented on June 24, 2024
I was curious about how borgbackup's currently bundled crc32 code performs on macOS 12 with M1 cpu (again on my local machine):
Name (time in us) Mean StdDev Median OPS
---------------------------------------------------------------------------------------------------------------
test_zlib_crc32 560.6912 (1.0) 14.5614 (1.0) 563.9375 (1.0) 1,783.5129 (1.0)
test_borg_crc32_slice8 7,326.7399 (13.07) 117.9650 (8.10) 7,324.9590 (12.99) 136.4864 (0.08)
have_clmul is False, thus borg_crc32_clmul is not available (only implemented on x64 within the code currently bundled into borg).
from deflate.
ThomasWaldmann
commented on June 24, 2024
Benchmarks done on github CI - (linux, x64):
Name (time in us) Mean StdDev Median OPS
---------------------------------------------------------------------------------------------------------------
test_borg_crc32_clmul 515.9855 (1.0) 19.2178 (1.0) 520.4060 (1.0) 1,938.0391 (1.0)
test_borg_crc32_slice8 3,958.2522 (7.67) 84.5450 (4.40) 3,973.1480 (7.63) 252.6368 (0.13)
test_zlib_crc32 7,500.5678 (14.54) 116.3165 (6.05) 7,520.1550 (14.45) 133.3232 (0.07)
Benchmarks done on github CI - (macOS, x64):
Name (time in ms) Mean StdDev Median OPS
---------------------------------------------------------------------------------------------------
test_zlib_crc32 3.1880 (1.0) 0.39 (5.85) 3.0442 (1.0) 313.6777 (1.0)
test_borg_crc32_slice8 4.6606 (1.46) 0.0656 (1.0) 4.6442 (1.53) 214.5655 (0.68)
from deflate.
ThomasWaldmann
commented on June 24, 2024
code: borgbackup/borg#6387 - it would also benchmark deflate.crc32 as soon is that is in a pypi release.
from deflate.
braewoods
commented on June 24, 2024
It makes me wonder how libdeflate would fair against zlib-ng. That might explain why Python on MacOS is so different. Whichever version is in active use may be using zlib-ng instead of regular zlib. If so, should we just import the zlib-ng code since it may be doing better than libdeflate?
from deflate.
ThomasWaldmann
commented on June 24, 2024
yeah, zlib-ng definitely also worth testing (but maybe a little bit off-topic here).
from deflate.
ThomasWaldmann
commented on June 24, 2024
Updated performance results using libdeflate 1.12 on macOS M1:
(borg-env) tw@mba2020 borg % borg benchmark cpu
Non-cryptographic checksums / hashes ===========================
crc32 (zlib, used) 1GB 0.055s
crc32 (libdeflate) 1GB 0.027s
xxh64 1GB 0.122s
Great update, it used to be slower, but now libdeflate 1.12 is twice as fast as zlib crc32 on macOS M1!
from deflate.
ThomasWaldmann
commented on June 24, 2024
guess this is solved by the new libdeflate.
from deflate.
Related Issues (20)
- Hi from python-isal HOT 4
- add deflate.crc32 api? HOT 1
- tests? HOT 1
- AttributeError: module 'deflate' has no attribute 'DeflateError'
- CI via github actions? HOT 4
- add tests for crc32
- update submodule to v1.10?
- new release? HOT 15
- add git tags? HOT 3
- columns discrepancy
- macos "build wheels" ran for 5h when i cancelled it
- fix pypy support? HOT 5
- Upgrade to libdeflate 1.18 HOT 1
- Python 3.12 Wheels HOT 1
- segmentation fault using deflate_decompress HOT 6
- Memory Leak HOT 4
- Windows Installation Issue HOT 4
- Add libdeflate gunzip logic to handle gzipped files which decompress to files larger than 4G or which contain multiple concatenated gzip files. HOT 2
- Can this library be used for streaming decompression? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deflate.