Comments (6)
I had a surprising amount of luck with blosc on a recent ARM core. Eagerly looking forward to c-blosc2's NEON support, but figured I'd chime in and suggest that there no fundamental issues with c-blosc or python-blosc on ARM.
Some stats and comparisons below.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
python-blosc version: 1.4.4
Blosc version: 1.11.2 ($Date:: 2017-01-27 #$)
Compressors available: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd']
Compressor library versions:
BloscLZ: 1.0.5
LZ4: 1.7.5
Snappy: 1.1.1
Zlib: 1.2.8
Zstd: 1.1.2
Python version: 3.6.0 (default, Dec 31 2016, 21:20:16)
[GCC 4.9.2]
Platform: Linux-3.4.113-sun8i-armv7l (#50 SMP PREEMPT Mon Nov 14 08:41:55 CET 2016)
Linux dist: debian 9.0
Processor: not recognized
Byte-ordering: little
Detected cores: 4
Number of threads to use by default: 4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
*** ctypes.memmove() *** Time for memcpy(): 0.015 s (93.57 MB/s)
Times for compressing/decompressing with clevel=5 and 4 threads
*** user input ***
*** blosclz , noshuffle *** 0.015 s (89.93 MB/s) / 0.010 s (138.32 MB/s) Compr. ratio: 2.7x
*** blosclz , shuffle *** 0.023 s (60.25 MB/s) / 0.012 s (112.71 MB/s) Compr. ratio: 2.3x
*** blosclz , bitshuffle *** 0.018 s (77.63 MB/s) / 0.021 s (66.76 MB/s) Compr. ratio: 7.3x
*** lz4 , noshuffle *** 0.008 s (177.14 MB/s) / 0.009 s (159.00 MB/s) Compr. ratio: 3.6x
*** lz4 , shuffle *** 0.010 s (131.29 MB/s) / 0.012 s (117.69 MB/s) Compr. ratio: 3.5x
*** lz4 , bitshuffle *** 0.015 s (89.97 MB/s) / 0.022 s (63.62 MB/s) Compr. ratio: 8.4x
*** lz4hc , noshuffle *** 0.071 s (19.30 MB/s) / 0.007 s (186.64 MB/s) Compr. ratio: 8.6x
*** lz4hc , shuffle *** 0.079 s (17.30 MB/s) / 0.014 s (95.99 MB/s) Compr. ratio: 6.2x
*** lz4hc , bitshuffle *** 0.062 s (22.23 MB/s) / 0.027 s (51.53 MB/s) Compr. ratio: 9.7x
*** snappy , noshuffle *** 0.008 s (173.87 MB/s) / 0.009 s (148.77 MB/s) Compr. ratio: 4.4x
*** snappy , shuffle *** 0.011 s (123.22 MB/s) / 0.016 s (85.16 MB/s) Compr. ratio: 4.4x
*** snappy , bitshuffle *** 0.015 s (89.02 MB/s) / 0.021 s (64.87 MB/s) Compr. ratio: 6.2x
*** zlib , noshuffle *** 0.047 s (29.26 MB/s) / 0.011 s (121.83 MB/s) Compr. ratio: 14.7x
*** zlib , shuffle *** 0.080 s (17.20 MB/s) / 0.022 s (63.61 MB/s) Compr. ratio: 9.4x
*** zlib , bitshuffle *** 0.059 s (23.50 MB/s) / 0.033 s (41.10 MB/s) Compr. ratio: 10.5x
*** zstd , noshuffle *** 0.113 s (12.21 MB/s) / 0.011 s (124.64 MB/s) Compr. ratio: 15.6x
*** zstd , shuffle *** 0.154 s (8.92 MB/s) / 0.026 s (52.56 MB/s) Compr. ratio: 9.9x
*** zstd , bitshuffle *** 0.116 s (11.86 MB/s) / 0.036 s (38.40 MB/s) Compr. ratio: 11.4x
Script is a slightly modified version of compress_ptr.py -
https://gist.github.com/itdaniher/544215e2ebd25495a95d9165633972c6
Tried a few different formats for my sampled analog signal compression - pysoundfile's ogg and flac encodings, python-lz4, and python's integrated zlib. Very happy with Blosc overall - stellar performance even without native SIMD (yet!)
from python-blosc.
Hi,
Yes, python-blosc is being developed in Intel and normally only checked with it. To overcome this problem we need to recognize the architecture prior to call the C compiler: this would allow to set compiler options accordingly. PR welcome!
from python-blosc.
Ah, I thought that because c-blosc is supposedly compatible with arm/RPi (at least, according to the cblosc 1.3.6 to 1.4.0 change notes), that python-blosc would be as well.
from python-blosc.
Any news on this? Perhaps a new version has fixed the issue? Or maybe it magically disappeared?
from python-blosc.
@FrancescAlted I'd be glad to order your choice of single board computer to better enable you to test on ARM CPUs. Also have plenty of recommendations, the ability to offer native hosted ARM continuous integration, etc. Contact me @gmail.com if you're interested! Thanks for Blosc-and-friends!
from python-blosc.
@itdaniher Thanks for reporting. As you had no fundamental flaws with C-Blosc in ARM, I am going to close this one. I already contacted you about the ARM board and CI for ARM. Thanks again.
from python-blosc.
Related Issues (20)
- Issues decompressing bytes from files HOT 1
- Replace obsolete `popen2` HOT 1
- Properly identify vendored `cpuinfo.py` version
- Blosc_ROOT cmake warning: Policy CMP0074 is not set HOT 2
- "RuntimeError: Cannot decompress" for a compressed sequence of more than 7240 zero bytes HOT 1
- Very bad compression on short inputs 1-127 bytes long HOT 5
- “python_requires” should be set with “>=3.6”, as blosc 1.10.6 is not compatible with all Python versions. HOT 2
- wrong setuptools build command
- Concatenate two blosc compressed bytes objects HOT 2
- LICENSES/BLOSC.txt HOT 4
- Rename default branch HOT 1
- Update pypi with latest blosc version HOT 3
- Wheel for Python 3.10 and Python 3.11 HOT 3
- Cannot install blosc 1.11.0 on apple M1 machine HOT 3
- decompress in fore-end HOT 1
- README link to python-blosc2 seems useful
- __pack_tensor__ must be made portable and not depend on Python HOT 2
- __pack_tensor__ should be in the beginning of the file to avoid seeking the whole file HOT 2
- Python 3.12 compatibility HOT 6
- Numpy 2 compatibility
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-blosc.