Comments (9)
@aklomp what are your ideas on this?
from base64.
Yes, support for the AVX* codecs is detected at runtime if they are compiled in.
IIRC, there are one or more X86 instructions that are not available on Atoms, I think movbe
might be one. What's the exact illegal instruction that the code is failing on? My guess is it's not an AVX* opcode, but one of these unavailable ones.
What I think is happening is that by enabling -mavx2
, you are implicitly telling the compiler "this code will be run on a machine that supports AVX2". And that gives the compiler a license to use instructions that are available on all AVX2-supporting platforms, such as movbe
.
The fix for this is indeed to not compile at the AVX support level, or maybe do something like march=
or mtune=
.
from base64.
Yes, support for the AVX* codecs is detected at runtime if they are compiled in.
No, that is for the library. For the benchmark codes are "forced" (if they are compiled in). And then the forced codec is tested using a test string. And this test crashes.
I made a patch that uses the detection code and solves this issue by not running the test string if there is no support in the CPU (on x86) if an additional flag is set. But I don't know if you like that since the comments mention explicitly forcing is for testing purposes.
from base64.
But that's the same thing, right? You're running code compiled for an AVX-level machine on a machine that has no AVX support. Surely the fix is to not compile the tests with AVX codecs enabled?
from base64.
Correct.
The problem happens only when you run the same benchmark binary on different machines.
from base64.
Ah, I think I understand the problem now. You expect the benchmark binary to be compiled for the lowest common denominator (a machine that supports only the plain
codec), and only the arch-specific codecs to use any more advanced instructions.
Well, you'd be correct, that's how the test Makefile
is set up. The test binaries should run on the lowest common denominator machine.
Are you compiling using the Makefile
, or using the new CMake
stuff?
from base64.
I would expect the benchmark (for x86) to run all codecs supported by the cpu - not crash. However, as the compiled in codecs are forced, an exception happens on the unsupported instructions.
I use both makefile and cmake, there is no difference. Patching so that the cpu detection code is run to see if a codec is really supported fixes the problem. This code is not working.
from base64.
Now I remember, it's been a while. Yeah, the base binary is built without any SIMD flags, but you're right, all codecs that are built with the current flags, will always be run (forced) by the binary. Forcing is necessary because otherwise the library would always choose the "best" codec. Forcing is a way to downgrade the codec to the one we want to test. That is what the comments mean when they talk about needing forcing for testing purposes.
My assumption was that the test binary didn't need to be portable because it's something that's only run by developers. That's maybe not a very good assumption, I can see the use case for wanting to benchmark across systems from a single packaged tool.
There are various ways to fix this. The cleanest one (and one that I've been thinking about in other contexts) is to expose the various codecs through some introspection interface so that the client can choose which codec to use. Then we could get rid of the whole runtime CPU feature detection thing, or make it optional. I'd like to make this change some time, but it won't be soon (because it's a lot of rework and testing).
Another way of fixing it would be to run the CPU feature detection code in the test binary and only run codecs which are compatible with the current CPU. That's probably more straightforward, if we can factor out the feature detection code so that it can be easily reused.
from base64.
Maybe you could have a look at PR #82 (draft) - this implements the latter.
from base64.
Related Issues (20)
- bin/base64: modernize the demo program
- Add a macro to calculate encoded size from raw size and vice versa
- Investigate `gf2p8affineqb` for the shuffle step
- enc: asm: add memory and flags as clobbers
- Integrate with google/oss-fuzz for continuous fuzz testing
- Create release 0.5.1
- Codec detection doesnโt work in test_base64 on musl libc HOT 3
- v0.5.1 breaks `base64 -d` on Alpine Linux (musl libc) HOT 15
- Build of 0.5.1 broken with MinGW HOT 15
- I made a online base64 decoder tool. We can use it HOT 1
- Simplify codepath selection HOT 4
- build failure using mingw32: `error: 'asm' operand has impossible constraints` HOT 4
- CMakeLists: update version to 0.5.2
- bin/base64: add command line options for decoding
- Add `BASE64_FORCE_INLINE` macro to always inline inner loop functions
- Decoding error on Windows (CRLF?) HOT 18
- Codepage error on Windows HOT 2
- Build fails on macOS: `make: objcopy: No such file or directory` HOT 2
- Add installation guide for lib HOT 2
- error: implicit declaration of function 'bit' is invalid in C99 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from base64.