Illegal instruction on Node 8 about carmen-cache HOT 3 CLOSED

am inclined to build portable binaries

👍 Then next steps (please take it from here) would be:

• merge mapbox/mason#635
• update to that package: 4.13-dev
• new carmen-cache release

I put up https://github.com/mapbox/carmen-cache/compare/portable-rocksdb originally to test this out. You could run with that branch or create a new one.

from carmen-cache.

@apendleton - looked into this with @aarthykc and @alliecrevier. We found that we can workaround this problem with either:

• source compiling carmen-cache on circle; or
• disabling -march=native in the compile flags for librocksdb.a in the mason package (refs mapbox/mason#635)

I think what is happening is that travis supports avx2 instructions but circle does not:

• On travis:
• avx2 shows up in the cat /proc/cpuinfo output
• -mavx2 shows up in the gcc -O2 -march=native -E -v - </dev/null 2>&1 | grep cc1 output
• On circle:
• avx2 does not show up in the cat /proc/cpuinfo
• -mno-avx2 shows up in the gcc -O2 -march=native -E -v - </dev/null 2>&1 | grep cc1 output

I see these avx ( I presume the avx2 versions) in the carmen-cache binary that was build for node v8:

# objdump -d lib/carmen.node | grep vbroadcastsd
  1e8bce:	c4 62 7d 19 15 b1 5f 	vbroadcastsd 0x95fb1(%rip),%ymm10        # 27eb88 <_ZTSSt23_Sp_counted_ptr_inplaceIN7rocksdb8LRUCacheESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE+0x338>
  1e8bd7:	c4 62 7d 19 0d b0 5f 	vbroadcastsd 0x95fb0(%rip),%ymm9        # 27eb90 <_ZTSSt23_Sp_counted_ptr_inplaceIN7rocksdb8LRUCacheESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE+0x340>
  1e8c4e:	c4 62 7d 19 15 41 5f 	vbroadcastsd 0x95f41(%rip),%ymm10        # 27eb98 <_ZTSSt23_Sp_counted_ptr_inplaceIN7rocksdb8LRUCacheESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE+0x348>
  1e8c57:	c4 62 7d 19 0d 40 5f 	vbroadcastsd 0x95f40(%rip),%ymm9        # 27eba0 <_ZTSSt23_Sp_counted_ptr_inplaceIN7rocksdb8LRUCacheESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE+0x350>

The same instructions are not in the node v6 binary, which explains why this is only happening for node v8.

both the Node 6 and Node 8 builds use the same precompiled RocksDB static binary from Mason, so I wouldn't expect them to be any different?

Right, that is the question. One thing to realize is that the RocksDB static binary from Mason is not identical between builds since we don't actually pull a binary from mason. Rather we source-compile rocksdb for each carmen-cache build (using only the source build scripts from mason) due to the need to keep a consistent clang++ version for LTO working (#119).

So, one explanation is that travis machines are variable: some support avx2 and some do not and you got two different types of CPU between the node v6 and v8 builds when you last published carmen-cache. If this is possible then it indicates that this problem has been lurking for some time, in the sense that we've been using -march=native in the rocksdb compile flags and this portability no-no finally bit us because of variability on travis.

Now, overall we should consider whether supporting avx2 instructions are important for rocksdb performance - I don't have bandwidth to help with that. But one way to ascertain that would be to test carmen-cache node v8 binaries for the v0.21.2 release against binaries build against https://github.com/mapbox/carmen-cache/compare/portable-rocksdb (which uses the mason package @alliecrevier created that makes the binary portable (in terms of avx usage)).

from carmen-cache.

Awesome @springmeyer, thanks for digging into this. Source-compiling on circle is probably something we could live with for Carmen tests, but some of the tests for repos that depend on Carmen are already dreadfully slow, so I'd rather avoid it if possible, and am inclined to build portable binaries for the moment. We've gotten by this long without avx2 binaries so I imagine we can continue to do so. Whether or not there would be performance improvements from enabling it in the future is a question I'm curious about as well, but I probably won't have the bandwidth either in the immediate future, so I'd say we should probably put a pin in it and circle back in the fall (along, possibly, with upgrading our RocksDB version).

from carmen-cache.