During initial sync with bitcoin-core and btcd, temporary delays are common, but with libbitcoin-server delays seem more frequent and longer lasting.

During initial sync, libbitcoin-server will occasionally idle (while still claiming 10% CPU), but without syncing any blocks for many minutes. After 30 or so mins have passed with no syncing, I usually just restart bs. Restarting bs has always resulted in immediate syncing of blocks in these cases.

I have an strace of this delay occuring where I waited it out with no restart.

This does not reproduce on Linux or Windows, and appears consistently on OSX once the blockchain has achieved some (unknown, less than 164,500 blocks) size. The shutdown appears to succeed and subsequent startups work correctly.

Please wait while server is stopping...
Server stopped cleanly.
Please wait while files are unmapped...
18:27:41.290825 INFO [database] Unmapping: "blockchain/stealth_rows"
18:27:41.290976 INFO [database] Unmapping: "blockchain/stealth_index"
18:27:41.291075 INFO [database] Unmapping: "blockchain/history_rows"
18:27:41.291233 INFO [database] Unmapping: "blockchain/history_lookup"
18:27:41.293490 INFO [database] Unmapping: "blockchain/txs"
18:27:41.295112 INFO [database] Unmapping: "blockchain/spends"
18:27:41.297537 INFO [database] Unmapping: "blockchain/blocks_rows"
18:27:41.297644 INFO [database] Unmapping: "blockchain/blocks_lookup"
bs(1312,0x7fff7a6fc300) malloc: *** error for object 0x7fb66c81fa08: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Please wait while server is stopping...
Abort trap: 6

bs sails past this block after restarting, but this seemed worth reporting.

16:49:32.683464 INFO [poller]: Block #250307 00000000000000445efd25b8a3635275b31c18c2915882f09598f089d78b5c06
Bus error (core dumped)

bs.error.log around that time:

16:48:57.839239 WARNING [poller]: Storing block 0000000000000034237ac86c6c7fdc2c96cd0bcf43ce128381b50e2bb8c3e57d: Matching previous object found
16:49:00.345434 WARNING [node]: Failed to store transaction in mempool 40b3e10d3330539d4a5a6388b6f7147e4f1f29c4eaf1ffa595fdc3d6c1366d56: Spent input not found
16:49:00.345489 WARNING [node]: Failed to store transaction in mempool 6f6d8039af3ae4fc3785de48dba57a3286c427e4d2a8cf6e3e453320f9b7202c: Spent input not found
16:49:00.345526 WARNING [node]: Failed to store transaction in mempool 20717107a83e4700eb5b5f1b794894bd1705d6b14aba8fe510f05733e276cef6: Spent input not found
16:49:05.559918 WARNING [poller]: Orphan block 000000000000006a199f46969e03ac657b929f3c3490eaf555eaf4c01fd3a1f1
16:49:05.560317 WARNING [poller]: Orphan block 0000000000000067d935074609fca5609ece9df3406b351a8bb0703264770c48
16:49:05.561160 WARNING [poller]: Orphan block 00000000000000026fc72184e1c6177e00941fc51d860e9ee28d6099d5b5cb7c
16:49:05.562705 WARNING [poller]: Orphan block 000000000000000971d821ddb0c093de3a056bdbd3c23449575a04439a8cdd34
16:49:05.563383 WARNING [poller]: Orphan block 00000000000000351c159820729dffe5308a6337f34911c581ed3aece0aa3e2d
16:49:05.563731 WARNING [poller]: Orphan block 00000000000000583a7a6502166e4575cf08c12175e5479c453784bbe05b0862
16:49:05.564020 WARNING [poller]: Orphan block 0000000000000054508d72a01fe7a56cf76ee354819def13f2a64a38aa085afe
16:49:05.564518 WARNING [poller]: Storing block 00000000000000674bad2d7e856edced5cd6c62360ba3e3212358d63b984172a: Matching previous object found
16:49:05.565926 WARNING [poller]: Orphan block 0000000000000071ee2cb57493c41bbe4472fb83e58cf36036707184a14f343b
16:49:05.567798 WARNING [poller]: Orphan block 0000000000000036cfc5dce00eefc84f7690ae3e83b4d0eeba1422fa6062e6e9
16:49:05.568632 WARNING [poller]: Orphan block 000000000000002189b5afe112ee730f9884f78cce6629bf74bb6659a65082e1
16:49:05.569048 WARNING [poller]: Orphan block 0000000000000033ecda049490b6880b3b018dca17ed945b1396eb1c0e6d99c4
16:49:13.080514 WARNING [poller]: Storing block 0000000000000082553e6860c7bb89b9a515a17ad760c161670cbfe064dd20a0: Matching previous object found
16:49:13.081426 WARNING [poller]: Storing block 000000000000000e09670e36441080e2bb250cef013876c0255760a0da0e2987: Matching previous object found
16:49:13.540619 WARNING [node]: Failed to store transaction in mempool 102491ec595baa891231e3ec794476e9f1aa7055196c96eb6a0760df5962ded5: Spent input not found
16:49:13.540746 WARNING [node]: Failed to store transaction in mempool 6b9f651b043379e85e4dfca1beb37f713c4e2aa36006c123df3b82e508d8e6df: Spent input not found
16:49:18.339662 WARNING [poller]: Storing block 000000000000007b10347cf5f239e2b5e20c556fe7e7088d8f02285616b6e93c: Matching previous object found
16:49:19.214086 WARNING [poller]: Storing block 000000000000001e86b9ed31471601dad32e650c6fe474463b6d2525a118f13b: Matching previous object found
16:49:24.714063 WARNING [poller]: Orphan block 0000000000000006faef57eaefab78a22f1dc465064a34b2c44a364013f9bae2
16:49:24.734247 WARNING [poller]: Orphan block 000000000000002c5a443a3a822d4c3804e077600a07b3cb2bf74c046edc2c9b
16:49:24.898967 WARNING [poller]: Orphan block 0000000000000010f6abc8c90e560cd4e6e18ffe33fae2e01bad0b4360f9675b
16:49:24.952626 WARNING [poller]: Orphan block 0000000000000036d5285cc95cac3a3f5394d0d617b7f27f6e576e1005c6bc46
16:49:24.952825 WARNING [poller]: Orphan block 00000000000000113c76e7506192d73f346300e95003c325c9e668bda75ffcf3
16:49:24.953053 WARNING [poller]: Orphan block 0000000000000051c26ca89df84471f7d09093bc49787e2f7f0540e0ac4c92d1

bs.debug.log around that time:

16:49:30.921184 INFO [poller]: Block #250303 00000000000000484378057b7bf000cbdb896ef43123ccb153bf4d0f02bb5ec6
16:49:30.927232 DEBUG [network]: r: tx (372 bytes)
16:49:30.927462 DEBUG [network]: r: tx (374 bytes)
16:49:30.928292 DEBUG [network]: r: tx (257 bytes)
16:49:30.928406 DEBUG [network]: r: tx (1263 bytes)
16:49:30.931690 DEBUG [network]: r: tx (799 bytes)
16:49:30.933662 DEBUG [network]: r: tx (258 bytes)
16:49:30.933777 DEBUG [network]: r: tx (225 bytes)
16:49:30.933835 DEBUG [network]: r: tx (441 bytes)
16:49:30.933906 DEBUG [network]: r: tx (4347 bytes)
16:49:30.962004 DEBUG [network]: r: tx (4113 bytes)
16:49:30.975618 DEBUG [network]: r: tx (586 bytes)
16:49:30.975693 DEBUG [network]: r: tx (1277 bytes)
16:49:30.980806 DEBUG [network]: r: tx (258 bytes)
16:49:30.980913 DEBUG [network]: r: tx (192 bytes)
16:49:30.981026 DEBUG [network]: r: tx (1943 bytes)
16:49:30.986988 DEBUG [network]: r: tx (374 bytes)
16:49:30.992398 DEBUG [network]: r: tx (374 bytes)
16:49:30.992548 DEBUG [network]: r: tx (668 bytes)
16:49:30.992726 DEBUG [network]: r: tx (405 bytes)
16:49:30.998082 DEBUG [network]: r: tx (781 bytes)
16:49:30.998261 DEBUG [network]: r: tx (406 bytes)
16:49:31.010349 DEBUG [network]: r: tx (374 bytes)
16:49:31.010491 DEBUG [network]: r: tx (191 bytes)
16:49:31.010562 DEBUG [network]: r: tx (226 bytes)
16:49:31.010647 DEBUG [network]: r: tx (406 bytes)
16:49:31.010725 DEBUG [network]: r: tx (554 bytes)
16:49:31.014216 DEBUG [network]: r: tx (226 bytes)
16:49:31.135839 DEBUG [network]: r: tx (371 bytes)
16:49:31.167731 DEBUG [network]: r: tx (372 bytes)
16:49:31.167895 DEBUG [network]: r: tx (226 bytes)
16:49:31.167990 DEBUG [network]: r: tx (226 bytes)
16:49:31.168073 DEBUG [network]: r: tx (257 bytes)
16:49:31.168227 DEBUG [network]: r: tx (762 bytes)
16:49:31.171464 DEBUG [network]: r: tx (374 bytes)
16:49:31.176128 DEBUG [network]: r: tx (226 bytes)
16:49:31.176204 DEBUG [network]: r: tx (405 bytes)
16:49:31.176266 DEBUG [network]: r: tx (520 bytes)
16:49:31.176335 DEBUG [network]: r: tx (1405 bytes)
16:49:31.182470 DEBUG [network]: r: tx (226 bytes)
16:49:31.188211 DEBUG [network]: r: tx (226 bytes)

bs: bb7afdc
bx: 7185278805fefdb1f7eacaf705ba8e065017012f

Built both today, changed BITCOIN_OPTIONS and BITCOIN_BLOCKCHAIN_OPTIONS in install.sh to have --enable-testnet for both.

Generated certificates with

bx cert-new server.private
bx cert-new client.private
bx cert-public server.private server.public
bx cert-public client.private client.public

Edited configs:

bs.cfg

---
[identity]
# The path to the ZPL-encoded server private certificate file.
cert_file = /home/veox/usr/etc/libbitcoin/server.private
# The directory for ZPL-encoded client public certificate files, allows anonymous clients if not set.
client_certs_path = /home/veox/usr/etc/libbitcoin

bx.cfg

---
[testnet]
# The URL of the Libbitcoin/Obelisk testnet server.
url = tcp://localhost:9091
# The Z85-encoded public key of the server certificate.
server_cert_key = N3EY58Dy:K-PMiiZObPa[53EO[/vq@wkBNiBkG}R
# The path to the ZPL-encoded client private certificate file.
cert_file = /home/veox/usr/etc/libbitcoin/client.private

Running bs, and then bx fetch-height results in:

LIBZMQ I: security failure, self=CURVE peer=NULL

On server's side, several (2-4) lines of this are shown:

LIBZMQ I: security failure, self=NULL peer=CURVE

So it seems the server doesn't load its certificate.

In addition to that, can't check what server thinks it has loaded - bs --settings results in:

Configuration Settings:

[endpoints]
# The block publishing service endpoint, defaults to 'tcp://*:9093'.
block_publish = <VALUE>
# The heartbeat endpoint, defaults to 'tcp://*:9092'.
heartbeat = <VALUE>
# The query service endpoint, defaults to 'tcp://*:9091'.
service = <VALUE>
# The transaction publishing service endpoint, defaults to 'tcp://*:9094'.
tx_publish = <VALUE>

[general]
# The blockchain directory, defaults to 'blockchain'.
blockchain_path = <VALUE>
# The minimum height of the history database, defaults to 0.
history_height = <VALUE>
# The peer cache file path, defaults to 'hosts'.
hosts_file = <VALUE>

...

Trying to build from source I get:

make[1]: Entering directory `/tmp/buildd/libbitcoin-server-2.0.0+nmu1'
  CXX      src/src_bitcoin_server-config.o
  CXX      src/src_bitcoin_server-echo.o
  CXX      src/src_bitcoin_server-endpoint.o
src/endpoint.cpp:34:24: error: 'algorithm' is not a namespace-name
 using namespace boost::algorithm;
                        ^
src/endpoint.cpp:34:33: error: expected namespace-name before ';' token
 using namespace boost::algorithm;
                                 ^
make[1]: *** [src/src_bitcoin_server-endpoint.o] Error 1
make[1]: Leaving directory `/tmp/buildd/libbitcoin-server-2.0.0+nmu1'

Ubuntu 14.04 amd64. gcc is 4.8.2. libboost is 1.54

Used the following to build an Obelisk server for the testnet:

% ./install.sh --prefix=/Users/XXX/Projects/BX/bitcoin-server/test --build-dir=/Users/XXX/Projects/BX/bitcoin-server/build --with-boost=/Users/XXX/Projects/BX/boost/1.54.0 --without-consensus --enable-testnet

The server tries to establish connections on port 8333 instead of 18333.

% ./test/bin/bs bs-testnet.cfg
Using config file: "bs-testnet.cfg"
Press CTRL-C to stop server.
INFO [worker]: worker ready
INFO [database]: Mapping: "blockchain/blocks_lookup"
INFO [database]: Mapping: "blockchain/blocks_rows"
INFO [database]: Mapping: "blockchain/spends"
INFO [database]: Mapping: "blockchain/txs"
INFO [database]: Mapping: "blockchain/history_lookup"
INFO [database]: Mapping: "blockchain/history_rows"
INFO [database]: Mapping: "blockchain/stealth_index"
INFO [database]: Mapping: "blockchain/stealth_rows"
INFO [database]: Mapping: "blockchain/blocks_lookup"
INFO [database]: Mapping: "blockchain/blocks_rows"
INFO [database]: Mapping: "blockchain/spends"
INFO [database]: Mapping: "blockchain/txs"
INFO [database]: Mapping: "blockchain/history_lookup"
INFO [database]: Mapping: "blockchain/history_rows"
INFO [database]: Mapping: "blockchain/stealth_index"
INFO [database]: Mapping: "blockchain/stealth_rows"
23:32:17.415758 FATAL [node] ================= Startup =================
23:32:17.416564 ERROR [node] ================= Startup =================
23:32:17.416792 INFO [node] ================= Startup =================
23:32:17.419177 INFO [session] Set start height (0)
Server started.
23:32:17.645649 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:17.821733 INFO [network] Channel failure [88.112.146.73:8333] unable to reach remote host (boost code 2)
23:32:21.633041 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:22.483576 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:22.488846 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:32.667072 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:32.707722 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:43.778840 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:47.599534 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:32:52.860946 INFO [network] Channel failure [0.0.0.0:0] unable to reach remote host (boost code 2)
23:33:04.061238 INFO [network] Channel failure [82.192.64.136:8333] unable to reach remote host (boost code 2)
23:33:17.536704 INFO [network] Channel timeout [52.24.1.217:8333]
23:33:17.911306 INFO [network] Channel timeout [68.102.107.108:8333]
23:33:22.690015 INFO [network] Channel timeout [207.58.177.193:8333]
23:33:26.990805 INFO [network] Channel timeout [128.199.251.163:8333]

The full-node constructor is overriding server_node parameterization. This is the result of a hasty integration. Server config needs to be passed to node constructor override.

This is a libbitcoin-node problem, described in detail here:

libbitcoin/libbitcoin-node#43

In a --with-consensus build the server can become stalled in what appears to be a delay in performing full validation on a single block. Given that we don't perform full validation until we pass the last checkpoint, most blocks never hit this code. I've seen it happen a couple of times in the last day of blocks. Each time it has happened the server has been fully-synced and fell behind as a result.

The "flood" is affecting these blocks so it might possibly be a result of large numbers of transactions. It's not the result of a large tx mempool. I've let the mempool grow to 15,000 entries and processing continues on other blocks. I've reset (clearing the mempool) and the problem returned on startup. Only bypassing the block using a configured checkpoint allowed it to be accepted.

However I've seen blocks with larger numbers of txs and similar byte size pass normally, and operation continues normally even with 15,000 entries in the tx mempool. I'm syncing now with non-consensus builds to see if the problem may be tied to the consensus lib.

The issue does not coincide with excessive memory consumption, although that may occur as a consequence. When this problem occurs the shutdown action fails to complete. This indicates that at least one thread is failing to terminate. This doesn't happen on a normal stall. While the shutdown may be slow it eventually clears all threads.

It's very hard to follow a debug trace when the more significant events are not included in the log but instead flow to the error log.

This is a limitation of the underlying ZMQ library, which is implemented as ANSI for Windows. The library also reads and writes files directly from paths, not supporting buffering/streaming by the application. Need to modify ZMQ to not require path-based operation.

16:48:36.461661 DEBUG [network] Receive version [127.0.0.1:13704] (103 bytes)
16:48:36.454660 INFO [protocol] Accepted connection from [127.0.0.1:13704] (1 total)
16:48:36.471661 DEBUG [network] Send version [127.0.0.1:13704] (127 bytes)
16:48:36.477662 DEBUG [network] Receive verack [127.0.0.1:13704] (0 bytes)

This may result from the fact that another libbitcoin-server reports itself as localhost in the handshake.

Add log rotation feature.

Is there an existing way of making libbitcoinserver connections via the Tor network?

Using bx send-tx-p2p (when connecting locally) and bx send-tx-node.

Also closing the connection doesn't appear to clear the connection from the server queue.

This would make it symmetrical with watch-address, which is required for privacy.

See: libbitcoin/libbitcoin-explorer#207

CXXLD  bitcoin_server
src/bitcoin_server-config.o: In function `server::load_config(server::config_type&, boost::filesystem::path&)':
src/config.cpp:(.text+0x5b3): undefined reference to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const'
src/config.cpp:(.text+0x5c4): undefined reference to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const'
src/config.cpp:(.text+0x5d5): undefined reference to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const'
src/config.cpp:(.text+0x5e6): undefined reference to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const'
src/config.cpp:(.text+0x5f7): undefined reference to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const'
src/bitcoin_server-config.o:src/config.cpp:(.text+0x608): more undefined references to `libconfig::Setting::lookupValue(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) const' follow
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [bitcoin_server] Error 1

OK on Linux/GCC, OSX/CLang and VC++.

This is a libbitcoin-blockchain issue, see: libbitcoin/libbitcoin-blockchain#81

I tried to reuse a libbitcoin-server files from a build created about 60 days ago with a new build from code freshly downloaded all at once last night. I got a segmentation fault after OSX requested that I authorize the servers ports. So then I decided to delete the old blockchain with a "rm -fr blockchain", and start fresh only to find out a "df" indicated my MacBookPro did not have any filesystem disk space freed up. Somewhere around 80GB from a 256GB solid state disk drive is now in limbo. Any suggestions for regaining the disk space? (A disk drive lobotomy really kills the mood to experiment and learn...)

FYSA - I created the blockchain directory in my libbitcoin-server/etc/libbitcoin/ directory with "mkdir blockchain" then I executed the libbitcoin-blockchain/bin/initchain in the same directory as the mkdir. This allowed me to execute "../../bin/bitcoin-server" to have the blockchain built. Was within 20K blocks of having a full obelisk node built, but started to realize the MacBookPro was tending to run hot and slowed way... down. This was the motivation to make a brand new build with the latest code base using my own build scripts after figuring out what https://github.com/libbitcoin/libbitcoin-server/blob/master/install.sh was really accomplishing. I noticed that the latest code has openssl package dependencies (specifically crypto_CFLAGS and crypto_LIBS). Everything in the new build used the version2 branch except that for the new libbitcoin-consensus package that was based upon its version1 branch.

Build: bfda2fa

Windows edition: Windows 7 Professional, Service Pack 1
Processor: Intel Core 2 Quad CPU Q9550 @ 2.83GHz
Installed RAM: 8.00 GB
System type: 64-bit Operating System

Sync was completed through:

INFO [poller]: Block #162191 0000000000000217c7cb7d9880449bf2a9d198251bac0dca1c9b5e6485581a27

No console message on termination.
Contact me for logs.

During initial sync the process periodically terminates with:

Segmentation fault: 11

It restarts and continues fine, eventually hitting it again.

This could be resulting from a memory allocation failure due to limited RAM on this older machine (with 4GB RAM).

I am seeing what appears to be memory exhaustion on Ubuntu (8GB) and several times on OSX (4GB). In both scenarios it came on suddenly.

The Windows server (8GB) does not exhibit the behavior, but is not yet able to sync past about 156k blocks due to an independent issue, probably integer overflow.

Typically my Ubuntu machine runs at about 30% memory utilization, but at one point it had grown to 75% and the process appeared to be non-responsive. After restarting the process on Ubuntu the problem does eventually return.

A similar issue has occurred frequently on the OSX box, which is possibly related to the smaller amount of RAM.

I have seen problems in both cases, but not in all instances, stopping the server. The ^C signal is captured and responded to, but the process never actually ends, and must be terminated.

Running make check gives this:

make[2]: Entering directory `/home/bitcoin/build-libbitcoin-server/libbitcoin-server'
*** Error in `/home/bitcoin/build-libbitcoin-server/libbitcoin-server/test/.libs/lt-libbitcoin_server_test': double free or corruption (!prev): 0x0000000000a311c0 ***
FAIL: libbitcoin_server_test_runner.sh

Running under valgrind gives:

==13633== Memcheck, a memory error detector
==13633== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==13633== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==13633== Command: ./lt-libbitcoin_server_test
==13633== 

*** No errors detected
==13633== Invalid read of size 4
==13633==    at 0x596C64D: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x79D07B2: ??? (in /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.54.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633==  Address 0xa983350 is 16 bytes inside a block of size 28 free'd
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x6904C42: ??? (in /usr/local/lib/libboost_program_options.so.1.55.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633== 
==13633== Invalid free() / delete / delete[] / realloc()
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x79D07B2: ??? (in /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.54.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633==  Address 0xa983340 is 0 bytes inside a block of size 28 free'd
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x6904C42: ??? (in /usr/local/lib/libboost_program_options.so.1.55.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633== 
==13633== 
==13633== HEAP SUMMARY:
==13633==     in use at exit: 28 bytes in 1 blocks
==13633==   total heap usage: 742 allocs, 742 frees, 128,988 bytes allocated
==13633== 
==13633== LEAK SUMMARY:
==13633==    definitely lost: 28 bytes in 1 blocks
==13633==    indirectly lost: 0 bytes in 0 blocks
==13633==      possibly lost: 0 bytes in 0 blocks
==13633==    still reachable: 0 bytes in 0 blocks
==13633==         suppressed: 0 bytes in 0 blocks
==13633== Rerun with --leak-check=full to see details of leaked memory
==13633== 
==13633== For counts of detected and suppressed errors, rerun with: -v
==13633== ERROR SUMMARY: 4 errors from 2 contexts (suppressed: 0 from 0)

It seems like everything runs cleanly (*** No errors detected), but something goes wrong during the shutdown procedure. This is a freshly-updated Ubuntu 14.04 box, and the libbitcoin-server is the latest version (525e3950), compiled like sudo ./install.sh --build-boost --without-consensus.

Bitcoin_server installed using installer.sh script:

Version Information:

libbitcoin-server: 2.0.0
libbitcoin-node: 2.0.0
libbitcoin-blockchain: 2.0.0
libbitcoin [mainnet]: 2.8.0

./bitcoin_server -i
Using config file: "/usr/local/bitcoin-dev-pristine/etc/libbitcoin/server.cfg"
Please wait while initializing "blockchain" directory...
INFO [blockchain]: Mapping: "blockchain/blocks_lookup"
INFO [blockchain]: Mapping: "blockchain/blocks_rows"
INFO [blockchain]: Mapping: "blockchain/spends"
INFO [blockchain]: Mapping: "blockchain/txs"
INFO [blockchain]: Mapping: "blockchain/history_lookup"
INFO [blockchain]: Mapping: "blockchain/history_rows"
INFO [blockchain]: Mapping: "blockchain/stealth_index"
INFO [blockchain]: Mapping: "blockchain/stealth_rows"
INFO [blockchain]: Unmapping: "blockchain/stealth_rows"
INFO [blockchain]: Unmapping: "blockchain/stealth_index"
INFO [blockchain]: Unmapping: "blockchain/history_rows"
INFO [blockchain]: Unmapping: "blockchain/history_lookup"
INFO [blockchain]: Unmapping: "blockchain/txs"
INFO [blockchain]: Unmapping: "blockchain/spends"
INFO [blockchain]: Unmapping: "blockchain/blocks_rows"
INFO [blockchain]: Unmapping: "blockchain/blocks_lookup"
INFO [blockchain]: Mapping: "blockchain/blocks_lookup"
INFO [blockchain]: Mapping: "blockchain/blocks_rows"
INFO [blockchain]: Mapping: "blockchain/spends"
INFO [blockchain]: Mapping: "blockchain/txs"
INFO [blockchain]: Mapping: "blockchain/history_lookup"
INFO [blockchain]: Mapping: "blockchain/history_rows"
INFO [blockchain]: Mapping: "blockchain/stealth_index"
INFO [blockchain]: Mapping: "blockchain/stealth_rows"
Segmentation fault (core dumped)

gdb ./bitcoin_server -c core
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bitcoin_server...done.
[New LWP 1607]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./bitcoin_server -i'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 basic_string_inserter_imp<char, std::char_traits, std::allocator > (delim=34 '"', escape=38 '&',

string=..., os=...) at /usr/include/boost/io/detail/quoted_manip.hpp:79

79 if (*it == delim || *it == escape)
(gdb) bt
#0 basic_string_inserter_imp<char, std::char_traits, std::allocator > (delim=34 '"', escape=38 '&',

string="", os=...) at /usr/include/boost/io/detail/quoted_manip.hpp:79

#1 operator<< <char, std::char_traits, std::allocator > (proxy=..., proxy=..., proxy=..., os=...)

at /usr/include/boost/io/detail/quoted_manip.hpp:93

#2 operator<< <char, std::char_traits > (p=..., os=...) at /usr/include/boost/filesystem/path.hpp:659
#3 operator<< boost::filesystem::path (value=..., this=0x7ffebe119fd0)

at /usr/local/bitcoin-dev-pristine/include/bitcoin/bitcoin/utility/logger.hpp:55

#4 libbitcoin::chain::mmfile::~mmfile (this=0x7ffebe11b038, __in_chrg=)

at src/database/mmfile.cpp:73

#5 0x0000000000431f45 in ~stealth_database (this=0x7ffebe11aff8, __in_chrg=)

at /usr/local/bitcoin-dev-pristine/include/bitcoin/blockchain/database/stealth_database.hpp:31

#6 ~db_interface (this=0x7ffebe11ae00, __in_chrg=)

at /usr/local/bitcoin-dev-pristine/include/bitcoin/blockchain/db_interface.hpp:59

#7 init_chain (error=..., output=..., directory=...) at src/server.cpp:162
#8 libbitcoin::server::dispatch (argc=2, argv=, output=..., error=...) at src/server.cpp:336
#9 0x00007fe4dcc91b45 in __libc_start_main (main=0x41ab60 <main(int, char**)>, argc=2, argv=0x7ffebe11c598,

init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffebe11c588)
at libc-start.c:287

#10 0x000000000041cd16 in _start ()

OS=uname -s
if [[ $TRAVIS == true ]]; then
PARALLEL=$SEQUENTIAL
elif [[ $OS == Linux ]]; then
PARALLEL=nproc
elif [[ $OS == Darwin ]]; then
PARALLEL=sysctl -n hw.ncpu <= used to inquire number of core VMs on Darwin OSX that can be applied for parallelizing make jobs

This is a BX test case that is failing against libbitcoin-server but not obelisk\version1.

$ bx send-tx-node
0100000001b3807042c92f449bbf79b33ca59d7dfec7f4cc71096704a9c526dddf496ee0970100000069463044022039a36013301597daef41fbe593a02cc513d0b55527ec2df1050e2e8ff49c85c202204fcc407ce9b6f719ee7d009aeb8d8d21423f400a5b871394ca32e00c26b348dd2103c40cbd64c9c608df2c9730f49b0888c4db1c436e8b2b74aead6c6afbd10428c0ffffffff01905f0100000000001976a91418c0bd8d1818f1bf99cb1df2269c645318ef7b7388ac00000000

Each time the above call is executed (on ubuntu) where libbitcoin-server is running locally, the server emits the following console messages:

22:48:26.540978 INFO [protocol]: Accepted connection: 2
22:48:26.541496 ERROR [protocol]: Problem receiving addresses: Service stopped
22:48:26.541539 ERROR [session]: inventory: Service stopped
22:48:26.541547 ERROR [poller]: Received bad inventory: Service stopped
22:48:26.541571 ERROR [session]: get_blocks: Service stopped
22:48:26.541604 ERROR: recv_transaction: Service stopped
22:48:26.541632 ERROR [poller]: Received bad block: Service stopped
22:48:26.541662 ERROR [protocol]: Channel stopped (inbound): Service stopped
22:48:26.541690 ERROR [protocol]: Sending error: Service stopped

and the BX command returns:

Service stopped

Perhaps due to excessive hard kill's which were perhaps due to segfaults during the sync process, but now when I start bs it will run for a short time and then segfault

below is stack trace:

Process: bs [10511]
Path: /usr/local/bin/bs
Identifier: bs
Version: 0
Code Type: X86-64 (Native)
Parent Process: bash [9514]
Responsible: iTerm [8737]
User ID: 501

Date/Time: 2015-07-30 00:50:33.771 +0900
OS Version: Mac OS X 10.10.4 (14E46)
Report Version: 11
Anonymous UUID: 4DB43E3F-BF51-1844-5FF0-550D8CC619F1

Sleep/Wake UUID: 5F717EDD-A136-4440-BB81-9A80D3B356B9

Time Awake Since Boot: 68000 seconds
Time Since Wake: 23000 seconds

Crashed Thread: 4

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000088

VM Regions Near 0x88:
-->
__TEXT 0000000109f3c000-0000000109f3f000 [ 12K] r-x/rwx SM=COW /usr/local/bin/bs

Thread 0:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff869595c2 poll + 10
1 libczmq.3.dylib 0x000000010a821937 zpoller_wait + 311
2 libczmq++.0.dylib 0x000000010a7f857f czmqpp::poller::wait(int) + 31
3 libbitcoin-server.0.dylib 0x0000000109f72f95 libbitcoin::server::request_worker::poll() + 101 (worker.cpp:146)
4 libbitcoin-server.0.dylib 0x0000000109f529d6 libbitcoin::server::run(libbitcoin::server::settings_type&, std::__1::basic_ostream<char, std::__1::char_traits >&, std::__1::basic_ostream<char, std::__1::char_traits >&) + 7686 (dispatch.cpp:294)
5 libbitcoin-server.0.dylib 0x0000000109f4f068 libbitcoin::server::dispatch(int, char const**, std::__1::basic_istream<char, std::__1::char_traits >&, std::__1::basic_ostream<char, std::__1::char_traits >&, std::__1::basic_ostream<char, std::__1::char_traits >&) + 3176 (dispatch.cpp:341)
6 libdyld.dylib 0x00007fff8a4e05c9 start + 1

Thread 1:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libzmq.5.dylib 0x000000010a8add5f thread_routine(void*) + 47
2 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
3 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
4 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 2:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libzmq.5.dylib 0x000000010a8add5f thread_routine(void*) + 47
2 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
3 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
4 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 3:
0 libsystem_kernel.dylib 0x00007fff869595c2 poll + 10
1 libczmq.3.dylib 0x000000010a821937 zpoller_wait + 311
2 libczmq.3.dylib 0x000000010a835891 s_agent_task + 289
3 libczmq.3.dylib 0x000000010a844799 s_thread_shim + 41
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 4 Crashed:
0 libbitcoin.0.dylib 0x000000010a299a67 boost::asio::detail::kqueue_reactor::start_op(int, int, boost::asio::detail::kqueue_reactor::descriptor_state_&, boost::asio::detail::reactor_op_, bool, bool) + 87
1 libbitcoin.0.dylib 0x000000010a2ab806 boost::asio::detail::reactive_socket_service_base::start_op(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, int, boost::asio::detail::reactor_op_, bool, bool, bool) + 198
2 libbitcoin.0.dylib 0x000000010a2c0a84 void boost::asio::detail::reactive_socket_service_base::async_receive<boost::asio::mutable_buffers_1, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, libbitcoin::wrapped_handler_impl<std::__1::__bind<void (libbitcoin::network::channel_proxy::_)(boost::system::error_code const&, unsigned long), std::__1::shared_ptrlibbitcoin::network::channel_proxy, std::__1::placeholders::__ph<1>&, std::__1::placeholders::__ph<2>&> > > >(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, boost::asio::mutable_buffers_1 const&, int, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, libbitcoin::wrapped_handler_impl<std::__1::__bind<void (libbitcoin::network::channel_proxy::)(boost::system::error_code const&, unsigned long), std::__1::shared_ptrlibbitcoin::network::channel_proxy, std::__1::placeholders::__ph<1>&, std::__1::placeholders::__ph<2>&> > >&) + 420
3 libbitcoin.0.dylib 0x000000010a2a37d6 libbitcoin::network::channel_proxy::read_header() + 198
4 libbitcoin.0.dylib 0x000000010a2a36ce libbitcoin::network::channel_proxy::start() + 14
5 libbitcoin.0.dylib 0x000000010a2e22cb libbitcoin::network::connect_with_timeout::call_connect_handler(boost::system::error_code const&, boost::asio::ip::basic_resolver_iteratorboost::asio::ip::tcp, std::__1::function<void (std::__1::error_code const&, std::__1::shared_ptrlibbitcoin::network::channel)>) + 155
6 libbitcoin.0.dylib 0x000000010a2e2b0f boost::asio::detail::connect_op<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp, boost::asio::ip::basic_resolver_iteratorboost::asio::ip::tcp, boost::asio::detail::default_connect_condition, std::__1::__bind<void (libbitcoin::network::connect_with_timeout::)(boost::system::error_code const&, boost::asio::ip::basic_resolver_iteratorboost::asio::ip::tcp, std::__1::function<void (std::__1::error_code const&, std::__1::shared_ptrlibbitcoin::network::channel)>), std::__1::shared_ptrlibbitcoin::network::connect_with_timeout, std::__1::placeholders::__ph<1>&, std::__1::placeholders::__ph<2>&, std::__1::function<void (std::__1::error_code const&, std::__1::shared_ptrlibbitcoin::network::channel)>&> >::operator()(boost::system::error_code, int) + 1039
7 libbitcoin.0.dylib 0x000000010a2e3f75 boost::asio::detail::reactive_socket_connect_op<boost::asio::detail::connect_op<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp, boost::asio::ip::basic_resolver_iteratorboost::asio::ip::tcp, boost::asio::detail::default_connect_condition, std::__1::__bind<void (libbitcoin::network::connect_with_timeout::)(boost::system::error_code const&, boost::asio::ip::basic_resolver_iteratorboost::asio::ip::tcp, std::__1::function<void (std::__1::error_code const&, std::__1::shared_ptrlibbitcoin::network::channel)>), std::__1::shared_ptrlibbitcoin::network::connect_with_timeout, std::__1::placeholders::__ph<1>&, std::__1::placeholders::__ph<2>&, std::__1::function<void (std::__1::error_code const&, std::1::shared_ptrlibbitcoin::network::channel)>&> > >::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 133
8 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
9 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
10 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void) + 133
11 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
12 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
13 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 5:
0 libbitcoin.0.dylib 0x000000010a2d9a80 libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::do_relay(std::__1::error_code const&, libbitcoin::address_type const&) + 0
1 libbitcoin.0.dylib 0x000000010a2da5e2 boost::asio::detail::completion_handler<std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::1::error_code const&, libbitcoin::address_type const&>&> >::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 210
2 libbitcoin.0.dylib 0x000000010a2da491 void boost::asio::detail::strand_service::dispatch<std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::_1::error_code const&, libbitcoin::address_type const&>&> >(boost::asio::detail::strand_service::strand_impl&, std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::__1::error_code const&, libbitcoin::address_type const&>&>&) + 449
3 libbitcoin.0.dylib 0x000000010a2d9f5b boost::asio::async_result<boost::asio::handler_type<std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::__1::error_code const&, libbitcoin::address_type const&>&>, void ()>::type>::type boost::asio::io_service::strand::dispatch<std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::__1::error_code const&, libbitcoin::address_type const&>&> >(std::__1::__bind<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::__1::error_code const&, libbitcoin::address_type const&>&>&&) + 139
4 libbitcoin.0.dylib 0x000000010a2d9e13 void libbitcoin::wrapped_handler_impl<std::__1::__bind<void (libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::_)(std::__1::error_code const&, libbitcoin::address_type const&), std::__1::shared_ptr<libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&> >, std::__1::error_code const&, libbitcoin::address_type const&> >::operator()<>() + 99
5 libbitcoin.0.dylib 0x000000010a2a867a libbitcoin::subscriber<std::__1::error_code const&, libbitcoin::address_type const&>::relay(std::__1::error_code const&, libbitcoin::address_type const&) + 138
6 libbitcoin.0.dylib 0x000000010a2a4164 libbitcoin::network::channel_proxy::clear_subscriptions() + 276
7 libbitcoin.0.dylib 0x000000010a2a4030 libbitcoin::network::channel_proxy::stop_impl() + 624
8 libbitcoin.0.dylib 0x000000010a2a831d boost::asio::detail::completion_handler<std::__1::_bind<libbitcoin::network::channel_proxy::stop(std::1::error_code const&)::$0 const&> >::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 109
9 libbitcoin.0.dylib 0x000000010a319002 boost::asio::detail::strand_service::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 146
10 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
11 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
12 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
13 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
14 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
15 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 6:
0 libsystem_kernel.dylib 0x00007fff86958136 __psynch_cvwait + 10
1 libbitcoin.0.dylib 0x000000010a2fae87 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 263
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 7:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libbitcoin.0.dylib 0x000000010a2faf09 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 393
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 8:
0 libsystem_kernel.dylib 0x00007fff8695848a __semwait_signal + 10
1 libc++.1.dylib 0x00007fff89948d28 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2 libbitcoin-blockchain.0.dylib 0x000000010a1a2dcf boost::asio::detail::completion_handler<libbitcoin::chain::blockchain_impl::fetch(std::_1::function<bool (unsigned long long)>)::$2>::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 319 (thread:426)
3 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
4 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
5 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
6 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
8 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 9:
0 libsystem_kernel.dylib 0x00007fff8695848a __semwait_signal + 10
1 libc++.1.dylib 0x00007fff89948d28 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2 libbitcoin-blockchain.0.dylib 0x000000010a1a2dcf boost::asio::detail::completion_handler<libbitcoin::chain::blockchain_impl::fetch(std::_1::function<bool (unsigned long long)>)::$2>::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 319 (thread:426)
3 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
4 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
5 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
6 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
8 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 10:
0 libsystem_kernel.dylib 0x00007fff8695848a __semwait_signal + 10
1 libc++.1.dylib 0x00007fff89948d28 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2 libbitcoin-blockchain.0.dylib 0x000000010a1a2dcf boost::asio::detail::completion_handler<libbitcoin::chain::blockchain_impl::fetch(std::_1::function<bool (unsigned long long)>)::$2>::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 319 (thread:426)
3 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
4 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
5 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
6 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
8 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 11:
0 libsystem_kernel.dylib 0x00007fff8695848a __semwait_signal + 10
1 libc++.1.dylib 0x00007fff89948d28 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2 libbitcoin-blockchain.0.dylib 0x000000010a1a2dcf boost::asio::detail::completion_handler<libbitcoin::chain::blockchain_impl::fetch(std::_1::function<bool (unsigned long long)>)::$2>::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 319 (thread:426)
3 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
4 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
5 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
6 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
8 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 12:
0 libsystem_kernel.dylib 0x00007fff8695848a __semwait_signal + 10
1 libc++.1.dylib 0x00007fff89948d28 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2 libbitcoin-blockchain.0.dylib 0x000000010a1a2dcf boost::asio::detail::completion_handler<libbitcoin::chain::blockchain_impl::fetch(std::_1::function<bool (unsigned long long)>)::$2>::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 319 (thread:426)
3 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
4 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
5 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
6 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
8 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 13:
0 libbitcoin-blockchain.0.dylib 0x000000010a18ee79 libbitcoin::chain::mmfile::data() + 9 (mmfile.cpp:111)
1 libbitcoin-blockchain.0.dylib 0x000000010a1975f5 libbitcoin::chain::slab_allocator::get(unsigned long long) const + 21 (slab_allocator.cpp:70)
2 libbitcoin-blockchain.0.dylib 0x000000010a18c365 libbitcoin::chain::htdb_slab<std::__1::array<unsigned char, 32ul> >::get(std::__1::array<unsigned char, 32ul> const&) const + 197 (htdb_slab_list_item.ipp:67)
3 libbitcoin-blockchain.0.dylib 0x000000010a1ad809 libbitcoin::chain::validate_block_impl::transaction_exists(std::__1::array<unsigned char, 32ul> const&) + 25 (validate_block_impl.cpp:92)
4 libbitcoin-blockchain.0.dylib 0x000000010a182c1b libbitcoin::chain::validate_block::connect_block() + 139 (validate_block.cpp:386)
5 libbitcoin-blockchain.0.dylib 0x000000010a1acb13 libbitcoin::chain::organizer_impl::verify(unsigned long, std::__1::vectorstd::__1::shared_ptr<libbitcoin::chain::block_detail, std::__1::allocatorstd::__1::shared_ptr<libbitcoin::chain::block_detail > > const&, unsigned long) + 147 (organizer_impl.cpp:57)
6 libbitcoin-blockchain.0.dylib 0x000000010a178479 libbitcoin::chain::organizer::replace_chain(unsigned long, std::__1::vectorstd::__1::shared_ptr<libbitcoin::chain::block_detail, std::__1::allocatorstd::__1::shared_ptr<libbitcoin::chain::block_detail > >&) + 121 (organizer.cpp:77)
7 libbitcoin-blockchain.0.dylib 0x000000010a178343 libbitcoin::chain::organizer::process(std::__1::shared_ptrlibbitcoin::chain::block_detail) + 131 (memory:3919)
8 libbitcoin-blockchain.0.dylib 0x000000010a178262 libbitcoin::chain::organizer::start() + 306 (memory:4447)
9 libbitcoin-blockchain.0.dylib 0x000000010a19cf2b libbitcoin::chain::blockchain_impl::do_store(libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>) + 283 (atomic:913)
10 libbitcoin-blockchain.0.dylib 0x000000010a1aa305 void boost::asio::asio_handler_invoke<std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> > >(std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >&, ...) + 229 (functional:1705)
11 libbitcoin-blockchain.0.dylib 0x000000010a1a9c47 void boost::asio::detail::strand_service::dispatch<std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>&> > >(boost::asio::detail::strand_service::strand_impl&, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >&) + 87 (macos_fenced_block.hpp:51)
12 libbitcoin-blockchain.0.dylib 0x000000010a1a9acf boost::asio::async_result<boost::asio::handler_type<std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >, void ()>::type>::type boost::asio::io_service::strand::dispatch<std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> > >(std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >&&) + 271 (functional:1705)
13 libbitcoin-blockchain.0.dylib 0x000000010a1a935c boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >, boost::asio::detail::is_continuation_if_running>, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::1::error_code const&, libbitcoin::block_info)>&> > > >::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 428 (macos_fenced_block.hpp:51)
14 libbitcoin-blockchain.0.dylib 0x000000010a1a9123 void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >, boost::asio::detail::is_continuation_if_running>, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>&> > > >(boost::asio::detail::strand_service::strand_impl&, boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> >, boost::asio::detail::is_continuation_if_running>, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::__1::function<void (std::__1::error_code const&, libbitcoin::block_info)>&> > >&) + 323 (strand_service.hpp:88)
15 libbitcoin-blockchain.0.dylib 0x000000010a1a87fe boost::asio::detail::completion_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::__1::__bind<std::__1::__bind<void (libbitcoin::chain::blockchain_impl::_)(libbitcoin::block_type const&, std::__1::function<void (std::_1::error_code const&, libbitcoin::block_info)>), libbitcoin::chain::blockchain_impl, libbitcoin::block_type const&, std::_1::function<void (std::1::error_code const&, libbitcoin::block_info)>&> >, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service, boost::asio::detail::task_io_service_operation, boost::system::error_code const&, unsigned long) + 430 (tuple:398)
16 libbitcoin.0.dylib 0x000000010a2faff8 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 632
17 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
18 libbitcoin.0.dylib 0x000000010a31eb05 void std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
19 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
20 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
21 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 14:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libbitcoin.0.dylib 0x000000010a2faf09 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 393
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 15:
0 libsystem_kernel.dylib 0x00007fff86958136 __psynch_cvwait + 10
1 libbitcoin.0.dylib 0x000000010a2fae87 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 263
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 16:
0 libsystem_kernel.dylib 0x00007fff86958136 __psynch_cvwait + 10
1 libbitcoin.0.dylib 0x000000010a2fae87 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 263
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 17:
0 libsystem_kernel.dylib 0x00007fff86958136 __psynch_cvwait + 10
1 libbitcoin.0.dylib 0x000000010a2fae87 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 263
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a31eb05 void* std::__1::__thread_proxystd::__1::tuple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::$_0 >(void*) + 133
4 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
5 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
6 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 18:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libzmq.5.dylib 0x000000010a8add5f thread_routine(void*) + 47
2 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
3 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
4 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 19:
0 libsystem_kernel.dylib 0x00007fff8695921a kevent + 10
1 libzmq.5.dylib 0x000000010a8add5f thread_routine(void*) + 47
2 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
3 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
4 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 20:
0 libsystem_kernel.dylib 0x00007fff86958136 __psynch_cvwait + 10
1 libbitcoin.0.dylib 0x000000010a2fae87 boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lockboost::asio::detail::posix_mutex&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) + 263
2 libbitcoin.0.dylib 0x000000010a2fab45 boost::asio::detail::task_io_service::run(boost::system::error_code&) + 181
3 libbitcoin.0.dylib 0x000000010a2faa7a boost::asio::detail::posix_thread::funcboost::asio::detail::resolver_service_base::work_io_service_runner::run() + 42
4 libbitcoin.0.dylib 0x000000010a2fb7e3 boost_asio_detail_posix_thread_function + 19
5 libsystem_pthread.dylib 0x00007fff85b2f268 _pthread_body + 131
6 libsystem_pthread.dylib 0x00007fff85b2f1e5 _pthread_start + 176
7 libsystem_pthread.dylib 0x00007fff85b2d41d thread_start + 13

Thread 4 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007f8cc3504740 rcx: 0x0000000000000000 rdx: 0x0000000000000000
rdi: 0x00007f8cb9501490 rsi: 0x0000000000000000 rbp: 0x000000010abce7b0 rsp: 0x000000010abce6e0
r8: 0x0000000000002060 r9: 0x0000000000000000 r10: 0xffffffffffffffff r11: 0x0000000000000246
r12: 0x0000000000000000 r13: 0x00007f8cc1a5d190 r14: 0x0000000000000037 r15: 0x00007f8cb94056e0
rip: 0x000000010a299a67 rfl: 0x0000000000010202 cr2: 0x0000000000000088

Logical CPU: 1
Error Code: 0x00000004
Trap Number: 14

Binary Images:
0x109f3c000 - 0x109f3eff7 +bs (0) <30D2D212-5E1C-3798-97DA-063174EC988F> /usr/local/bin/bs
0x109f4d000 - 0x109faaff7 +libbitcoin-server.0.dylib (0) /usr/local/lib/libbitcoin-server.0.dylib
0x10a055000 - 0x10a0aafff +libbitcoin-node.0.dylib (0) <2775E455-BFBF-30D1-8DF8-6ED1D592BF78> /usr/local/lib/libbitcoin-node.0.dylib
0x10a16b000 - 0x10a1befff +libbitcoin-blockchain.0.dylib (0) /usr/local/lib/libbitcoin-blockchain.0.dylib
0x10a25b000 - 0x10a35cff7 +libbitcoin.0.dylib (0) <3F65AC27-B7B8-3AD3-A33F-E5868A6CC9BA> /usr/local/lib/libbitcoin.0.dylib
0x10a4e6000 - 0x10a4eaff3 +libboost_chrono-mt.dylib (0) <5FA88621-2010-37A1-AE89-A4C59E7F5F9A> /usr/local/lib/libboost_chrono-mt.dylib
0x10a4f2000 - 0x10a4f6ff7 +libboost_date_time-mt.dylib (0) <508538DC-FD6D-3919-B31B-CC50B9A751CA> /usr/local/lib/libboost_date_time-mt.dylib
0x10a50b000 - 0x10a519fff +libboost_filesystem.dylib (0) <3A29899F-CFB1-3E88-9E40-ADD9BC06E5C1> /usr/local/lib/libboost_filesystem.dylib
0x10a52f000 - 0x10a588fff +libboost_locale-mt.dylib (0) <3C8291F2-F385-33DD-8B47-F7852CCC4465> /usr/local/lib/libboost_locale-mt.dylib
0x10a5f7000 - 0x10a632fff +libboost_program_options-mt.dylib (0) <03E5429F-0719-34AE-84BB-349550E432F2> /usr/local/lib/libboost_program_options-mt.dylib
0x10a691000 - 0x10a722fff +libboost_regex-mt.dylib (0) <7F06F452-D477-338C-9607-A3148216202D> /usr/local/lib/libboost_regex-mt.dylib
0x10a7a9000 - 0x10a7abff3 +libboost_system.dylib (0) /usr/local/lib/libboost_system.dylib
0x10a7b1000 - 0x10a7bfffb +libboost_thread-mt.dylib (0) /usr/local/lib/libboost_thread-mt.dylib
0x10a7dc000 - 0x10a7ecff7 +libsecp256k1.0.dylib (0) /usr/local/lib/libsecp256k1.0.dylib
0x10a7f7000 - 0x10a7f9fff +libczmq++.0.dylib (0) /usr/local/lib/libczmq++.0.dylib
0x10a802000 - 0x10a850fff +libczmq.3.dylib (0) <298FC5A0-C2C5-3E46-8E16-E1D4946B8D28> /usr/local/lib/libczmq.3.dylib
0x10a874000 - 0x10a8c1fff +libzmq.5.dylib (0) <4FC79D31-A55B-3EB7-8D24-FFB2E677D3C4> /usr/local/lib/libzmq.5.dylib
0x10a919000 - 0x10a964fe7 +libsodium.13.dylib (0) /usr/local/lib/libsodium.13.dylib
0x10a990000 - 0x10a992ff3 +libboost_system-mt.dylib (0) <1F805F25-9640-3288-AEC9-23884EDCF5B4> /usr/local/lib/libboost_system-mt.dylib
0x7fff6e343000 - 0x7fff6e379837 dyld (353.2.1) <72A99D0F-0B56-3938-ABC5-67A0F33757C4> /usr/lib/dyld
0x7fff846cb000 - 0x7fff846e1ff7 libsystem_asl.dylib (267) /usr/lib/system/libsystem_asl.dylib
0x7fff84815000 - 0x7fff84816ff3 libSystem.B.dylib (1213) /usr/lib/libSystem.B.dylib
0x7fff848a5000 - 0x7fff848a5ff7 liblaunch.dylib (559.30.1) /usr/lib/system/liblaunch.dylib
0x7fff848a6000 - 0x7fff848a8fff libquarantine.dylib (76.20.1) <7AF90041-2768-378A-925A-D83161863642> /usr/lib/system/libquarantine.dylib
0x7fff84c4d000 - 0x7fff84c4fff7 libsystem_coreservices.dylib (9) <41B7C578-5A53-31C8-A96F-C73E030B0938> /usr/lib/system/libsystem_coreservices.dylib
0x7fff84edb000 - 0x7fff850d546f libobjc.A.dylib (647) <759E155D-BC42-3D4E-869B-6F57D477177C> /usr/lib/libobjc.A.dylib
0x7fff85103000 - 0x7fff85109fff libsystem_trace.dylib (72.20.1) <840F5301-B55A-3078-90B9-FEFFD6CD741A> /usr/lib/system/libsystem_trace.dylib
0x7fff855a1000 - 0x7fff855b2ff3 libsystem_coretls.dylib (35.30.2) <0F7BAD0C-FC28-3E4B-8D21-06B426599043> /usr/lib/system/libsystem_coretls.dylib
0x7fff855ba000 - 0x7fff855befff libcache.dylib (69) <45E9A2E7-99C4-36B2-BEE3-0C4E11614AD1> /usr/lib/system/libcache.dylib
0x7fff85b2c000 - 0x7fff85b35fff libsystem_pthread.dylib (105.10.1) <3103AA7F-3BAE-3673-9649-47FFD7E15C97> /usr/lib/system/libsystem_pthread.dylib
0x7fff86942000 - 0x7fff8695ffff libsystem_kernel.dylib (2782.30.5) <101D28C0-AF07-3B81-87BE-CA27ADED33AB> /usr/lib/system/libsystem_kernel.dylib
0x7fff86cc1000 - 0x7fff86cecfff libc++abi.dylib (125) <88A22A0F-87C6-3002-BFBA-AC0F2808B8B9> /usr/lib/libc++abi.dylib
0x7fff87009000 - 0x7fff8700aff7 libsystem_blocks.dylib (65) <9615D10A-FCA7-3BE4-AA1A-1B195DACE1A1> /usr/lib/system/libsystem_blocks.dylib
0x7fff87083000 - 0x7fff87084ffb libremovefile.dylib (35) <3485B5F4-6CE8-3C62-8DFD-8736ED6E8531> /usr/lib/system/libremovefile.dylib
0x7fff87c0f000 - 0x7fff87c1afff libcommonCrypto.dylib (60061.30.1) /usr/lib/system/libcommonCrypto.dylib
0x7fff87c4d000 - 0x7fff87c75fff libsystem_info.dylib (459.20.1) /usr/lib/system/libsystem_info.dylib
0x7fff8960d000 - 0x7fff8960ffff libsystem_sandbox.dylib (358.20.5) <3F5E973F-C702-31AC-97BC-05F5C195683C> /usr/lib/system/libsystem_sandbox.dylib
0x7fff89902000 - 0x7fff89956fff libc++.1.dylib (120) <1B9530FD-989B-3174-BB1C-BDC159501710> /usr/lib/libc++.1.dylib
0x7fff89971000 - 0x7fff899a1fff libsystem_m.dylib (3086.1) <1E12AB45-6D96-36D0-A226-F24D9FB0D9D6> /usr/lib/system/libsystem_m.dylib
0x7fff89a9e000 - 0x7fff89a9ffff libsystem_secinit.dylib (18) <581DAD0F-6B63-3A48-B63B-917AF799ABAA> /usr/lib/system/libsystem_secinit.dylib
0x7fff8a4dd000 - 0x7fff8a4e0ff7 libdyld.dylib (353.2.1) <78E8F33D-0C86-3DB6-A93D-B67A25BA3522> /usr/lib/system/libdyld.dylib
0x7fff8a940000 - 0x7fff8a942fff libsystem_configuration.dylib (699.30.1) /usr/lib/system/libsystem_configuration.dylib
0x7fff8aa28000 - 0x7fff8aa2fff7 libcompiler_rt.dylib (35) /usr/lib/system/libcompiler_rt.dylib
0x7fff8aa31000 - 0x7fff8aabdff7 libsystem_c.dylib (1044.10.1) <86FBED7A-F2C8-3591-AD6F-486DD57E6B6A> /usr/lib/system/libsystem_c.dylib
0x7fff8b7e7000 - 0x7fff8b811ff7 libdispatch.dylib (442.1.4) <502CF32B-669B-3709-8862-08188225E4F0> /usr/lib/system/libdispatch.dylib
0x7fff8b8cf000 - 0x7fff8b8d7fff libsystem_platform.dylib (63) <64E34079-D712-3D66-9CE2-418624A5C040> /usr/lib/system/libsystem_platform.dylib
0x7fff8edd0000 - 0x7fff8edd8ffb libcopyfile.dylib (118.1.2) <0C68D3A6-ACDD-3EF3-991A-CC82C32AB836> /usr/lib/system/libcopyfile.dylib
0x7fff8f9ae000 - 0x7fff8f9aeff7 libkeymgr.dylib (28) <77845842-DE70-3CC5-BD01-C3D14227CED5> /usr/lib/system/libkeymgr.dylib
0x7fff91e92000 - 0x7fff91ecafff libsystem_network.dylib (412.20.3) <6105C134-6722-3C0A-A4CE-5E1261E2E1CC> /usr/lib/system/libsystem_network.dylib
0x7fff91f3b000 - 0x7fff91f43fff libsystem_dnssd.dylib (576.30.4) <4EA2DEC3-77EE-3941-A703-DE6DC2056B15> /usr/lib/system/libsystem_dnssd.dylib
0x7fff91f44000 - 0x7fff92036ff7 libiconv.2.dylib (42) <2A06D02F-8B76-3864-8D96-64EF5B40BC6C> /usr/lib/libiconv.2.dylib
0x7fff920d6000 - 0x7fff920d6ff7 libunc.dylib (29) <5676F7EA-C1DF-329F-B006-D2C3022B7D70> /usr/lib/system/libunc.dylib
0x7fff921df000 - 0x7fff921fbff7 libsystem_malloc.dylib (53.30.1) /usr/lib/system/libsystem_malloc.dylib
0x7fff92d81000 - 0x7fff92da9fff libxpc.dylib (559.30.1) <80D68997-17B9-32B6-A5FA-A218216415E5> /usr/lib/system/libxpc.dylib
0x7fff92daa000 - 0x7fff92daffff libsystem_stats.dylib (163.30.2) <48A9387D-5C63-3E79-979C-F675552F6FC9> /usr/lib/system/libsystem_stats.dylib
0x7fff92dbb000 - 0x7fff92dc0ff7 libunwind.dylib (35.3) /usr/lib/system/libunwind.dylib
0x7fff92ea6000 - 0x7fff92eecff7 libauto.dylib (186) /usr/lib/libauto.dylib
0x7fff93e51000 - 0x7fff93e57ff7 libsystem_networkextension.dylib (167.30.1) <3E99FF35-DCBB-3A4C-8853-F1F39A792D29> /usr/lib/system/libsystem_networkextension.dylib
0x7fff93ebe000 - 0x7fff93ebffff libDiagnosticMessagesClient.dylib (100) <2EE8E436-5CDC-34C5-9959-5BA218D507FB> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff93ec0000 - 0x7fff93f39fe7 libcorecrypto.dylib (233.30.1) <5779FFA0-4D9A-3AD4-B7F2-618227621DC8> /usr/lib/system/libcorecrypto.dylib
0x7fff93f3b000 - 0x7fff93f40ff7 libmacho.dylib (862) <126CA2ED-DE91-308F-8881-B9DAEC3C63B6> /usr/lib/system/libmacho.dylib
0x7fff94014000 - 0x7fff9401dff7 libsystem_notify.dylib (133.1.1) <61147800-F320-3DAA-850C-BADF33855F29> /usr/lib/system/libsystem_notify.dylib

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 7
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 2997550
thread_create: 0
thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=87.5M resident=84.8M(97%) swapped_out_or_unallocated=2764K(3%)
Writable regions: Total=258.2M written=188.8M(73%) resident=189.1M(73%) swapped_out=0K(0%) unallocated=69.1M(27%)

REGION TYPE VIRTUAL
=========== =======
Kernel Alloc Once 4K
MALLOC 239.2M
MALLOC (admin) 16K
MALLOC_LARGE (reserved) 284K reserved VM address space (unallocated)
STACK GUARD 56.1M
Stack 18.2M
VM_ALLOCATE 12K
__DATA 1024K
__LINKEDIT 76.4M
__TEXT 11.1M
mapped file 86.5G
shared memory 4K
=========== =======
TOTAL 86.9G
TOTAL, minus reserved VM space 86.9G

dyld: Symbol not found: _fchmodat
Referenced from: /usr/local/bin/bs
Expected in: /usr/lib/libSystem.B.dylib
in /usr/local/bin/bs
Trace/BPT trap: 5 

Works in 10.10 (Yosemite).

$ otool -L bs
bs:
    /opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

Based on Airbitz deployment experience:
https://github.com/libbitcoin/libbitcoin-server/wiki/Airbitz-Server-Configuration#useful-tidbits

I am telling the server to connect to a specific peer via a command in the config file:

peer = obelisk-sol2.airbitz.co:8334

Sadly, this doesn't work. The libbitcoin-server program spews a bunch of lines like these:

22:39:42.759410 INFO [protocol]: Manual connection established: obelisk-sol2.airbitz.co:8334
22:39:42.760244 ERROR [protocol]: Channel stopped (manual): Service stopped
22:39:42.760368 ERROR [poller]: Send problem: Service stopped

The Satoshi peer (Bitcoin Core Daemon version v0.10.0.0-g047a898) spews lines like these to its ~/.bitcoin/debug.log:

2015-02-27 22:44:01 receive version message: /libbitcoin:2.8.0/: version 60000, blocks=17457, us=10.0.0.1:8333, peer=67546

This isn't a connectivity problem, since the two servers obviously see each other. They just aren't able to negotiate a handshake. I can't get any success the other way around either (adding the libbitcoin peer to the Satoshi ~/.bitcoin/bitcoin.conf).

As you know, we run a bunch of libbitcoin servers at different versions of the software. We have a server compiled with the latest master (June 01), and this one does not fetch incoming transactions into the mempool - the mempool stays completely empty, even though the blockchain is fully synced. We have another server compiled on May 11, and this one runs fine. At some point between those two dates, the ability to populate the mempool was lost.

We are using the vanilla build scripts, plus --disable-consensus or whatever that flag is called.

The recent changes make a world of difference. I was syncing past block 330,000 with no problem, albeit slowly (2-3 blocks per min) and so today I started over from block zero for testing. All ran smoothly until block 92603, where I began seeing this warning:

16:58:34.615363 WARNING [node]: Failed to store transaction in mempool 6337eafeafedf2df5599431b4b4b7946a7baa2e12e52f7cb2b99dbf4cedd1ee9: Spent input not found
16:58:34.681852 WARNING [node]: Failed to store transaction in mempool 543db7c66df7074736309dcff65aea21a4751013736a7d45993350b40b602b4e: Spent input not found
16:58:36.713706 WARNING [node]: Failed to store transaction in mempool fff32b926f2869fccb67633b860a5177c22aac7ed57f88272cd9f7363991b852: Spent input not found
16:58:36.905803 WARNING [node]: Failed to store transaction in mempool 3160cd35e198c5bc3ff8d6d964e93dec908e8db2b3b9dfa443830bbbc3ff4144: Spent input not found
16:58:37.283318 WARNING [node]: Failed to store transaction in mempool aa3ac61bde0ff5b2f7630624be316c1a7fa4ff83e39ff284500a89ec94ffdd03: Spent input not found
16:58:38.207241 WARNING [node]: Failed to store transaction in mempool a4abb98660068f65260568755d2540f7a817d0046bf49febe56680a2e749b983: Spent input not found

(i am running entire libbitcoin suite built from latest git master)

It can be difficult to determine which exact version of the server is running. If it was downloaded using git, then git describe can be used. However, if a stable version is used (think Obelisk ATM), there's no way to find out for non-priviledged users that don't have access to sources.

This complicates bug reporting, including cases where a "git HEAD" archive was used for deployment.

I suggest either a -V command-line argument be introduced, or the info be printed at lauch time. Minimal versioning info is available in configure.ac ATM.

The proactor pattern, which is employed by libbitcoin, achieves its performance advantages by queuing most actions. Currently there is insufficient management of the queue to prevent denial of service due to excessive activity. In some cases libbitcoin inadvertently generates this activity itself. This can be resolved by shutting down the service and restarting. If the message queue is very large it may take some time for a normal shutdown to complete. Note that a hard shutdown has a slight chance of corrupting the blockchain database and should be avoided.

My local server on Ubuntu 14.04.1 managed to finish syncing today, but I noticed it won't actually respond to requests from BX. I have all the default port settings and set my BX config to use url = tcp://localhost:9091. However, output of netstat doesn't show any of the relevant ports.

I looked through the worker.cpp code and didn't find anything obvious. Is this supposed to be working right now? I realize it's still very much in development, but the server doesn't seem very useful without its service endpoint.

I can provide logs and other information about my setup if needed.

libbitcoin/libbitcoin-system#6

As of f34b4ac the following BX calls have compatibility breaks against libbitcoin-server in relation to the obelisk/version1 protocol.

• fetch-balance
• fetch-history
• fetch-stealth
• watch-address

The cause of the fetch-balance break is not yet known.

The fetch-history break is the result of an intended API change may be resolved by subsequent commit bb657e3 which splits out the new feature into a distinct call. Testing is underway.

The fetch-stealth and watch-address breaks are known to be the result of protocol changes. It may be OK to break fetch-stealth as it has been experimental and use is likely limited. However we may want to consider compatibility for watch-address. The cause of the breaks is a sentinel in the protocol to allow for the detection of a stealth prefix vs. a bitcoin address.

I tried turning on the publisher, but there's an immediate error during launch.

~ $ bitcoin_server
Publisher failed to start: Invalid argument
terminate called without an active exception
[1]   <pid> abort      bitcoin_server

When attempting to compile on Ubuntu 14.04 using GCC 4.8.2, I'm receiving an error that says, "src/service/compat.hpp:10:50: error: 'incoming_message' does not name a type." With this type not being defined, compiling fails.

Based on build:

$ ./install.sh --disable-shared --prefix=/Users/Eric/bs-prefix-test --build-dir=/Users/Eric/bx-build-test --with-consensus --build-boost --enable-testnet

this error:

...
20:00:36.371123 INFO [poller] Block #54493 00000000ab5f6ca3268326facf05287e6e5cf240f0f7b403c2d3c0a42947ae28
libc++abi.dylib: terminating with uncaught exception of type std::invalid_argument: prevout_script

recurs on each subsequent startup.

The exception fires from here:
https://github.com/libbitcoin/libbitcoin-consensus/blob/master/src/consensus/consensus.cpp#L183

Which indicates that the prevout_script pointer is null here:
https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/validate_transaction.cpp#L320

Which arises from the serialization here:
https://github.com/libbitcoin/libbitcoin/blob/master/src/script.cpp#L2189

Which implies the script is empty. The value derives from a std::vector of the script bytes. On some systems std::vector.data() returns a null/zero value when the vector is empty (the particular behavior is unspecified).

A question is why this script is comping back empty, but this should not get far enough to bring down the process with an exception, so should be guarded earlier and logged as a warning.

Build: bfda2fa

ubuntu 14.04 LTS
Memory: 7.7 GiB
Processor: Intel Core 2 Quad CPU Q9550 @ 2.83GHz x 4
OS type: 64-bit
Disk: 483.7 GB

Example segment of console output:

10:48:34.085396 INFO [poller]: Block #195248 00000000000003526f1bcd8d2f677dccb7af194ba5b43c0be222fe0061470963
10:49:58.382702 ERROR [protocol]: Problem receiving addresses: Service stopped
10:49:58.382757 ERROR [session]: inventory: Service stopped
10:49:58.382797 ERROR [session]: get_blocks: Service stopped
10:49:58.382834 ERROR: recv_transaction: Service stopped
10:49:58.382809 ERROR [poller]: Received bad inventory: Service stopped
10:49:58.382872 ERROR [poller]: Received bad block: Service stopped
10:49:58.382960 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:49:58.383040 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:49:58.866901 INFO [protocol]: Connected to 46.105.210.130:8333 (8 connections)
10:50:59.928554 ERROR [protocol]: Problem receiving addresses: Service stopped
10:50:59.928624 ERROR [session]: inventory: Service stopped
10:50:59.928638 ERROR [poller]: Received bad inventory: Service stopped
10:50:59.928668 ERROR [session]: get_blocks: Service stopped
10:50:59.928722 ERROR: recv_transaction: Service stopped
10:50:59.928758 ERROR [poller]: Received bad block: Service stopped
10:50:59.928798 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:50:59.928874 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:51:00.409888 INFO [protocol]: Connected to 46.105.210.31:8333 (8 connections)
10:52:23.371445 ERROR [protocol]: Problem receiving addresses: Service stopped
10:52:23.371519 ERROR [session]: inventory: Service stopped
10:52:23.371535 ERROR [poller]: Received bad inventory: Service stopped
10:52:23.371561 ERROR [session]: get_blocks: Service stopped
10:52:23.371629 ERROR: recv_transaction: Service stopped
10:52:23.371664 ERROR [poller]: Received bad block: Service stopped
10:52:23.371713 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:52:23.371793 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:52:24.014197 INFO [protocol]: Connected to 46.105.210.178:8333 (8 connections)
10:55:12.848391 ERROR [protocol]: Problem receiving addresses: Service stopped
10:55:12.848465 ERROR [session]: inventory: Service stopped
10:55:12.848505 ERROR [session]: get_blocks: Service stopped
10:55:12.848543 ERROR: recv_transaction: Service stopped
10:55:12.848580 ERROR [poller]: Received bad block: Service stopped
10:55:12.848620 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:55:12.848562 ERROR [poller]: Received bad inventory: Service stopped
10:55:12.848701 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:55:13.330194 INFO [protocol]: Connected to 46.105.210.6:8333 (8 connections)
10:56:32.224324 ERROR [protocol]: Problem receiving addresses: Service stopped
10:56:32.224380 ERROR [session]: inventory: Service stopped
10:56:32.224448 ERROR [session]: get_blocks: Service stopped
10:56:32.224487 ERROR: recv_transaction: Service stopped
10:56:32.224525 ERROR [poller]: Received bad block: Service stopped
10:56:32.224566 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:56:32.224645 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:56:32.224410 ERROR [poller]: Received bad inventory: Service stopped
10:56:32.225716 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:56:32.714619 INFO [protocol]: Connected to 46.105.210.164:8333 (8 connections)
10:57:41.965183 ERROR [protocol]: Problem receiving addresses: Service stopped
10:57:41.965247 ERROR [session]: inventory: Service stopped
10:57:41.965258 ERROR [poller]: Received bad inventory: Service stopped
10:57:41.965290 ERROR [session]: get_blocks: Service stopped
10:57:41.965331 ERROR: recv_transaction: Service stopped
10:57:41.965366 ERROR [poller]: Received bad block: Service stopped
10:57:41.965404 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:57:41.965480 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:57:42.447400 INFO [protocol]: Connected to 46.105.210.79:8333 (8 connections)
10:58:36.033657 ERROR [protocol]: Problem receiving addresses: Service stopped
10:58:36.033742 ERROR [session]: inventory: Service stopped
10:58:36.033758 ERROR [poller]: Received bad inventory: Service stopped
10:58:36.033790 ERROR [session]: get_blocks: Service stopped
10:58:36.033851 ERROR: recv_transaction: Service stopped
10:58:36.033896 ERROR [poller]: Received bad block: Service stopped
10:58:36.033940 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:58:36.525383 INFO [protocol]: Connected to 46.105.210.179:8333 (8 connections)
10:58:45.656621 ERROR [protocol]: Problem receiving addresses: Service stopped
10:58:45.656690 ERROR [session]: inventory: Service stopped
10:58:45.656693 ERROR [poller]: Received bad inventory: Service stopped
10:58:45.656736 ERROR [session]: get_blocks: Service stopped
10:58:45.656785 ERROR: recv_transaction: Service stopped
10:58:45.656831 ERROR [poller]: Received bad block: Service stopped
10:58:45.656877 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:58:45.791791 ERROR [protocol]: Problem receiving addresses: Service stopped
10:58:45.791854 ERROR [session]: inventory: Service stopped
10:58:45.791895 ERROR [session]: get_blocks: Service stopped
10:58:45.791934 ERROR: recv_transaction: Service stopped
10:58:45.791987 ERROR [poller]: Received bad block: Service stopped
10:58:45.792030 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:58:45.791972 ERROR [poller]: Received bad inventory: Service stopped
10:58:46.289165 INFO [protocol]: Connected to 46.105.210.106:8333 (7 connections)
10:58:46.297316 INFO [protocol]: Connected to 46.105.210.188:8333 (8 connections)
10:58:49.195249 ERROR [protocol]: Problem receiving addresses: Service stopped
10:58:49.195320 ERROR [session]: inventory: Service stopped
10:58:49.195328 ERROR [poller]: Received bad inventory: Service stopped
10:58:49.195422 ERROR [session]: get_blocks: Service stopped
10:58:49.195474 ERROR: recv_transaction: Service stopped
10:58:49.195515 ERROR [poller]: Received bad block: Service stopped
10:58:49.195561 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:58:49.731509 INFO [protocol]: Connected to 46.105.210.114:8333 (8 connections)
10:59:24.476443 ERROR [protocol]: Problem receiving addresses: Service stopped
10:59:24.476530 ERROR [poller]: Received bad inventory: Service stopped
10:59:24.476572 ERROR [session]: inventory: Service stopped
10:59:24.476615 ERROR [session]: get_blocks: Service stopped
10:59:24.476662 ERROR: recv_transaction: Service stopped
10:59:24.476703 ERROR [poller]: Received bad block: Service stopped
10:59:24.476751 ERROR [protocol]: Channel stopped (outbound): Service stopped
10:59:24.476839 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
10:59:24.969011 INFO [protocol]: Connected to 46.105.210.243:8333 (8 connections)
11:01:03.362250 ERROR [protocol]: Problem receiving addresses: Service stopped
11:01:03.362319 ERROR [session]: inventory: Service stopped
11:01:03.362331 ERROR [poller]: Received bad inventory: Service stopped
11:01:03.362544 ERROR [session]: get_blocks: Service stopped
11:01:03.362597 ERROR: recv_transaction: Service stopped
11:01:03.362638 ERROR [poller]: Received bad block: Service stopped
11:01:03.362685 ERROR [protocol]: Channel stopped (outbound): Service stopped
11:01:03.362817 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
11:01:03.913625 INFO [protocol]: Connected to 46.105.210.226:8333 (8 connections)
11:01:25.049277 ERROR [protocol]: Problem receiving addresses: Service stopped
11:01:25.049342 ERROR [session]: inventory: Service stopped
11:01:25.049383 ERROR [session]: get_blocks: Service stopped
11:01:25.049421 ERROR: recv_transaction: Service stopped
11:01:25.049405 ERROR [poller]: Received bad inventory: Service stopped
11:01:25.049480 ERROR [poller]: Received bad block: Service stopped
11:01:25.049608 ERROR [protocol]: Channel stopped (outbound): Service stopped
11:01:25.693246 INFO [protocol]: Connected to 46.105.210.147:8333 (8 connections)
11:02:08.370741 ERROR [protocol]: Problem receiving addresses: Service stopped
11:02:08.370813 ERROR [session]: inventory: Service stopped
11:02:08.370854 ERROR [session]: get_blocks: Service stopped
11:02:08.370893 ERROR: recv_transaction: Service stopped
11:02:08.370870 ERROR [poller]: Received bad inventory: Service stopped
11:02:08.370932 ERROR [poller]: Received bad block: Service stopped
11:02:08.371054 ERROR [protocol]: Channel stopped (outbound): Service stopped
11:02:08.371134 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
11:02:08.981958 INFO [protocol]: Connected to 46.105.210.219:8333 (8 connections)
11:02:11.621358 ERROR [protocol]: Problem receiving addresses: Service stopped
11:02:11.621428 ERROR [session]: inventory: Service stopped
11:02:11.621438 ERROR [poller]: Received bad inventory: Service stopped
11:02:11.621473 ERROR [session]: get_blocks: Service stopped
11:02:11.621513 ERROR: recv_transaction: Service stopped
11:02:11.621556 ERROR [poller]: Received bad block: Service stopped
11:02:11.621602 ERROR [protocol]: Channel stopped (outbound): Service stopped
11:02:11.621684 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
11:02:12.167813 INFO [protocol]: Connected to 46.105.210.89:8333 (8 connections)
11:02:56.761575 ERROR [protocol]: Problem receiving addresses: Service stopped
11:02:56.761640 ERROR [session]: inventory: Service stopped
11:02:56.761713 ERROR [poller]: Received bad inventory: Service stopped
11:02:56.761792 ERROR [session]: get_blocks: Service stopped
11:02:56.761858 ERROR: recv_transaction: Service stopped
11:02:56.761895 ERROR [poller]: Received bad block: Service stopped
11:02:56.761935 ERROR [protocol]: Channel stopped (outbound): Service stopped
11:02:56.762013 ERROR [poller]: Send problem: Service stopped
<large number of the above error>
11:02:57.361149 INFO [protocol]: Connected to 46.105.210.43:8333 (8 connections)
11:03:12.535751 INFO [poller]: Block #195260 00000000000003a4af96bbc6affcd987fba28ffa09ca0de0db9b5abfad1edcc0

Contact me for logs.

Build: bfda2fa

  Model Name:   iMac
  Model Identifier: iMac8,1
  Processor Name:   Intel Core 2 Duo
  Processor Speed:  2.4 GHz
  Number of Processors: 1
  Total Number of Cores:    2
  L2 Cache: 6 MB
  Memory:   4 GB
  Bus Speed:    1.07 GHz
  Boot ROM Version: IM81.00C1.B00
  SMC Version (system): 1.29f1

Sync was completed through:

INFO [poller]: Block #158199 0000000000000e716e6593afab209ed612f8a74b7fc014adee935ecaca46018e

Console message:

Bus error: 10

Contact me for logs.

Otherwise the caller must always follow-up with requests for all transactions, which is very wasteful.

At random points during normal operation, the blockchain will stop syncing. During one particular instance, error.log showed:

15:09:10.960411 WARNING [validate]: Input script consensus validation failed
15:09:10.960601 WARNING [validate]: Validate input 46581355dad0b59ac492d9df490292d5311776aac0c5164c8e14fe32012ce43a:1 failed
15:09:10.960745 WARNING [poller]: Storing block 000000000000000009e531e76f8a1c35829b663eef46158ed3505c134a5d6319: Validation of inputs failed

Rebuilding the libbitcoin-blockchain part --without-consensus seemed to clear this up, and the chain started syncing again:

16:59:52.077616 INFO [poller]: Block #351252 000000000000000009e531e76f8a1c35829b663eef46158ed3505c134a5d6319

I've had this happen twice now, both at different points in the blockchain.

Things were running smoothly, downloading blocks until height 212579, when the server segfaulted. It now segfaults on every startup.

#0  libbitcoin::satoshi_load<unsigned char*, false> (first=0x0, last=last@entry=0x0, packet=...)
    at /usr/local/include/bitcoin/bitcoin/impl/serialize/block.ipp:144
#1  0x00007ffff67e47f4 in deserialize<unsigned char*> (first=<optimized out>) at src/database/block_database.cpp:49
#2  libbitcoin::chain::block_result::header (this=this@entry=0x7ffff0a51f20) at src/database/block_database.cpp:66
#3  0x00007ffff6804a1c in libbitcoin::chain::validate_block_impl::fetch_block (this=<optimized out>, fetch_height=<optimized out>)
    at src/implementation/validate_block_impl.cpp:52
#4  0x00007ffff68048b8 in libbitcoin::chain::validate_block_impl::previous_block_bits (this=<optimized out>)
    at src/implementation/validate_block_impl.cpp:58
#5  0x00007ffff67d8e2a in libbitcoin::chain::validate_block::work_required (this=0x7ffff0a52080) at src/validate_block.cpp:276
#6  0x00007ffff67d913b in libbitcoin::chain::validate_block::accept_block (this=this@entry=0x7ffff0a52080)
    at src/validate_block.cpp:210
#7  0x00007ffff68040f8 in libbitcoin::chain::organizer_impl::verify (this=<optimized out>, fork_index=212580, 
    orphan_chain=std::vector of length 1, capacity 1 = {...}, orphan_index=0) at src/implementation/organizer_impl.cpp:51
#8  0x00007ffff67cc0d1 in libbitcoin::chain::organizer::replace_chain (this=this@entry=0x7fffffffd600, fork_index=212580, 
    orphan_chain=std::vector of length 1, capacity 1 = {...}) at src/organizer.cpp:77
#9  0x00007ffff67ccae0 in libbitcoin::chain::organizer::process (this=this@entry=0x7fffffffd600, 
    process_block=std::shared_ptr (count 5, weak 0) 0x7ff86c004600) at src/organizer.cpp:64
#10 0x00007ffff67cce4e in libbitcoin::chain::organizer::start (this=this@entry=0x7fffffffd600) at src/organizer.cpp:50
#11 0x00007ffff67fff04 in libbitcoin::chain::blockchain_impl::do_store(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>) (this=0x7fffffffd260, block=..., handle_store=...)
    at src/implementation/blockchain_impl.cpp:154
#12 0x00007ffff68036c3 in operator()<libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>&, void> (__object=0x7fffffffd260, this=0x7ffff0a52600) at /usr/include/c++/4.9/functional:569
#13 __call<void, 0ul, 1ul, 2ul> (__args=<optimized out>, this=0x7ffff0a52600) at /usr/include/c++/4.9/functional:1264
#14 operator()<, void> (this=0x7ffff0a52600) at /usr/include/c++/4.9/functional:1323
#15 __call<void> (__args=<optimized out>, this=0x7ffff0a52600) at /usr/include/c++/4.9/functional:1264
#16 operator()<, void> (this=0x7ffff0a52600) at /usr/include/c++/4.9/functional:1323
#17 asio_handler_invoke<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (function=...)
    at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69
#18 invoke<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (context=..., 
    function=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#19 boost::asio::detail::strand_service::dispatch<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> >(boost::asio::detail::strand_service::strand_impl*&, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>&) (this=<optimized out>, impl=@0x7ffff0a52760: 0x6253d0, 
    handler=...) at /usr/local/include/boost/asio/detail/impl/strand_service.hpp:62
#20 0x00007ffff6803a11 in dispatch<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (
    handler=<unknown type in /usr/local/lib/libbitcoin-blockchain.so.0, CU 0x401151, DIE 0x4854f8>, this=0x7ffff0a52758)
    at /usr/local/include/boost/asio/strand.hpp:158
#21 operator() (this=0x7ffff0a52758) at /usr/local/include/boost/asio/detail/wrapped_handler.hpp:76
#22 operator() (this=0x7ffff0a526b0) at /usr/local/include/boost/asio/detail/wrapped_handler.hpp:191
#23 asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (function=...) at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69
#24 invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (context=..., function=...)
    at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#25 asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (this_handler=<optimized out>, 
    function=...) at /usr/local/include/boost/asio/detail/wrapped_handler.hpp:275
#26 invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (
    context=..., function=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#27 boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (owner=0x60d750, base=base@entry=0x7ff86c0045f0)
---Type <return> to continue, or q <return> to quit---
    at /usr/local/include/boost/asio/detail/completion_handler.hpp:68
#28 0x00007ffff6803d19 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> >&) (
    this=0x624d60, impl=@0x7ffff0a52998: 0x6253d0, handler=...) at /usr/local/include/boost/asio/detail/impl/strand_service.hpp:88
#29 0x00007ffff6803f79 in dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (handler=<unknown type in /usr/local/lib/libbitcoin-blockchain.so.0, CU 0x401151, DIE 0x48721f>, 
    this=0x7ffff0a52990) at /usr/local/include/boost/asio/strand.hpp:158
#30 asio_handler_invoke<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running> (this_handler=0x7ffff0a52990, function=...)
    at /usr/local/include/boost/asio/detail/wrapped_handler.hpp:232
#31 invoke<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running> > (context=..., function=...)
    at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#32 boost::asio::detail::completion_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (owner=<optimized out>, 
    base=<optimized out>) at /usr/local/include/boost/asio/detail/completion_handler.hpp:68
#33 0x00007ffff78cef58 in complete (bytes_transferred=<optimized out>, ec=..., owner=..., this=<optimized out>)
    at /usr/local/include/boost/asio/detail/task_io_service_operation.hpp:38
#34 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>)
    at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:384
#35 run (ec=..., this=<optimized out>) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:153
#36 run (this=<optimized out>) at /usr/local/include/boost/asio/impl/io_service.ipp:59
#37 operator() (__closure=<optimized out>, __closure=<optimized out>) at src/utility/threadpool.cpp:56
#38 _M_invoke<> (this=<optimized out>) at /usr/include/c++/4.9/functional:1700
#39 operator() (this=<optimized out>) at /usr/include/c++/4.9/functional:1688
#40 std::thread::_Impl<std::_Bind_simple<libbitcoin::threadpool::spawn_once(libbitcoin::thread_priority)::<lambda()>()> >::_M_run(void) (this=<optimized out>) at /usr/include/c++/4.9/thread:115
#41 0x00007ffff7327e30 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#42 0x00007ffff5e046aa in start_thread (arg=0x7ffff0a53700) at pthread_create.c:333
#43 0x00007ffff6d93eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

It appears that block 4033 contains something libbitcoin disagrees with, so the sync simply stops:

Storing block 00000000db623a1752143f2f805c4527573570d9b4ca0a3cfe371e703ac429aa: Proof of work does not match bits field

Windows 7, x64, libbitcoin-consensus, OpenSSL, 2.83GHz, 8GB RAM, mechanical drive.

Notice that in the last image it goes completely off-the-rails.

The CPU is stuck at 100%, and the server is no longer storing blocks. Restarting the process and rebooting the box do nothing to fix this.

Here is a stack trace of the offending thread (all other threads are asleep in poll functions):

#0  libbitcoin::chain::htdb_record_list_item<std::array<unsigned char, 20ul> >::next_index (this=this@entry=0x7fffebffdbe0)
    at ./include/bitcoin/blockchain/impl/htdb_record_list_item.ipp:101
#1  0x00007ffff72688b8 in libbitcoin::chain::htdb_record<std::array<unsigned char, 20ul> >::get (this=0x7fffffffe1c0, key=...)
    at ./include/bitcoin/blockchain/impl/htdb_record.ipp:62
#2  0x00007ffff7268be2 in libbitcoin::chain::multimap_records<std::array<unsigned char, 20ul> >::add_row(std::array<unsigned char, 20ul> const&, std::function<void (unsigned char*)>) (this=this@entry=0x7fffffffe210, key=..., write=...)
    at ./include/bitcoin/blockchain/impl/multimap_records.ipp:47
#3  0x00007ffff7268151 in libbitcoin::chain::history_database::add_output (
    this=this@entry=0x7fffffffe170, key=..., outpoint=..., 
    output_height=output_height@entry=337460, value=value@entry=19296)
    at src/database/history_database.cpp:80
#4  0x00007ffff72628c9 in libbitcoin::chain::db_interface::push_outputs (
    this=this@entry=0x7fffffffe028, tx_hash=..., 
    block_height=block_height@entry=337460, outputs=...)
    at src/db_interface.cpp:211
#5  0x00007ffff7262ba5 in libbitcoin::chain::db_interface::push (
    this=0x7fffffffe028, block=...) at src/db_interface.cpp:119
#6  0x00007ffff7260623 in libbitcoin::chain::organizer::replace_chain (
    this=this@entry=0x6c48c8, fork_index=337459, orphan_chain=...)
    at src/organizer.cpp:222
#7  0x00007ffff7260869 in libbitcoin::chain::organizer::process (
    this=this@entry=0x6c48c8, process_block=...) at src/organizer.cpp:171
#8  0x00007ffff72609d7 in libbitcoin::chain::organizer::start (this=0x6c48c8)
    at src/organizer.cpp:159
#9  0x00007ffff726e074 in libbitcoin::chain::blockchain_impl::do_store(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>) (this=0x7fffffffdfd0, block=..., handle_store=...)
    at src/impl/blockchain_impl.cpp:142
#10 0x00007ffff7270ff0 in operator()<libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>&, void> (
    __object=0x7fffffffdfd0, this=0x7fffebffe120)
    at /usr/include/c++/4.8/functional:601
#11 __call<void, 0ul, 1ul, 2ul> (__args=<optimized out>, this=0x7fffebffe120)
    at /usr/include/c++/4.8/functional:1296
#12 operator()<, void> (this=0x7fffebffe120)
    at /usr/include/c++/4.8/functional:1355
#13 __call<void> (__args=<optimized out>, this=0x7fffebffe120)
    at /usr/include/c++/4.8/functional:1296
#14 operator()<, void> (this=0x7fffebffe120)
    at /usr/include/c++/4.8/functional:1355
#15 asio_handler_invoke<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (function=...)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#16 boost_asio_handler_invoke_helpers::invoke<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >(std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> &, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> &) (function=..., context=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 0x00007ffff7277550 in boost::asio::detail::strand_service::dispatch<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> >(boost::asio::detail::strand_service::strand_impl*&, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>&) (this=0x6b7990, 
---Type <return> to continue, or q <return> to quit---
    impl=@0x7fffebffe570: 0x6b8000, handler=...)
    at /usr/include/boost/asio/detail/impl/strand_service.hpp:62
#18 0x00007ffff7277927 in dispatch<std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>&> (handler=..., this=0x7fffebffe568)
    at /usr/include/boost/asio/strand.hpp:155
#19 operator() (this=0x7fffebffe568)
    at /usr/include/boost/asio/detail/wrapped_handler.hpp:76
#20 operator() (this=0x7fffebffe4c0)
    at /usr/include/boost/asio/detail/wrapped_handler.hpp:191
#21 asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (function=...)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#22 invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#23 asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > (
    this_handler=<optimized out>, function=...)
    at /usr/include/boost/asio/detail/wrapped_handler.hpp:275
#24 invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> >, boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, li---Type <return> to continue, or q <return> to quit---
bbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (
    context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#25 boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (owner=<optimized out>, 
    base=base@entry=0x7fea246979a0)
    at /usr/include/boost/asio/detail/completion_handler.hpp:68
#26 0x00007ffff7277d43 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()> >&) (this=0x6b7990, 
    impl=@0x7fffebffe918: 0x6b8000, handler=...)
    at /usr/include/boost/asio/detail/impl/strand_service.hpp:88
#27 0x00007ffff7277fcc in dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()> > > (
    handler=<unknown type in /usr/local/lib/libbitcoin-blockchain.so.0, CU 0x72e9ef, DIE 0x7fbd0d>, this=0x7fffebffe910)
    at /usr/include/boost/asio/strand.hpp:155
#28 asio_handler_invoke<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running> (
    this_handler=0x7fffebffe910, function=...)
    at /usr/include/boost/asio/detail/wrapped_handler.hpp:232
#29 invoke<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_typ---Type <return> to continue, or q <return> to quit---
e, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running>, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(const libbitcoin::block_type&, std::function<void(const std::error_code&, libbitcoin::block_info)>)>(libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void(const std::error_code&, libbitcoin::block_info)>)>()>, boost::asio::detail::is_continuation_if_running> > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#30 boost::asio::detail::completion_handler<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Bind<std::_Mem_fn<void (libbitcoin::chain::blockchain_impl::*)(libbitcoin::block_type const&, std::function<void (std::error_code const&, libbitcoin::block_info)>)> (libbitcoin::chain::blockchain_impl*, libbitcoin::block_type, std::function<void (std::error_code const&, libbitcoin::block_info)>)> ()>, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (
    owner=<optimized out>, base=<optimized out>)
    at /usr/include/boost/asio/detail/completion_handler.hpp:68
#31 0x00007ffff6fb794f in complete (bytes_transferred=<optimized out>, ec=..., 
    owner=..., this=0x7fea45b28340)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#32 do_run_one (ec=..., this_thread=..., lock=..., this=0x6a2290)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#33 boost::asio::detail::task_io_service::run (this=0x6a2290, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#34 0x00007ffff6fe5487 in run (this=0x7fffffffdd80)
    at /usr/include/boost/asio/impl/io_service.ipp:59
#35 operator() (__closure=<optimized out>) at src/utility/threadpool.cpp:52
#36 _M_invoke<> (this=<optimized out>) at /usr/include/c++/4.8/functional:1732
#37 operator() (this=<optimized out>) at /usr/include/c++/4.8/functional:1720
#38 std::thread::_Impl<std::_Bind_simple<libbitcoin::threadpool::spawn_once()::__lambda0()> >::_M_run(void) (this=<optimized out>)
    at /usr/include/c++/4.8/thread:115
#39 0x00007ffff60d9bf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#40 0x00007ffff6cc3182 in start_thread (arg=0x7fffebfff700)
    at pthread_create.c:312
#41 0x00007ffff5b4700d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I can organize ssh access to the box if this would help.

#68 (comment)