How to compact the database? (struct.error: 'H' format requires 0 <= number <= 65535) about docker-electrumx HOT 6 OPEN

You can get a shell inside the container with the following command:
docker exec -it electrumx /bin/sh
There you can run electrumx_compact_history
You might need to replace "electrumx" in the docker exec with the name of your docker container or it's id.

from docker-electrumx.

Thanks @Mave95, I think I'm making progress.

How can I stop the electrumx_server command without terminating the container?

Here's what I've tried so far. With the container running, I ran docker container ls which produced these results:

$ docker container ls
CONTAINER ID        IMAGE                  COMMAND             CREATED             STATUS              PORTS                                                      NAMES
fda9a23863c2        lukechilds/electrumx   "init"              3 minutes ago       Up 3 minutes        8000/tcp, 50001/tcp, 50004/tcp, 0.0.0.0:50002->50002/tcp   busy_bhabha

The CONTAINER ID works for opening up a shell.

$ docker exec -it fda9a23863c2 /bin/sh
/data # 

Once there, I can see that there's a python3 process electrumx_server running with PID 1:

/data # ps
PID   USER     TIME  COMMAND
    1 root      4:33 python3 /electrumx/electrumx_server
   53 root      0:00 /bin/sh
   58 root      0:00 ps

I tried to run the /electrumx/electrumx_compact_history command, but it failed due to DB lock:

/data # /electrumx/electrumx_compact_history
INFO:root:Starting history compaction...
INFO:electrumx.server.db.DB:switching current directory to /data
INFO:electrumx.server.db.DB:using leveldb for DB backend
Traceback (most recent call last):
  File "/electrumx/electrumx_compact_history", line 73, in main
    loop.run_until_complete(compact_history())
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/electrumx/electrumx_compact_history", line 51, in compact_history
    await db.open_for_compacting()
  File "/electrumx/electrumx/server/db.py", line 147, in open_for_compacting
    await self._open_dbs(True, True)
  File "/electrumx/electrumx/server/db.py", line 123, in _open_dbs
    self.utxo_db = self.db_class('utxo', for_sync)
  File "/electrumx/electrumx/server/storage.py", line 31, in __init__
    self.open(name, create=self.is_new)
  File "/electrumx/electrumx/server/storage.py", line 83, in open
    max_open_files=mof)
  File "plyvel/_plyvel.pyx", line 247, in plyvel._plyvel.DB.__init__
  File "plyvel/_plyvel.pyx", line 88, in plyvel._plyvel.raise_for_status
plyvel._plyvel.IOError: b'IO error: lock utxo/LOCK: Resource temporarily unavailable'
CRITICAL:root:History compaction terminated abnormally

I tried to shut down the server according to the documentation for terminating ElectrumX by using the electrumx_rpc command but that failed:

/data # electrumx_rpc stop
cannot connect - is ElectrumX catching up, not running, or is 8000 the wrong RPC port?

So instead, I tried sending the INT signal using the kill command:

/data # kill -s INT 1

This worked in shutting down the ElectrumX server, but it also shut down the container and booted me right out of the shell.

$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

I'm not sure what to do next. How can I stop the electrumx_server command without terminating the container?

from docker-electrumx.

You have to modify the entrypoint of your container to start it without running electrumx_server. Here's a tutorial that explains how: https://thorsten-hans.com/how-to-run-commands-in-stopped-docker-containers
Note that you might have to add your container configuration (volumes, env vars, ...) to the docker run command.

from docker-electrumx.

@jimbojw take a look here
kyuupichan/electrumx#185 (comment)

from docker-electrumx.

@sancoder Yes, thanks, I believe that's eventually what I did, to my recollection.

If we're gonna live with this compact_history stuff it'd be great to have such a pause for service merged into the electrumx_server script.

I 100% agree with this comment of yours. Having to muck about with containers is not a good UX. If we have to run periodic maintenance like compacting the database, it should at least be easy to perform. This doesn't have to be a manual process.

from docker-electrumx.

Here's a one-liner I constructed to perform history compact operations. It should work on any system that has docker installed and that has r/w access to the electrum data set.

docker run --rm --env COIN=BitcoinSegwit --env DB_DIRECTORY=/data -v "${PWD}/electrumx:/data" --entrypoint=/usr/local/bin/electrumx_compact_history lukechilds/electrumx:v1.16.0

Regarding manual maintenance tasks, this specific issue (struct.error: 'H' format) occurs often enough for me, that I believe electrumx itself should gracefully detect and recover from these sort of issues.

from docker-electrumx.