DB::flush_count overflow about electrumx HOT 15 OPEN

see kyuupichan/electrumx#185

you should run the electrumx_compact_history script, while the server is stopped, using the same environment variables as otherwise

from electrumx.

Note that the database is being re-architectured in #80.
When that is merged, this issue will be categorically fixed.

from electrumx.

electrumx_compact_history seemed to work for me. Problem is now solved.

Thank you!

from electrumx.

I also got this issue and posted in kyuupichan/electrumx#185

I am trying to run the compaction script as follows in this link, although it did not work for me.
kyuupichan/electrumx#185 (comment) (contains the errors I had when running script)

I am running Bitcoin Segwit rather than BCASH/BSV unlike most others running the compaction script mentioned Kyuupichan's repo.

Do I have to modify this compaction script to work properly with Bitcoin Segwit ElectrumX? (Spesmilo ElectrumX 1.15)

from electrumx.

@99ytrewq911 no modification is needed. Make sure you run the electrumx_compact_history script with the same ENV variables as you normally run the main script with.
If it still does not work, please post your actual traceback.

from electrumx.

@99ytrewq911 no modification is needed. Make sure you run the electrumx_compact_history script with the same ENV variables as you normally run the main script with.
If it still does not work, please post your actual traceback.

@SomberNight

Hello, thank you for the response,

Here was my default syntax to execute and the results:

export $(cat /etc/electrumx.conf | xargs) && python3.7 ~/electrumcompact/compact_history.py
Traceback (most recent call last):
  File "/root/electrumcompact/compact_history.py", line 52, in <module>
    from server.env import Env
ModuleNotFoundError: No module named 'server.env'

I also attempted to manually set the variables mentioned in the script's comment section as follows, using a .env file. I used a print statement to verify that they were being set:

The new import section:

import os
import logging
import sys
import traceback
from os import environ

from dotenv import load_dotenv
load_dotenv()

COIN = os.getenv('COIN')

print(COIN)

DB_ENGINE= os.getenv('DB_ENGINE')

print(DB_ENGINE)

DB_DIRECTORY= os.getenv('DB_DIRECTORY')


print(DB_DIRECTORY)


from server.env import Env
from server.db import DB

And here was the output:

BitcoinSegwit
leveldb
/var/lib/electrumx
Traceback (most recent call last):
  File "/root/electrumcompact/compact_history.py", line 52, in <module>
    from server.env import Env
ModuleNotFoundError: No module named 'server.env'

I feel as if I am missing something quite important and obvious. I sourced the original script from this page:
kyuupichan/electrumx@2f26e81
I don't by chance require the other files in that commit, do I?

Thank you for your time

from electrumx.

I feel as if I am missing something quite important and obvious. I sourced the original script from this page:
kyuupichan/electrumx@2f26e81

Ah! That script is part of this repo -- it is a living thing like all files in git: you should use the one corresponding to your other files, probably latest. Find the script at the top level folder, latest version atm:
https://github.com/spesmilo/electrumx/blob/011980616900d42dfec03170f1d8369fcfcd4e6b/electrumx_compact_history

from electrumx.

@SomberNight This was indeed the issue. I see the file now in the root of this repo. Thank you for your assistance!

from electrumx.

Is it possible to update the readme to warn about this error in the meantime, and possibly spit out some warning messages in the logs in advance?

We've been taken by surprise with this error as it's just occurred for the first time after more than a year of running the server peacefully without issue. The fixing script takes hours to run, and of course our backup server simultaneously went down with the same error, causing critical interruption to customers.

Putting some warnings in would potentially save other developers a lot of aggravation (it's too late for us lol)

from electrumx.

@SomberNight Is it still worth working on a fix / automatic workaround for this issue? Would this concept have any chance at all: Warn in the logs occasionally if flush count higher than 50k, if flush count >= 60k raise special exception that shuts down everything, launch compact history from within main run loop, restart everything from within main run loop. Should be fairly simple.

from electrumx.

The first part of that (logging warnings) sounds simple and still useful.
The second part might not be so simple. If it can be done without complex/invasive changes, then ok.

from electrumx.

You are right, properly restarting from within Python is not trivial. Also shell solutions I could think of all had problems. So I suggest this extended documentation so that users can get up and running again a bit faster: #212

from electrumx.

I encountered the same issue today, after running it for a few months.
I also think that a warning could help the user. I found nothing in the docs about this issue.

• if (self.flush_count > 65535): warn the user with something like Running electrumx_compact_history should fix this issue.

from electrumx.

Autocompact history via shell script: https://github.com/ghoober/electrumx

from electrumx.

@SomberNight from what I gather, #101 is removing the need for DB compaction.

I would appreciate if you could proceed with this - either as part of the above PR or split into a smaller PR if it helps get it merged more quickly. The commit related to this seems simple enough, is there a specific reason it has stalled - any unexpected problem with it?

I was reminded of this issue as our compaction cronjob ran at an opportune time 😅

from electrumx.