Comments (27)
JustinTArthur
commented on August 25, 2024
2
Aw man, what did we do! Thanks, @curious0101
from electrumx.
SomberNight
commented on August 25, 2024
1
I can reproduce btw, am getting a memory leak too.
I have tried going back to before adding the cache commits.
Tested with 029ffb3, but still there is a leak.
Result: 5714MB memory and growing, few megabytes every second.
"uptime": "01h 11m 00s",
For me, it takes ~4 hours to get to 4 GB, so it is growing a lot slower here. This unfortunately makes testing harder :/
from electrumx.
SomberNight
commented on August 25, 2024
1
So there are three logical explanations AFAICT why this started happening only recently
- some new commit (in this repo) introduced it
- a dependency released a new version that introduced it
- sessions (clients) changed their behaviour (an attack...)
from electrumx.
github12101
commented on August 25, 2024
Server memory usage from Grafana, this zigzag is created everytime I restart electrumX, it just creeps up back again.
from electrumx.
Jamiejoin
commented on August 25, 2024
I encountered the same problem. My server has 128G memory, but electrumx can be used up. I am python3.7.7，Operating system ubuntu 16.04,There is a memory leak in electrumx,Below is my configuration file。
# default /etc/electrumx.conf for systemd
# REQUIRED
DB_DIRECTORY = /db
# Bitcoin Node RPC Credentials
DAEMON_URL = http://test:[email protected]:8332/
COIN = Bitcoin
# See http://electrumx.readthedocs.io/en/latest/environment.html for
# information about other configuration settings you probably want to consider.
DB_ENGINE=rocksdb
SSL_CERTFILE=/etc/electrumx/server.crt
SSL_KEYFILE=/etc/electrumx/server.key
# Listen on all interfaces:
SERVICES=tcp://:50001,ssl://:50002,ws://:50003,rpc://:8000
MAX_SEND=800000000
MAX_SESSIONS=1000
LOG_LEVE=debug
BANDWIDTH_UNIT_COST= 9000000
REQUEST_TIMEOUT=600
SESSION_TIMEOUT=2400
CACHE_MB=1200
@JustinTArthur Sorry to disturb you, have you encountered this problem? Hope to get some suggestions, thank you very much.
from electrumx.
github12101
commented on August 25, 2024
I am on default db, you are on rocksdb, so it's not it. We also have different python versions, so it's not it (probably). It could be one of pip packages. How can I list pip packages electrumX depends on?
This is full list of my pip packages in user directory:
pip list
pip list
Package Version
------------------- -------------
acme 1.9.0
aiohttp 3.7.0
aiorpcX 0.18.4
appdirs 1.4.4
asn1crypto 1.4.0
async-timeout 3.0.1
attrs 20.2.0
bernhard 0.2.6
bottle 0.12.18
btrfs 11
cassandra-driver 3.24.0
certbot 1.9.0
certbot-apache 1.9.0
certifi 2020.6.20
cffi 1.14.3
chardet 3.0.4
click 7.1.2
ConfigArgParse 1.2.3
configobj 5.0.6
CouchDB 1.2
cryptography 3.1.1
cycler 0.10.0
distlib 0.3.1
distro 1.5.0
docker 4.3.1
docker-pycreds 0.4.0
ecdsa 0.16.0
elasticsearch 7.9.1
electrumX 1.15.0
entrypoints 0.3
fail2ban 0.10.2
filelock 3.0.12
future 0.18.2
geomet 0.2.1.post1
Glances 3.1.5
httplib2 0.18.1
idna 2.10
ifaddr 0.1.7
importlib-metadata 2.0.0
influxdb 5.3.0
iotop 0.6
jeepney 0.4.3
josepy 1.4.0
kafka-python 2.0.2
keyring 21.4.0
keyrings.alt 4.0.0
kiwisolver 1.2.0
leveldb 0.201
Mako 1.1.3
MarkupSafe 1.1.1
matplotlib 3.3.2
mock 4.0.2
msgpack 1.0.0
multidict 5.0.0
netifaces 0.10.9
noop 1.0
numpy 1.19.2
olefile 0.46
packaging 20.4
paho-mqtt 1.5.1
parsedatetime 2.6
pbkdf2 1.3
pbr 5.5.1
pika 1.1.0
Pillow 8.0.1
pip 20.2.4
pip-review 1.1.0
ply 3.11
plyvel 1.3.0
potsdb 1.0.3
prometheus-client 0.8.0
protobuf 3.13.0
psutil 5.7.2
py-cpuinfo 7.0.0
py3nvml 0.2.6
pyasn1 0.4.8
pycairo 1.16.2
pycparser 2.20
pycrypto 2.6.1
pycryptodomex 3.9.8
pycurl 7.43.0.6
pyenchant 3.1.1
pygal 2.4.0
PyGObject 3.30.4
pygtkspellcheck 4.0.5
pyinotify 0.9.6
pylru 1.2.0
pymdstat 0.4.2
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyRFC3339 1.1
PySimpleSOAP 1.16.2
pySMART.smartx 0.3.10
pysmi 0.3.4
pysnmp 4.4.12
pystache 0.5.4
python-apt 1.8.4.1
python-augeas 1.1.0
python-dateutil 2.8.1
python-debian 0.1.38
python-debianbts 3.0.2
pytz 2020.1
pyxdg 0.27
pyzmq 19.0.2
reportbug 7.5.3-deb10u1
requests 2.24.0
requests-toolbelt 0.9.1
scandir 1.10.0
scipy 1.5.3
SecretStorage 3.1.2
setuptools 50.3.2
six 1.15.0
sparklines 0.4.2
statsd 3.3.0
systemd-python 234
typing-extensions 3.7.4.3
ujson 3.2.0
urllib3 1.25.11
uvloop 0.14.0
virtualenv 20.0.35
websocket-client 0.57.0
websockets 8.1
wheel 0.35.1
wifi 0.3.8
xmltodict 0.12.0
yarl 1.6.2
zeroconf 0.28.6
zipp 3.3.2
zope.component 4.6.2
zope.deferredimport 4.3.1
zope.deprecation 4.4.0
zope.event 4.5.0
zope.hookable 5.0.1
zope.interface 5.1.2
zope.proxy 4.3.5
from electrumx.
SomberNight
commented on August 25, 2024
Is this very recent? Could it be because of #24 ?
Try going back a few commits, e.g. checkout 10f0faa
from electrumx.
github12101
commented on August 25, 2024
Thanks for your reply,
Yes, I think this is recent, I did git fetch not long time ago and it started. But I also updated my pip packages (I was debugging something else and updated all pip packages in the process).
Now, I installed rocksdb and I'm resynchronizing whole electrumX again, it will take some hours. Will report how that performs. If there is no change, I will revert git back to the one you just mentioned.
from electrumx.
SomberNight
commented on August 25, 2024
Note that there has been also a new aiohttp version released.
aiohttp 3.6.2 was latest for ~1 year, and now there has been several new versions released in a burst in the past few days.
from electrumx.
github12101
commented on August 25, 2024
Acknowledged. I just downgraded to 3.6.2. Continuing with rocksdb sync. Once finished, I will go several tests with different aiohttp versions.
from electrumx.
Jamiejoin
commented on August 25, 2024
Acknowledged. I just downgraded to 3.6.2. Continuing with rocksdb sync. Once finished, I will go several tests with different aiohttp versions.
Is the downgrade problem fixed?
from electrumx.
github12101
commented on August 25, 2024
Don't know yet, electrumX is synchronizing from the scratch after I changed database engine.
2020-10-26 02:48:03,120:INFO:DB:sync time: 09h 42m 48s ETA: 07h 44m 54s
2020-10-26 02:48:34,085:INFO:BlockProcessor:our height: 549,489 daemon: 654,311 UTXOs 467MB hist 175MB
from electrumx.
github12101
commented on August 25, 2024
So my electrumX is fully synced and working, with uvloop and rocksdb. Result: 5714MB memory and growing, few megabytes every second.
Log:
2020-10-26 12:23:24,854:INFO:electrumx:ElectrumX server starting
2020-10-26 12:23:24,857:INFO:electrumx:logging level: INFO
2020-10-26 12:23:24,858:INFO:Controller:Python version: 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]
2020-10-26 12:23:24,862:INFO:Controller:software version: ElectrumX 1.15.0
2020-10-26 12:23:24,862:INFO:Controller:aiorpcX version: 0.18.4
2020-10-26 12:23:24,862:INFO:Controller:supported protocol versions: 1.4-1.4.2
2020-10-26 12:23:24,862:INFO:Controller:event loop policy: <uvloop.EventLoopPolicy object at 0x7ff14d788be0>
2020-10-26 12:23:24,862:INFO:Controller:reorg limit is 200 blocks
2020-10-26 12:23:24,862:INFO:Daemon:daemon #1 at localhost:8332/ (current)
2020-10-26 12:23:24,863:INFO:DB:switching current directory to /home/nv01/.electrumx
2020-10-26 12:23:24,869:INFO:DB:using rocksdb for DB backend
2020-10-26 12:23:25,446:INFO:DB:opened UTXO DB (for sync: True)
Pip packages:
aiohttp 3.6.2
aiorpcX 0.18.4
electrumx_rpc getinfo
$ electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 654352,
"db height": 654352,
"db_flush_count": 843,
"groups": 6367,
"history cache": "14,978 lookups 9,381 hits 1,000 entries",
"merkle cache": "1,417 lookups 1,157 hits 260 entries",
"peers": {
"bad": 0,
"good": 82,
"never": 1,
"stale": 0,
"total": 83
},
"pid": 2646078,
"request counts": {
"blockchain.block.header": 680,
"blockchain.block.headers": 46,
"blockchain.estimatefee": 12113,
"blockchain.headers.subscribe": 559,
"blockchain.relayfee": 134,
"blockchain.scripthash.get_history": 684,
"blockchain.scripthash.subscribe": 14294,
"blockchain.transaction.get": 1527,
"blockchain.transaction.get_merkle": 1460,
"getinfo": 2,
"mempool.get_fee_histogram": 404,
"server.add_peer": 3,
"server.banner": 137,
"server.donation_address": 134,
"server.features": 26,
"server.peers.subscribe": 160,
"server.ping": 673,
"server.version": 16711
},
"request total": 49747,
"sessions": {
"count": 135,
"count with subs": 8,
"errors": 2,
"logged": 0,
"pending requests": 2,
"subs": 494
},
"tx hashes cache": "1,460 lookups 1,102 hits 282 entries",
"txs sent": 0,
"uptime": "01h 11m 00s",
"version": "ElectrumX 1.15.0"
}
from electrumx.
github12101
commented on August 25, 2024
Now I just did:
git checkout 10f0faa
pip3 install .[rocksdb,ujson]
git reset --hard
HEAD is now at 10f0faa Merge pull request #64 from SomberNight/202010_document_db
Will report in a moment.
from electrumx.
github12101
commented on August 25, 2024
With #10f0faa I have 1664MB already after 8 minutes.
electrumx_rpc getinfo
$ electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 654359,
"db height": 654359,
"db_flush_count": 850,
"groups": 2615,
"history cache": "176 lookups 31 hits 145 entries",
"merkle cache": "0 lookups 0 hits 0 entries",
"peers": {
"bad": 8,
"good": 65,
"never": 35,
"stale": 0,
"total": 108
},
"pid": 2761397,
"request counts": {
"blockchain.block.header": 42,
"blockchain.block.headers": 2,
"blockchain.estimatefee": 384,
"blockchain.headers.subscribe": 57,
"blockchain.relayfee": 6,
"blockchain.scripthash.subscribe": 176,
"getinfo": 2,
"mempool.get_fee_histogram": 13,
"server.banner": 6,
"server.donation_address": 6,
"server.features": 3,
"server.peers.subscribe": 9,
"server.ping": 9,
"server.version": 3028
},
"request total": 3743,
"sessions": {
"count": 31,
"count with subs": 1,
"errors": 3,
"logged": 0,
"pending requests": 4,
"subs": 45
},
"tx hashes cache": "0 lookups 0 hits 0 entries",
"txs sent": 0,
"uptime": "08m 41s",
"version": "ElectrumX 1.15.0"
2008MB by the time I finished this post and edited this sentence :) I have plenty of connections and public facing domain, maybe that's why it's faster here.
from electrumx.
github12101
commented on August 25, 2024
I noticed earlier, that overnight, when I left it running, it did not grow. I closed 50002 port in the router, I had no incoming connections and no memory leak. I could still run my Electrum wallet with electrumX as usual.
@xiaoqge So, solution could be to not accept any incoming connections (they are called sessions in electrumx_rpc) on advertised port and just use electrumX for yourself, while we wait for a patch. Bug must be somewhere in this sessions logic.
from electrumx.
github12101
commented on August 25, 2024
Running with port closed.
htop: RES 1238M
I can't quite remember what was normal memory load before, but I think it was less than that. I watched it grow from 600 up to +1200 MB now (2 hours 9 minutes). And it's basically just idling, serving no one, just keeping sync with blockchain and talking to other servers (no clients at all). I will leave it running for a while to see if it croses 1.5GB threshold, that would mean leak is still here, as I doubt I ever crossed 1.5GB with peers, in normal use, before.
electrumx_rpc getinfo
$ electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 654368,
"db height": 654368,
"db_flush_count": 859,
"groups": 4,
"history cache": "0 lookups 0 hits 0 entries",
"merkle cache": "0 lookups 0 hits 0 entries",
"peers": {
"bad": 2,
"good": 80,
"never": 0,
"stale": 0,
"total": 82
},
"pid": 2789806,
"request counts": {
"blockchain.block.header": 1,
"blockchain.headers.subscribe": 1,
"getinfo": 2,
"server.features": 1,
"server.peers.subscribe": 1,
"server.version": 1
},
"request total": 7,
"sessions": {
"count": 1,
"count with subs": 0,
"errors": 0,
"logged": 0,
"pending requests": 1,
"subs": 0
},
"tx hashes cache": "0 lookups 0 hits 0 entries",
"txs sent": 0,
"uptime": "02h 07m 17s",
"version": "ElectrumX 1.15.0"
from electrumx.
github12101
commented on August 25, 2024
1694MB and growing, but very slowly, that's why I didn't noticed it when I stopped electrumX once it synchronized last night. No sessions. I am using old commit 10f0faa
electrumx_rpc getinfo
$ electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 654381,
"db height": 654381,
"db_flush_count": 872,
"groups": 7,
"history cache": "0 lookups 0 hits 0 entries",
"merkle cache": "0 lookups 0 hits 0 entries",
"peers": {
"bad": 2,
"good": 82,
"never": 2,
"stale": 2,
"total": 88
},
"pid": 2789806,
"request counts": {
"blockchain.block.header": 2,
"blockchain.headers.subscribe": 2,
"getinfo": 4,
"server.features": 2,
"server.peers.subscribe": 2,
"server.version": 2
},
"request total": 14,
"sessions": {
"count": 1,
"count with subs": 0,
"errors": 0,
"logged": 0,
"pending requests": 1,
"subs": 0
},
"tx hashes cache": "0 lookups 0 hits 0 entries",
"txs sent": 0,
"uptime": "05h 27m 07s",
"version": "ElectrumX 1.15.0"
from electrumx.
SomberNight
commented on August 25, 2024
emzy on IRC said he is not experiencing a memory leak.
He is running commit 76dca02 with Python 3.6.9.
Mem usage atm RES 1.447g (but high uptime).
pip freeze
~/electrumx$ pip freeze
aiohttp==3.5.4
aiorpcX==0.18.3
asn1crypto==0.24.0
async-timeout==3.0.1
attrs==17.4.0
Automat==0.6.0
chardet==3.0.4
configobj==5.0.6
constantly==15.1.0
cryptography==2.1.4
distro-info===0.18ubuntu0.18.04.1
electrumX==1.12.0
fail2ban==0.10.2
hyperlink==17.3.1
idna==2.6
idna-ssl==1.1.0
incremental==16.10.1
iotop==0.6
keyring==10.6.0
keyrings.alt==3.0
leveldb==0.1
multidict==4.5.2
netifaces==0.10.4
plyvel==1.0.5
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycrypto==2.6.1
pygobject==3.26.1
pyinotify==0.9.6
pylru==1.2.0
pyOpenSSL==17.5.0
python-apt==1.6.5+ubuntu0.3
python-debian==0.1.32
pyxdg==0.25
PyYAML==3.12
SecretStorage==2.3.1
service-identity==16.0.0
six==1.11.0
systemd-python==234
Twisted==17.9.0
typing-extensions==3.7.2
websockets==7.0
yarl==1.3.0
zope.interface==4.3.2
from electrumx.
smmalis37
commented on August 25, 2024
I don't believe I'm currently experiencing a leak. After 12 hrs uptime I'm at RES 533.1m. Private server so I'm the only client. I'm running 835aa49 built through https://aur.archlinux.org/packages/electrumx/ on Python 3.8.6.
pip freeze
$ pip freeze
aiohttp==3.6.2
aiorpcX==0.18.4
appdirs==1.4.4
asn1crypto==1.4.0
async-timeout==3.0.1
attrs==20.2.0
Automat==20.2.0
Beaker==1.11.0
CacheControl==0.12.6
cffi==1.14.3
chardet==3.0.4
click==7.1.2
colorama==0.4.4
constantly==15.1.0
contextlib2==0.6.0.post1
cryptography==3.1.1
deluge==2.0.4.dev38
discord.py==1.5.0
distlib==0.3.1
distro==1.5.0
electrumX==1.15.0
filelock==3.0.12
GeoIP==1.3.2
gunicorn==20.0.4
html5lib==1.1
hyperlink==20.0.1
idna==2.10
incremental==17.5.0
Mako==1.1.3
MarkupSafe==1.1.1
msgpack==1.0.0
multidict==4.7.6
ordered-set==4.0.2
packaging==20.4
pep517==0.8.2
pikaur==1.6.12
Pillow==7.2.0
ply==3.11
plyvel==1.2.0
progress==1.5
pyalpm==0.9.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
PyHamcrest==1.9.0
pylru==1.2.0
pyOpenSSL==19.1.0
pyparsing==2.4.7
python-libtorrent==1.2.10
python-rapidjson==0.9.3
pyxdg==0.26
Reflector==2020.9.2.20.13.49
rencode==1.0.6
requests==2.24.0
resolvelib==0.4.0
retrying==1.3.3
service-identity==18.1.0
setproctitle==1.1.10
six==1.15.0
toml==0.10.1
Twisted==20.3.0
typing-extensions==3.7.4.3
urllib3==1.25.10
virtualenv==20.0.32
webencodings==0.5.1
websockets==8.1
yarl==1.5.1
zope.interface==5.1.2
from electrumx.
github12101
commented on August 25, 2024
I reverted to 1.15.0 835aa49 and:
aiohttp==3.5.4
aiorpcX==0.18.3
Seems to fix the problem! Been up 1h 21min, 81 sessions, res 1362 MB.
from electrumx.
github12101
commented on August 25, 2024
You are welcome! :)
Just upgraded aiohttp from 3.5.4 to 3.7.1.
Result after 1 hour:
1147 MB at 73 sessions.
So it doesn't seem to be aiohttp.
Edit: I am doing further tests with different versions, will report here back once I am done.
from electrumx.
Jamiejoin
commented on August 25, 2024
@JustinTArthur @smmalis37 @SomberNight I tried to upgrade to python3.9, but the problem is still leaked. If you just run electrumx synchronization, there will be no leakage, but using websocket to quickly and continuously connect and query tx_hash memory will increase, and it will not be released. If you need a problem To reproduce, only need to establish a websocket to quickly query transactions. My server memory is 128G, and it only takes 7 hours to run to 100G memory，
Below is my test code：
import asyncio
import websockets
import socket
import logging
from jsonrpcclient.clients.websockets_client import WebSocketsClient
class Conn():
def __init__(self):
self.method = ""
self.params = ""
self.flag = False
self.loop = asyncio.get_event_loop()
async def electrumx(self):
async with websockets.connect('ws://127.0.0.1:50003',max_size=1000000000000000) as ws:
res = None
if self.flag == True:
res = await WebSocketsClient(ws).request(self.method, self.params, self.flag)
else:
res = await WebSocketsClient(ws).request(self.method, self.params)
ws.close()
return res.data.__dict__
def get_balance(self, addr):
#Address to scripthash
hashaddr = address_to_scripthash(addr)
self.method = "blockchain.scripthash.get_balance"
self.params = hashaddr
self.flag = False
while True:
try:
rep = self.loop.run_until_complete(self.electrumx())
break
except socket.error as e:
logging.info("electrumx socket error,do reconnect, %s", e)
sleep(1)
except Exception as e:
logging.info("electrumx other error occur, error is %s", e)
sleep(1)
return rep['result']['confirmed']
from electrumx.
SomberNight
commented on August 25, 2024
Naturally it was I who introduced it, in 48d53da
Underhanded python... heh
from electrumx.
JustinTArthur
commented on August 25, 2024
Well, nice job tracking it down anyhow
from electrumx.
github12101
commented on August 25, 2024
Thanks for the fix 👍😁🎉
from electrumx.
Jamiejoin
commented on August 25, 2024
Great, I updated the installation and it has been fixed, thank you very much
from electrumx.
Related Issues (20)
- I get an error: electrumx.server.block_processor.ChainError: UTXO xxx / 0 not found in "h" table HOT 2
- Build fails on Fedora 39 (worked on 38) - plyvel doesn't build HOT 1
- new release, new git tag HOT 1
- error when installing on Debian Stable: This environment is externally managed HOT 2
- Please use a LRU library available in major distributions HOT 1
- OverflowError while running python electrumx_server in latest version, previuos veron is working. HOT 1
- version not the latest HOT 2
- ssl ports randomly close HOT 5
- ElectrumX Deutsche eMark (DEM) doesn't work HOT 1
- DESERIALIZER
- PermissionError: [Errno 13] Permission denied HOT 4
- electrumx crashes on debian 12 HOT 3
- Issue Report: Inaccurate Data Retrieval for Specific Addresses
- server start normaly but after 4 or 5 mit it show this error. plaese provide correct answers. HOT 4
- please solve this. HOT 3
- electrumx_rpc returns unknown method for every command HOT 2
- Are High Disk Writes Expected? HOT 3
- Abnormal termination with hashXs[hashX].remove(tx_hash) KeyError
- Electrumx won't start because UTXO not found in "h" table, but it's a confirmed tx HOT 1
- I have a PIVX frok that does not work! HOT 13
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from electrumx.