Code Monkey home page Code Monkey logo

eth-portal's Introduction

Portal Network Tools

Join the chat at https://gitter.im/ethereum/eth-portal Build Status PyPI version Python versions Docs build

A collection of utilities related to Ethereum's Portal Network

Read more in the documentation on ReadTheDocs. View the change log.

Quickstart

pip install eth-portal

Developer Setup

If you would like to hack on eth-portal, please check out the Snake Charmers Tactical Manual for information on how we do:

  • Testing
  • Pull Requests
  • Code Style
  • Documentation

Development Environment Setup

You can set up your dev environment with:

git clone [email protected]:ethereum/eth-portal.git
cd eth-portal
virtualenv -p python3 venv
. venv/bin/activate
pip install -e .[dev]

Testing Setup

During development, you might like to have tests run on every file save.

Show flake8 errors on file change:

# Test flake8
when-changed -v -s -r -1 eth_portal/ tests/ -c "clear; flake8 eth_portal tests && echo 'flake8 success' || echo 'error'"

Run multi-process tests in one command, but without color:

# in the project root:
pytest --numprocesses=4 --looponfail --maxfail=1
# the same thing, succinctly:
pytest -n 4 -f --maxfail=1

Run in one thread, with color and desktop notifications:

cd venv
ptw --onfail "notify-send -t 5000 'Test failure ⚠⚠⚠⚠⚠' 'python 3 test on eth-portal failed'" ../tests ../eth_portal

Release setup

For Debian-like systems:

apt install pandoc

To release a new version:

make release bump=$$VERSION_PART_TO_BUMP$$

How to bumpversion

The version format for this repo is {major}.{minor}.{patch} for stable, and {major}.{minor}.{patch}-{stage}.{devnum} for unstable (stage can be alpha or beta).

To issue the next version in line, specify which part to bump, like make release bump=minor or make release bump=devnum. This is typically done from the master branch, except when releasing a beta (in which case the beta is released from master, and the previous stable branch is released from said branch).

If you are in a beta version, make release bump=stage will switch to a stable.

To issue an unstable version when the current version is stable, specify the new version explicitly, like make release bump="--new-version 4.0.0-alpha.1 devnum"

eth-portal's People

Contributors

carver avatar davesque avatar kclowes avatar njgheorghita avatar pipermerriam avatar cburgdorf avatar tmckenzie51 avatar

Stargazers

蔡豐澤 avatar 0xYYY avatar Shritesh Jamulkar avatar Chen Kai avatar Ognyan Genev avatar

Watchers

 avatar  avatar James Cloos avatar  avatar  avatar Kostas Georgiou avatar  avatar Scotty avatar  avatar

eth-portal's Issues

Release v0.1.0 (stable)

Wait until:

  • switch the launch mechanism to python -m eth_portal.bridge so that you can run it more easily after a pip install. Addressing in #23
  • merge new serialization format for bodies and receipts into specs. See ethereum/portal-network-specs#152
  • #27
  • Handle when trin drops the web3 filter ID, by recreating it as needed - in #28
  • wait for a release merge of trin that includes ethereum/trin#411

Then release eth-portal v0.1.0

Error with content key encoding (or maybe on trin side with decoding)

currently getting this from all JSON-RPC requests that eth-portal is making to offer content to the network

Propagate new history content with key, value: 0x01010042f62736317fc8a9da7646fd47859f3a32d0e60ea351c67413aadc2dc9a7c6e7 , 0x080000009bc70100bc0200000c0600002a080000b50b00003d0e00009d0f0000fc110000a81200005713000005140000b11400005d15000009160000b51600002b180000f51e00001421000085210000312200001d2500003d270000dd2900008b2a0000422b0000f92b0000712c00002e2d0000a42e0000a1320000f03300008a340000a836000020370000983700000e380000c53800003f390000f63900006f3a0000263b0000a03b0000193c0000d03c00004a3d0000c43d00003d3e0000b63e0000303f0000e43f00009b40000052410000cc410000e74700009e48000055490000ca490000434a0000f64a00000c4e0000655100009c53000048540000f3540000385c0000ee5c00009e650000156600008a6600001d670000d067000085680000fd6800007c6a00002f6b00008a700000a97200005e730000e2770000577800000a7900000b7d00000c8100008481000039820000738500002b950000a295000022970000d697000089980000fe980000139a0000829a0000f19a00005f9b0000139c0000c79c00005a9d0000509f000005a00000b9a000008da1000088a200003da30000f1a3000066a40000faa40000dec8000092c900006fca0000e4ca00005ccc00005dd0000035d40000e9d4000060d50000f4d50000a9d600005ed7000011d80000c7d8000067da0000ddda000090db00006adf000021e2000099e300004ce40000c4e900007aea0000f2ea00008eeb000041ec0000f5ec0000d1fd000064fe000017ff0000caff0000870001005e010100b50201002d030100cd04010006060100ba06010025070100da07010049080100f8080100700a01001c0b0100b70b0100660c0100970d01000c0e0100b80e0100270f01001e1001008b10010062110100791201002d130100031401007d140100531501008b8f0100c1910100749201004a930100ecc301009ac4010049c50100f5c501006bc601001ec7010002f9034c018210358085031fcdc1d7830f42409400000000500e2fece27a7600435d0c48d64e0c0085f307db82e09bcfcd8b09f495a41965e22b9dde94c71f49484986c9000533440000f902c0f85994c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2f842a0245d32d2da911ebe35956892a50b3686a5bbeb9c3bc6e8bd766cbdeada720e9ea0ed0d5e13647388a52e9a977db17b6771616da3808c7cfb736a6c4783bc3f4e23f8dd94cd8b09f495a41965e22b9dde94c71f49484986c9f8c6a0000000000000000000000000000000000000000000000000000000000000000ca00000000000000000000000000000000000000000000000000000000000000008a00000000000000000000000000000000000000000000000000000000000000006a000000000000000000000000000000000000000000000... (231226 more)
Sent history item to 389e685744010df0f11c response: {'jsonrpc': '2.0', 'id': 16, 'error': 'Error while processing portal_historyOffer: Error returned from chain history subnetwork: "Invalid Offer params: ValidationError { code: \\"Unable to decode content_key\\", message: None, params: {} }"'}

Option to insert arbitrary pre-calculated content values

Preferred input format: file named with the content key as {hex_encoded_key}.portalcontent and the value as a pre-encoded content value (a binary file).

API proposal:

(venv)$ python -m eth_portal.bridge ~/master_accumulator/*.portalcontent

This would launch the trin nodes just like eth_portal.bridge but instead of following the head of the chain, it would load in the data files and offer them out to the connected peers.

Unexpected error running eth_portal.bridge on python 3.10

What was wrong?

python -m eth_portal.bridge crashes with unexpected TypeError.

Code that produced the error

python -m eth_portal.bridge

Full error output

(eth-portal-venv) ogi:dev/ $ python -m eth_portal.bridge                      [12:12:25]
Launching trin with node ID c0a6c424ac using...
./trin --discovery-port 9000 --unsafe-private-key 0000000000000000000000000000000000000000000000000000000000000001 --web3-ipc-path /tmp/trin-jsonrpc-c0a6c424ac7157ae4083.ipc --kb 20000 --networks history --bootnodes default
Launching trin
Exiting trin with node ID c0a6c424ac...
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/eth_portal/bridge/__main__.py", line 4, in <module>
    launch_bridge()
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/eth_portal/bridge/run.py", line 34, in launch_bridge
    block_filter = w3.eth.filter("latest")
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/module.py", line 57, in caller
    result = w3.manager.request_blocking(method_str,
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/manager.py", line 197, in request_blocking
    response = self._make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/manager.py", line 150, in _make_request
    return request_func(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/formatting.py", line 94, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/gas_price_strategy.py", line 90, in middleware
    return make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/formatting.py", line 94, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/attrdict.py", line 33, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/formatting.py", line 94, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/formatting.py", line 94, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/formatting.py", line 94, in middleware
    response = make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/middleware/buffered_gas_estimate.py", line 40, in middleware
    return make_request(method, params)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/providers/websocket.py", line 144, in make_request
    return future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/providers/websocket.py", line 124, in coro_make_request
    async with self.conn as conn:
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/web3/providers/websocket.py", line 72, in __aenter__
    self.ws = await connect(
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 622, in __await_impl__
    transport, protocol = await self._create_connection()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1089, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1107, in _create_connection_transport
    protocol = protocol_factory()
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/websockets/legacy/client.py", line 160, in __init__
    super().__init__(**kwargs)
  File "/home/ogi/dev/eth-portal-venv/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 154, in __init__
    self._drain_lock = asyncio.Lock(
  File "/usr/lib/python3.10/asyncio/locks.py", line 78, in __init__
    super().__init__(loop=loop)
  File "/usr/lib/python3.10/asyncio/mixins.py", line 17, in __init__
    raise TypeError(
TypeError: As of 3.10, the *loop* parameter was removed from Lock() since it is no longer necessary

Environment

Python version:
3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]

Operating System: Linux-5.15.0-10033-tuxedo-x86_64-with-glibc2.35 (Kubuntu 22.04 LTS)

pip freeze result:
aiohttp==3.8.1
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.4.0
base58==2.1.1
bitarray==2.5.1
cached-property==1.5.2
certifi==2022.6.15
charset-normalizer==2.0.12
cytoolz==0.11.2
eth-abi==2.1.1
eth-account==0.5.8
eth-bloom==1.0.4
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-portal==0.1.0b0
eth-rlp==0.2.1
eth-typing==2.3.0
eth-utils==1.10.0
frozenlist==1.3.0
hexbytes==0.2.2
idna==3.3
ipfshttpclient==0.8.0a2
jsonschema==4.6.0
lru-dict==1.1.7
multiaddr==0.0.9
multidict==6.0.2
mypy-extensions==0.4.3
netaddr==0.8.0
parsimonious==0.8.1
protobuf==3.20.1
py-ecc==5.2.0
py-evm==0.5.0a3
pycryptodome==3.14.1
pyethash==0.1.27
pyrsistent==0.16.1
requests==2.28.0
rlp==2.0.1
six==1.16.0
sortedcontainers==2.4.0
ssz==0.2.4
toolz==0.11.2
trie==2.0.0a5
typing-extensions==3.10.0.2
urllib3==1.26.9
varint==1.0.2
web3==5.29.2
websockets==9.1
yarl==1.7.2

Could not correctly encode receipts for header #15511259 0x3017b9e4d0bb214e90dc002c6a52bd7bd920fb1e2eddd1cf9687c702cd9dc72a

If this is a bug report, please fill in the following sections.
If this is a feature request, delete and describe what you would like with examples.

What was wrong?

Ran into this error while running the client.

Full error output

Traceback (most recent call last):
  File "/home/pipermerriam/.pyenv/versions/3.9.13/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/pipermerriam/.pyenv/versions/3.9.13/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/__main__.py", line 4, in <module>
    launch_bridge()
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/run.py", line 51, in launch_bridge
    header_log_loop(w3, portal_inserter, block_filter, 6)
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/run.py", line 34, in header_log_loop
    handle_new_header(w3, portal_inserter, header_hash)
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/handle.py", line 32, in handle_new_header
    propagate_receipts(w3, portal_inserter, chain_id, block_fields, transactions)
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/history.py", line 213, in propagate_receipts
    content_key, content_value = encode_receipts_content(
  File "/home/pipermerriam/projects/eth-portal/eth_portal/bridge/history.py", line 247, in encode_receipts_content
    raise ValidationError(
eth_utils.exceptions.ValidationError: Could not correctly encode receipts for header #15511259 0x3017b9e4d0bb214e90dc002c6a52bd7bd920fb1e2eddd1cf9687c702cd9dc72a
 

Expected Result

This section may be deleted if the expectation is "don't crash".

The client not to crash.

How can it be fixed?

I suspect we will want that validation error to include more information on the exact contents it was trying to encode....

Add to docs: how to debug by running trin manually

Add a section about how to run trin manually (based on the command in the log output of eth-portal) and perpetually, launching and closing eth-portal. Mention that eth-portal looks for the IPC file match. Include a harmless version of how to generate a key.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.