ethpandaops / dora Goto Github PK

Currently, the explorer fetches blobs directly from the CL node via RPC calls.
This approach won't work once blobs expire and are removed from the CL node.

To allow showing blobs after expiration, it should be possible to store blobs in DB or on an external storage (eg. S3).
This should be optional, so blob storage does not get a dependency to run the explorer.

I've noticed some issues with the orphaned block detection:
https://beaconlight-devnet.pk910.de/epoch/7346
https://beaconlight-devnet.pk910.de/epoch/7737

Looks like the reorg detection marks too many blocks as orphaned..
Needs further debugging

The explorer currently grabs the finalization status too early during epoch transition and therefore shows finalized epochs an epoch later than it should.

On kaustinen at least, it seems that the slot number is actually the block number (using a slot number of e.g. 74 yields the execution data for block number 74).

https://dora.kaustinen-testnet.ethpandaops.io/slot/74

But the block number itself is reported to be 116, which is wrong. If you look at the slots before and after, the block values are jumping widely, which has the result of the block numbers being AHEAD of the slot numbers. This should not happen on a testnet who merged at genesis.

Dora keeps getting stuck in rare situations.

This seems to be related to bad client connections, which makes dora running into a deadlock.
The dora core logic (including client pool & block cache) should be reworked to make such deadlocks impossible
(ie. use channels and contexts instead of locking)

I'm trying to start the explorer for a Devnet based on the Gnosis Chain Preset. Currently, it fails with the following message:

time="2023-10-03T18:27:49Z" level=fatal msg="error reading config file: tried to use unknown chain-preset: gnosis"

Possible Solution:

• Add Gnosis preset configuration to https://github.com/pk910/dora/tree/master/config. Perhaps including https://github.com/gobitfly/eth2-beaconchain-explorer/blob/master/config/gnosis.chain.yml

The explorer currently consumes quite much memory for operation.
Biggest memory consumptions:

• Loading & parsing of validator sets [~1.5GB per request on holesky]
• Unfinalized Block Cache [~1GB for >500 unfinalized epochs]
• Page Model Cache [200MB as per config]

Especially the first point should be improved.
The explorer currently does up to 2 validator set requests in parallel, so it's consuming up to 3GB just for parsing some giant json

The explorer should be able to load the current network specs from the connected beacon nodes.
This will make the explorer easier to run on custom devnets.

Noticed a panic in the holesky dora logs:

[negroni] PANIC: runtime error: cannot convert slice with length 0 to array or pointer to array with length 32
goroutine 223955 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
	/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:159 +0xc7
panic({0x1326780, 0xc03a70c570})
	/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/pk910/dora-the-explorer/handlers.SearchAhead({0x7f14ffc0d7e0, 0xc02118a028}, 0xc00dfb8700)
	/src/handlers/search.go:308 +0x210f
net/http.HandlerFunc.ServeHTTP(0xc00dfb8600?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0x446d3c?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000000, {0x7f14ffc0d7e0, 0xc02118a028}, 0xc00dfb8500)
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7f14ffc0d7e0, 0xc02118a028}, 0x12a5d60?, 0xc01fc0c120)
	/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x41e510?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0xc04dc8b900?, 0xc04dc8b978?)
	/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1817c80?, 0xc0004e6378?}, 0xc0004e63f0?}, {0x7f14ffc0d7e0, 0xc02118a028}, 0x12a5d60?)
	/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0x8?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0x0?, 0xc04dc8ba50?)
	/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:193 +0x86
github.com/urfave/negroni.middleware.ServeHTTP({{0x1815e60?, 0xc0001f6960?}, 0xc0004e63d8?}, {0x7f14ffc0d7e0, 0xc02118a028}, 0x0?)
	/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0001f3aa0, {0x1824c40?, 0xc020b340e0}, 0x46e7ce?)
	/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x125
net/http.serverHandler.ServeHTTP({0xc020cfc660?}, {0x1824c40, 0xc020b340e0}, 0xc00dfb8500)
	/usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc038e27dd0, {0x1825568, 0xc0001f3d70})
	/usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x5ed

Dora should be able to track & display deposits.

• Add execution layer rpc pool logic (copy from assertoor)
• Add deposits table to database
  (idx, time, pubkey, deposit origin, deposit sender, amount, withdrawal creds, valid)
• Add deposit contract log crawler

The attestantio/go-eth2-client implementation currently does not return the metadata fields supplied via various endpoints.
But the explorer needs the dependent_root field from /eth/v1/validator/duties/proposer/{epoch} call.
So there is currently a workaround code that does the getProposerDuties on its own.

This workaround code should be removed once the upstream library has support for these metadata fields.

• Wait for attestantio/go-eth2-client#35
• Remove workaround code here

Feature Request:
Add a list of attester / proposer slashings / track slashings in db

Dora currently stores blob sidecars with commitment as primary key.
This causes issues when commitments are re-used in multiple blob transactions.
In this situation, dora would currently override the blob entry from the older block.
It's effectively removing the blob sidecar link from the old block, which is obviously wrong.

The explorer currently can't use clients that do not explicitly follow the specs for some static api calls.
This is due to the initial connection checks implemented in attestantio/go-eth2-client.

I think these checks should be disabled for the explorer, as the explorer does its own error handling for the api calls it uses.
Especially for early testnets there might be clients that do not follow the specs that closely.
However, they should still be used as much as the explorer can use them.

• Wait for attestantio/go-eth2-client#75
• Disable initial connection check here

This issue currently prevents the explorer from using prysm nodes due to this prysm bug:
prysmaticlabs/prysm#12949
Edit: Prysm issue has been resolved

time="2024-02-02T12:45:44Z" level=info msg="http server listening on 0.0.0.0:8080"
time="2024-02-02T12:45:44Z" level=info msg="received block 0:0 [0x71ca733aff61e91d304fe23b869010fd267a4a3931a736c6729a7657ca670507] warmup, head" client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:45:44Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:45:54Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:46:04Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer

the client is stuck with the genesis block:

~# curl http://127.0.0.1:32832/eth/v1/beacon/headers
{"execution_optimistic":false,"finalized":false,"data":[{"root":"0x71ca733aff61e91d304fe23b869010fd267a4a3931a736c6729a7657ca670507","canonical":true,"header":{"message":{"slot":"0","proposer_index":"0","parent_root":"0x0000000000000000000000000000000000000000000000000000000000000000","state_root":"0xfe842cf4d68b07159db26cfca3ebfe14b030d48438a6e5e9745eea78f6b8153f","body_root":"0x74b4bb048d39c75f175fbb2311062eb9867d79b712907f39544fcaf2d7e1b433"},"signature":"0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}]}

Background

Whisk SSLE (Single Secret Leader Election) is a new Ethereum protocol feature being tested now. (ref: https://ethresear.ch/t/whisk-a-practical-shuffle-based-ssle-protocol-for-ethereum/11763)

The point of the feature is to identity of the next proposer private until its block is published. This means:

• proposer duties cannot be known in advance
• if there is a missed slot it will never be known how missed the slot

@parithosh has deployed a testnet for whisk https://github.com/ethpandaops/whisk-testnets/ with this explorer. BTW great work on the explorer, love it so far! ❤️

However the explorer stalls indexing at the Whisk fork epoch since it cannot poll duties. More precisely, the beacon node returns an error to calls to eth/v1/validator/duties/proposer/ since it cannot serve that data.

INFO[0008] synchronizing epoch 0                         module=synchronizer
WARN[0008] could not lazy load proposer duties for epoch 1501: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1501, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]}  client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1501  client=default module=indexer
WARN[0008] could not lazy load proposer duties for epoch 1503: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1503, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]}  client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1503  client=default module=indexer
WARN[0008] could not lazy load proposer duties for epoch 1502: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1502, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]}  client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1502  client=default module=indexer
INFO[0010] received block 1503:48117 [0x6a322ff15c787ce63c92a0c062f4ce462263cb8d58737e7829bc25d40ca88473] stream  client=default module=indexer
INFO[0015] index page called  

It would be great to have this explorer be compatible with Whisk for ease of debugging on devnets and to start iterating on how explorers would deal with post-whisk world with proposer privacy.

Please let me know what ideas you have to deal with case, I will suggest some ideas below

Suggested solution A (minimal)

Add a config value whisk_epoch after which the indexer fills the proposer duties response with dummy values (i.e. the 0 validator). This won't require any changes to the UI, API, etc and should allow to continue indexing

Suggested solution B (full)

Add a config value whisk_epoch after which indexer changes the render of proposer duties.

1. The author of the block can be shown only for non-missed blocks and should be read from the block.proposer_index
2. For missed slots, do not display any index but some tag like "unknown"
3. On the epoch view, list proposers only of already seen blocks, following points 1 and 2

Feature Request:
Dora should index the EL block extra data field and allow pattern search via the filtered slots page

The explorer sees the forked off chain from nimbus clients as canonical chain here.
This is a non-persistent issue, which already resolved itself a block after.
The explorer selects the canonical chain head on the fly based on the heads from all connected clients.
It looks like the nimbus head was selected here because it has the highest slot number. This logic should be improved.

Dear @ethpandaops

We used Dora for our project: Canxium, it's really awesome, could you please give us your wallet, we'd like to send you some CAU coin as a donation (metamask wallet is fine)

Best Regards.
Neo Krypt.

if config.yaml has minimal preset, dora becomes unhappy.

All nodes are still connected, but they all report that they stuck syncing at slot 1.

None of the slots seem to be proposed, even tho they were.

To repro:

kurtosis run github.com/kurtosis-tech/ethereum-package@bbusa/preset --args-file https://github.com/kurtosis-tech/ethereum-package/blob/bbusa/preset/.github/tests/minimal.yaml

this is need bigtable sql sir ?

Feature request:
The explorer should be able to show eth transaction details on the block details page.

Current status:
The explorer only displays the transaction count, but no further transaction details.
Technically the signed transactions are available for post-bellatrix blocks without adding further complexity by calling a EL RPC.
However, they're not yet decoded or visualized.

I think it'd be great to get some basic details (hash, from/to addr, amount, & call data) displayed for each transaction in the block details.
Further transaction details like receipt information, events, internal transactions or tx traces should not be included for the first implementation.
Showing that stuff would need some EL RPC interaction, which introduces additional complexity that might not be worth adding.

Decoding the transactions should be easy: example

I think that's a good first issue for anyone who wants to join development for this explorer :)
Anyone who contributed to beaconcha.in before will probably feel familiar with the structure of this lightweight explorer ;)

This is not implemented yet and will cause some index inconsistency when it happens.
However, I guess such big reorgs normally shouldn't happen ever.

Doesn't look good ;)
The graph column should be properly sized, so the graph is fully visible

the dora explorer does a good job displaying the consensus slot with the execution payload

https://dora-holesky.pk910.de/slot/2

however, it does not do it for genesis. the structure should be the same though?

https://dora-holesky.pk910.de/slot/0