coinbase / chainstorage Goto Github PK

First of all, huge thanks to all the contributors! Really appreciate all the work here.

I am trying to use this project to ingest a blockchain, for example, ethereum-mainnet. I have done the following steps:

1. set up AWS resources, DynamoDB tables, S3 bucket, and SQS queue.
2. set up a temporal instance.
3. update the config-template accordingly, and generate the config files.
4. I can successfully run the following command:
   go run ./cmd/admin workflow start --workflow backfiller --input '{"StartHeight": 11000000, "EndHeight": 11000010, "NumConcurrentExtractors": 24}' --blockchain ethereum --network mainnet --env production
   and I can verify that the data has been stored in the S3 bucket, block_table also have correct entries.

At this point, my understanding is that I manually backfill once by running the above command, then I should set up the poller by run the containers defined in docker-compose.yml. However, the cron service always throws error.

I also have trouble running other workflows, such as monitor, event_backfiller and poller.

Can anyone provide help to elaborate on how this project is supposed to be used? I have a hard time understanding the intentions from the readme doc. Thanks in advance!

We're working on automating deployment of ChainStorage on AWS, and I'm running into an issue with starting workflows.

There is currently no way to bypass the "Are you sure you want to start XXX" message in the admin CLI. I would love to be able to pass --confirm or set an environment variable so that no CLI interaction is required to start the workflows.

Unreviewed commit(s) COMPARE was merged to master by jiezhang.

Comment with appropriate number of 👍 to approve.

Reviewed requirements satisfied? No
Ancestors reviewed? Yes

Hi there,

I'm currently working on a workflow (#91) which replicate block data from another chainstorage.
Is there a way to apply for a launchpad api key (mentioned in the notion doc), so that I can test it?

We’ve been running Chainstorage on AWS and it’s been working out great so far. However, most of our infrastructure is on GCP. It would be great if Chainstorage could natively support GCP in addition to AWS. Are there any such plans in the pipeline? It would be a game changer for us and also expand its usability for a broader audience.

If there is any room for collaboration, we are open to contributing to the codebase and documentation.

Was able to run until bootstrap, but when trying to make build, got an error saying that there is No such file or directory. I also got the same error when executing as individual lines:
go run ./tools/config_gen ./config_templates/config . -- ran successfully
go-bindata -ignore secrets\.yml -ignore config/config.go -modtime 1640048354 -o config/config.go -pkg config config/.. -- failed

Error message:

make: go-bindata: No such file or directory
make: *** [config] Error 1

My guess is go-bindata is not where it's supposed to be?

ChainStorage is reliant on Temporal, which is a crucial dependency. However, it currently operates on an outdated version of Temporal. Even the current Temporal UI Docker is no longer supported.

It would be advisable to modify ChainStorage so that it can operate on a readily available Temporal deployment.

Current Implementation is AWS SQS based, we would like to provide abstraction and enable other storage impl

Abstract out the current DynamoDB based KV implementation and enable a pluggable storage layer

1. ChainNode as data source: Unlike normal node providers, Chainnode uses header to capture security token (yes, safer!) but currently Chainstorage doesn't seem to support this way to authenticate with a block provider.
2. ChainStorage as data source: Enable ChainStorage use another ChainStorage Blocks to sync can be 100x of the syncing performance (Assume 100 trx for each block)

Cc @jiezhang
#P1

Parent Ticket: #44

1. Provide blob storage abstraction for ChainStorage.
2. Move S3 dependency as a storage implementation

This is a large issue that may need to be broken down further:

Currently, ChainStorage is bound to AWS dependency

1. S3 as Blob storage
2. DynamoDB as Key-Value storage
3. SQS as dead letter queue

Why?
In order to help ChainStorage be more portable, we need to break down these hard-wired dependencies. A possible solution is to provide abstract interfaces for these storage solutions and implement the adaptor to different cloud providers accordingly.

We also explored a driver-level compatibility layer (e.g., CHainStorage continue to use S3 library interfaces, and we adopt the S3 driver to different cloud provider solution) - but it is a no-go due to complexity.

Tickets so far:

• #45
  • action #43 is working on Blob Storage Abstraction
• #59
• #60

PS We should have a local implementation - Blob Storage -> File, SQL (lite?) -> KV, and a simple table for DLQ, for testing and possibly local production use cases.

Would it be possible to get the expected schema/configuration of the 3 DynamoDB tables which store the metadata?

I tried to create them based on the source code, but I think I messed up one of the indexes, and an official guide would be much appreciated!

First off, I'd like to thank the contributors. This project has been incredibly useful for pulling various blockchains. However, I noticed that while there's an option for sidechains in the configuration, the implementation seems to be missing for sidechains in general. I'm particularly interested in the Ethereum Beacon Chain as its data is becoming increasingly important.

The request is to add support for pulling data from the Ethereum Beacon Chain. This would ideally include the ability to query Beacon APIs, particularly states, headers, blocks and light client APIs.

Thank you for considering this feature request.

For now, we don't have tests for dependency injections. We need a pipeline to run e2e tests for each of the supported blockchains with the following steps:

1. Run localstack
2. Run server
3. Run backfill to inject recent blocks
4. Use the sdk to fetch

This issue is an follow up of #29. Without a toolchain version manager, developers running different versions of the toolchain, especially golang and protoc, could run into conflicts.

One example is the generated protobuf code such as blockchain.pb.go.

As proposed by @bestmike007 we may leverage asdf to manage the toolchain version.

go install github.com/mikefarah/yq/v4@latest
required1.20 at least. and is incompatible with 1.18 as it's been deprecated

Mering CB internal ChainStorage development efforts into the opensource code base.