Implement the basic structure for the repo including but not limited to

• basic readme
• basic github workflows
• basic Makefile
• minimal config (yet another config file 😬)

Add the bare bones GRPC server that can be configured to server to an optimint node. It should follow the interface defined here along with the BlockRetriever interface. See the mocked version in optimint. Note that this issue only covers creating the basic structure of the server, not actually implementing the core functionality of each rpc endpoint.

celestia-node returns ipld.ErrNotFoundInRange when GetSharesByNamespace is not able to find any shares in given namespace.
From Optimint point of view it's not an error - it's just a situation.

We need to handle this correctly, so optimint can continue processing next blocks, instead of retrying.

Currently, we are downloading the entire block to create the DataAvailabilityHeader in order to DAS. This obviously defeats the purpose of DAS, and was added a few months ago before celestia-node had implemented the ability to gossip headers. We want to change that, and actually use celestia-node properly.

• Stop downloading the entire block to create the DAH
• Use celestia-node to gossip headers, DAS properly, retrieve optimint blocks properly

The dalc should be updated so that it can make retrieveBlock and checkDataAvailability calls by connecting to a celestia-node full node.

Currently, we are using a json map to keep track of optimint heights and their corresponding celestia heights. As discussed in #17, we want to use something more robust, like a standard KV store.

This will better position the DALC to keep optimint platform agnostic, and offload some responsibility as well.

• Replace HeightMapper's in memory map with a KV store

Currently, when the dalc attempts to find optimint blocks, it assumes that there is only a single message in the namespace. We should changes this so that optimint does not miss blocks if there are more than one message in a namespace.

This will also require changing the response of the RetrieveBlock method to return multiple optimint blocks

It was decided in #142 that we will separate out the data availability light client from optimint and into it's own repo. Fortunately, a sizable chunk of the core logic needed for the first devnet implementation has already been written. However, there is still some work needed to be done. Notably, we still need a to query application specific data (see our old planned implementation), and decide precisely how we plan on performing DAS (see old ValidateAvailability func)

• #11
• #3
• #4
• #5
• #6

Originally posted by @tzdybal in #34 (review)

We should stop using tendermint/spm dependency, as it is no longer maintained.

Debugging and troubleshooting would be improved if the dalc grpc server supported reflection. This should be trivial to add following the steps outlined here: https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md#enable-server-reflection

This is the counterpart of rollkit/rollkit#345.

When reading data from Celestia, DALC may encounter a situation, when height of newly found block is not equal to height of current block +1.

For example:

• after block n, there is block n+2 - block n+1 is missing, or blocks are reordered
• after block n, there is block n-1 - this might be a duplicate or invalid block

There should be an automated step that ensures that a CHANGELOG.md file is appropriately updated for all merges to the main branch.

There should be a proper description in an issue and corresponding PR as well as a summary in the CHANGELOG.md file. This will be enforced through an automated action. In rare cases for small PRs it may be acceptable to skip this.

We can take measures to increase the number of platforms for which dalc can compile without modification. One way we can do that is to incorporate https://github.com/mitchellh/go-homedir when attempting to find the home directory.

See ADR 007

It has still not been decided precisely how we will sample data using via the DALC. This code could exist with the rest of the ipld code in celestia-node, or it could exist here. Further discussion is needed.

See the old ValidateAvailability code.

Summary of Bug

Yesterday I found following error in the logs in Optimint:

7:18PM ERR error while publishing block error="DA layer submission failed: failed to submit tx: code 32: account sequence mismatch, expected 14615, got 14614: incorrect account sequence" module=BlockManager

Version

@evan-forbes please confirm which version of DALC was used on your server.

Steps to Reproduce

I'm not able to reproduce. This error occurred when my server was severely overloaded (but TBH I don't see how this could affect execution).

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

We have discussed that long term, we will move the three methods currently served via the dalc will be served via a standard celestia-node instead celestiaorg/celestia-node#414 (comment) celestiaorg/celestia-node#414 (comment).

However, optimint's multi aggregator node network is blocked by these features, so a short term solution is in order so that we can continue building that.

One possible solution was to create a branch on celestia-node that incorporates a slightly more generalized version of the dalc server, and another is to use the preexisting Plugin implementation #55

Optimint changed definition of Header. It was never updated in DALC.

Depending on usage, DALC is treating $HOME or $HOME/.dalc as home directory.

We will need to be able to submit blocks to celestia for optimint. Most of the this code has already been written, we just have to plug it into the rest of the DALC. See old optimint code and app's client code.

Along with being responsible for data availability sampling, we also need the DALC to act as a tendermint light client. Ideally we would port all of the typical tendermint LC functionality here, but at the very least we will need to ability to:

• verify the signatures for a given commit
• sync headers from a specific height and trusted header hash

Currently, the dalc will sign and submit any block that pushed through the SubmitBlock rpc endpoint.

That is terrible for many reasons, and we should add different verification mechanisms to protect against simple mistakes or malicious parties.

• check optimint commit sig before signing and submitting the optimint block

related to #161

This is a tracking issue for fully functional Celestia DALC, required for testnet.

Block submission is ready. Now we need to finish block retrieval and data availability checks. Discussion #24 summarize how we should proceed.

• #5
• #6

dalc init returns:

Error: open /home/<username>/height_map.json: no such file or directory

This file should be created during the init phase.

For example: 2 aggregators decide to create a block at the same height (because of timeouts/network conditions, or as attack on rollup).

The readme needs minimal documentation and a simple overview diagram of how this repo relates to optimint, app/core, and node (could even be hand-drawn).

Per the original ADR in the celestia-core, we need to have the ability to retrieve the data from a specific namespace for a given block. We had planned a previous implementation here, but our plans were in limbo and we never completed it. Further discussion is needed as to exactly how we should implement this functionality, and in which repo the implementation should reside.

There should be a test ensuring that configuration can be stored in file and loaded from file.

This is a tracking issue for all non-trivial cases for DALC:

• #26
• #27
• #28
• #29
• #30

We should have a Github Action that builds and pushes a docker image to ghcr.io/celestiaorg/dalc:latest on every merge to the main branch.