paritytech / substrate-api-sidecar Goto Github PK

View Code? Open in Web Editor NEW

244.0 13.0 151.0 31.45 MB

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.

Home Page: https://paritytech.github.io/substrate-api-sidecar/dist/

License: GNU General Public License v3.0

TypeScript 98.31% Rust 0.28% JavaScript 0.17% Shell 0.08% Dockerfile 0.01% Smarty 0.02% Lua 1.10% Just 0.02%

substrate-api-sidecar's Introduction

@substrate/api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.

Prerequisites

<= v15.0.0

This service requires Node versions 14 or higher.

Compatibility:

Node Version	Stablility
v14.x.x	Stable
v16.x.x	Stable
v17.x.x	Stable
v18.x.x	Stable
v19.x.x	stable

>= v16.0.0

This service requires Node versions 18.14 or higher.

Compatibility:

Node Version	Stablility
v18.14.x	Stable
v20.x.x	Stable
v21.x.x	Pending

NOTE: Node LTS (long term support) versions start with an even number, and odd number versions are subject to a 6 month testing period with active support before they are unsupported. It is recommended to use sidecar with a stable actively maintained version of node.js.

NPM package installation and usage
Source code installation and usage
Configuration
Debugging fee and staking payout calculations
Available endpoints
Chain integration guide
Docker
Notes for maintainers
Hardware requirements

NPM package installation and usage

Global installation

Install the service globally:

npm install -g @substrate/api-sidecar
# OR
yarn global add @substrate/api-sidecar

Run the service from any directory on your machine:

substrate-api-sidecar

To check your version you may append the --version flag to substrate-api-sidecar.

Local installation

Install the service locally:

npm install @substrate/api-sidecar
# OR
yarn add @substrate/api-sidecar

Run the service from within the local directory:

node_modules/.bin/substrate-api-sidecar

Finishing up

Jump to the configuration section for more details on connecting to a node.

Click here for full endpoint docs.

In the full endpoints doc, you will also find the following trace related endpoints :

/experimental/blocks/{blockId}/traces/operations?actions=false
/experimental/blocks/head/traces/operations?actions=false
/experimental/blocks/{blockId}/traces
/experimental/blocks/head/traces

To have access to these endpoints you need to :

Run your node with the flag —unsafe-rpc-external
Check in sidecar if BlocksTrace controller is active for the chain you are running.

Currently BlocksTrace controller is active in Polkadot and Kusama.

Source code installation and usage

Quick install

Simply run yarn.

Rust development installation

If you are looking to hack on the calc Rust crate make sure your machine has an up-to-date version of rustup installed to manage Rust dependencies.

Install wasm-pack if your machine does not already have it:

cargo install wasm-pack

Use yarn to do the remaining setup:

yarn

Running

# For live reload in development
yarn dev

# To build and run
yarn build
yarn start

Jump to the configuration section for more details on connecting to a node.

Configuration

To use a specific env profile (here for instance a profile called 'env.sample'):

NODE_ENV=sample yarn start

For more information on our configuration manager visit its readme here. See Specs.ts to view the env configuration spec.

Express server

SAS_EXPRESS_BIND_HOST: address on which the server will be listening, defaults to 127.0.0.1.
SAS_EXPRESS_PORT: port on which the server will be listening, defaults to 8080.
SAS_EXPRESS_KEEP_ALIVE_TIMEOUT: Set the keepAliveTimeout in express.

Substrate node

SAS_SUBSTRATE_URL: URL to which the RPC proxy will attempt to connect to, defaults to ws://127.0.0.1:9944. Accepts both a websocket, and http URL.

Metrics Server

SAS_METRICS_ENABLED: Boolean to enable the metrics server instance with Prometheus (server metrics) and Loki (logging) connections. Defaults to false.
SAS_METRICS_PROM_HOST: The host of the prometheus server used to listen to metrics emitted, defaults to 127.0.0.1.
SAS_METRICS_PROM_PORT: The port of the prometheus server, defaults to 9100.
SAS_METRICS_LOKI_HOST: The host of the loki server used to pull the logs, defaults to 127.0.0.1.
SAS_METRICS_LOKI_PORT: The port of the loki server, defaults to 3100

Custom substrate types

Some chains require custom type definitions in order for Sidecar to know how to decode the data retrieved from the node. Sidecar affords environment variables which allow the user to specify an absolute path to a JSON file that contains type definitions in the corresponding formats. Consult polkadot-js/api for more info on the type formats (see RegisteredTypes). There is a helper CLI tool called generate-type-bundle that can generate a typesBundle.json file for you using chain information from @polkadot/apps-config. The generated json file from this tool will work directly with the SAS_SUBSTRATE_TYPES_BUNDLE ENV variable.

SAS_SUBSTRATE_TYPES_BUNDLE: a bundle of types with versioning info, type aliases, derives, and rpc definitions. Format: OverrideBundleType (see typesBundle).
SAS_SUBSTRATE_TYPES_CHAIN: type definitions keyed by chainName. Format: Record<string, RegistryTypes> (see typesChain).
SAS_SUBSTRATE_TYPES_SPEC: type definitions keyed by specName. Format: Record<string, RegistryTypes> (see typesSpec).
SAS_SUBSTRATE_TYPES: type definitions and overrides, not keyed. Format: RegistryTypes (see types).

You can read more about defining types for polkadot-js here.

Connecting a modified node template

Polkadot-js can recognize the standard node template and inject the correct types, but if you have modified the name of your chain in the node template you will need to add the types manually in a JSON types file like so:

// my-chains-types.json
{
  "Address": "AccountId",
  "LookupSource": "AccountId"
}

and then set the enviroment variable to point to your definitions:

export SAS_SUBSTRATE_TYPES=/path/to/my-chains-types.json

Logging

SAS_LOG_LEVEL: The lowest priority log level to surface, defaults to info. Tip: set to http to see all HTTP requests.
SAS_LOG_JSON:Whether or not to have logs formatted as JSON, defaults to false. Useful when using stdout to programmatically process Sidecar log data.
SAS_LOG_FILTER_RPC: Whether or not to filter polkadot-js API-WS RPC logging, defaults to false.
SAS_LOG_STRIP_ANSI: Whether or not to strip ANSI characters from logs, defaults to false. Useful when logging RPC calls with JSON written to transports.
SAS_LOG_WRITE: Whether or not to write logs to a log file. Default is set to false. Accepts a boolean value. The log files will be written as logs.log. NOTE: It will only log what is available depending on what SAS_LOG_LEVEL is set to.
SAS_LOG_WRITE_PATH: Specifies the path to write the log files. Default will be where the package is installed.
SAS_LOG_WRITE_MAX_FILE_SIZE: Specifies in bytes what the max file size for the written log files should be. Default is 5242880 (5MB). NOTE Once the the max amount of files have reached their max size, the logger will start to rewrite over the first log file.
SAS_LOG_WRITE_MAX_FILES: Specifies how many files can be written. Default is 5.

Log levels

Log levels in order of decreasing importance are: error, warn, info, http, verbose, debug, silly.

http status code range	log level
`code` < 400	`http`
400 <= `code` < 500	`warn`
500 < `code`	`error`

RPC logging

If looking to track raw RPC requests/responses, one can use yarn start:log-rpc to turn on polkadot-js's logging. It is recommended to also set SAS_LOG_STRIP_ANSI=true to increase the readability of the logging stream.

N.B. If running yarn start:log-rpc, the NODE_ENV will be set to test. In order still run your .env file you can symlink it with .env.test. For example you could run ln -s .env.myEnv .env.test && yarn start:log-rpc to use .env.myEnv to set ENV variables. (see linux commands ln and unlink for more info.)

Prometheus server

Prometheus metrics can be enabled by running sidecar with the following env configuration: SAS_METRICS_ENABLED=true

You can also expand the metrics tracking capabilities to include query params by adding to the env configuration: SAS_METRICS_INCLUDE_QUERYPARAMS=true

The metrics endpoint can then be accessed :

on the default port : http://127.0.0.1:9100/metrics or
on your custom port if you defined one : http://127.0.0.1:<YOUR_CUSTOM_PORT>/metrics

A JSON format response is available at http://127.0.0.1:9100/metrics.json.

That way you will have access to the default prometheus node instance metrics and the following metrics will be emitted for each route:

sas_http_request_error: type counter and tracks http errors occuring in sidecar
sas_http_request_success: type counter and tracks successfull http requests
sas_http_requests: type counter and tracks all http requests
sas_request_duration_seconds: type histogram and tracks the latency of the requests
sas_response_size_bytes_seconds: type histogram and tracks the response size of the requests
sas_response_size_latency_ratio_seconds: type histogram and tracks the response bytes per second of the requests

The blocks controller also includes the following route-specific metrics:

sas_extrinsics_in_request: type histogram and tracks the number of extrinsics returned in the request when a range of blocks is queried
sas_extrinsics_per_second: type histogram and tracks the returned extrinics per second
sas_extrinsics_per_block: type histogram and tracks the returned extrinsics per block
sas_seconds_per_block: type histogram and tracks the request time per block

The metrics registry is injected in the Response object when the SAS_METRICS_ENABLED flag is set to true in the .env file, allowing to extend the controller based metrics to any given controller from within the controller functions.

To successfully run and access the metrics and logs in Grafana (for example) the following are required:

prometheus server (info here)
loki server and promtail (info here)

For mac users using homebrew:

brew install prometheus loki promtail

Debugging fee and staking payout calculations

It is possible to get more information about the fee and staking payout calculation process logged to the console. Because these calculations happens in the statically compiled web assembly part, a re-compile with the proper environment variable set is necessary:

CALC_DEBUG=1 sh calc/build.sh

Available endpoints

Click here for full endpoint docs.

Chain integration guide

Click here for chain integration guide.

Docker

With each release, the maintainers publish a docker image to dockerhub at parity/substrate-api-sidecar

Pull the latest release

docker pull docker.io/parity/substrate-api-sidecar:latest

The specific image tag matches the release version.

Or build from source

yarn build:docker

Run

# For default use run:
docker run --rm -it --read-only -p 8080:8080 substrate-api-sidecar

# Or if you want to use environment variables set in `.env.docker`, run:
docker run --rm -it --read-only --env-file .env.docker -p 8080:8080 substrate-api-sidecar

NOTE: While you could omit the --read-only flag, it is strongly recommended for containers used in production.

then you can test with:

curl -s http://0.0.0.0:8080/blocks/head | jq

N.B. The docker flow presented here is just a sample to help get started. Modifications may be necessary for secure usage.

Contribute

Need help or want to contribute ideas or code? Head over to our CONTRIBUTING doc for more information.

Notes for maintainers

Commits

All the commits in this repo follow the Conventional Commits spec. When merging a PR, make sure 1) to use squash merge and 2) that the title of the PR follows the Conventional Commits spec.

Updating polkadot-js dependencies

Whenever the polkadot-js ecosystem releases a new version, it's important to keep up with these updates and review the release notes for any breaking changes or high priority updates. In order to update all the dependencies and resolutions, create a new branch, such as yourname-update-pjs, and then run yarn up "@polkadot/*" in that branch.
- @polkadot/api release notes
- @polkadot/util-crypto release notes
- @substrate/calc npm release page

Ensure everything is up to date and working by running the following:

yarn
yarn dedupe
yarn build
yarn lint
yarn test
yarn test:historical-e2e-tests
yarn test:latest-e2e-tests

Commit the dependency updates with a name like chore(deps): update polkadot-js deps (adjust the title based on what was updated; refer to the commit history for examples). Then, wait for it to be merged.
Follow RELEASE.md next if you're working through a full sidecar release. This will involve creating a separate PR where the changelog and versions are bumped.

Maintenance Guide

A more complete list of the maintainer's tasks can be found in the MAINTENANCE.md guide.

Hardware requirements

Disk Space

Sidecar is a stateless program and thus should not use any disk space.

Memory

The requirements follow the default of node.js processes which is an upper bound in HEAP memory of a little less than 2GB thus 4GB of memory should be sufficient.

Running sidecar and a node

Please note that if you run sidecar next to a substrate node in a single machine then your system specifications should improve significantly.

Our official specifications related to validator nodes can be found in the polkadot wiki page.
Regarding archive nodes :
- again as mentioned in the polkadot wiki page, the space needed from an archive node depends on which block we are currently on (of the specific chain we are referring to).
- there are no other hardware requirements for an archive node since it is not time critical (archive nodes do not participate in the consensus).

Benchmarks

During the benchmarks we performed, we concluded that sidecar would use a max of 1.1GB of RSS memory.

The benchmarks were:

using 4 threads over 12 open http connections and
were overloading the cache with every runtime possible on polkadot.

Hardware specs in which the benchmarks were performed:

Machine type:
n2-standard-4 (4 vCPUs, 16 GB memory)

CPU Platform:
Intel Cascade Lake

Hard-Disk:
500GB

substrate-api-sidecar's People

Contributors

Stargazers

Watchers

Forkers

emostov nodlecode shenyaqi9527 chesterkuo tibicsko chevdor benwhitejam hewigovens krichard410 mogwaicoin darkfriend77 wo4zhuzi stafiprotocol tarikgul quoinefinancial yessineamor y12studio danforbes cennznet sekmet fantasyup ledgerhq daywednes brettkolodny loicttn amarts lovesh ljzn stateless-money datahighway-dhx michaelredman bifrost-finance littleshirin subgame-network playzero kiltprotocol polka-defi oxydev2021 hupsuni woss znmead xenna-blocks couragebc polymeshassociation konomi-network cryptopossum logion-network lxex sora-xor tomaszwaszczyk bhonetwork papermoonio eimex-exchange tankofzion deepbrainchain 5irechain rafaelapb szwx855 graomelo plugblockchain skydragonboy mon4lx-finance astarnetwork bradleystachurski qkl-repo zhangwweihui paulvi xlc xp-network evankhell561 hoonsubin ketamacbd toolsopen edgeware-network manta-network metapioneer amintalebi grenade meierpatrick hzccc timbrinded uni-arts-chain irediaes veydecapia sawvox vivekvpandya yashirooooo centrifuge ark930 lukanus muddlebee benxiao swowk jasonweb3 hashwarlock lhtvineettiwari damianstraszak wblockchain stella3d whisker17

substrate-api-sidecar's Issues

Requesting a block that doesn't yet exist returns a 500

If you try an request a block by number that does not exist you get {"error":"Interal Error"}. I think it should instead give a more descriptive error, like {"error":"requested block does not exist"}. Maybe also have an appropriate 4xx status code.

An example of the issue would be to hook sidecar up to wss://rpc.polkadot.io and send GET localhost:8080/block/{currentBlockNumber + 5}.

Parse call options

Some calls actually take other calls as their arguments, for example as_multi or batch transactions. Each inner call has a callIndex and call to identify the call. callIndex refers to the index of the pallet in the metadata, not counting those with null calls, and call is the index of the call within that module.

Instead of displaying { callIndex: '06', call: '05' }, display 'staking.nominate' (more generally, 'pallet.methodName').

Fix sanitizeNumbers

Currently sanitizeNumbers uses internal properties of Codec, which is not not reliable. Further it uses any for both param and return types.

Ideally sanitizeNumbers can gracefully handle options, and may possibly need logic specific to handling the types *Int, Option, Tuple, Vec, Struct, Result, *Map, *Set, and Enum.

A very rough code snippet from @jacogr may give inspiration to (part of a) possible solution.

function sanitize (value: Codec): number | string | null {
  if (value instanceof AbstractInt) {
    return value.toString(10);
  } else if (value instanceof Option) {
    return value.isSome
      ?  sanitize(value.unwrap()) // make another pass, these can be nested
      : null;
  } else if (value instanceof Struct) {
   ...
  } else if (...) { ... }
}

A good fix to this could also encapsulate #89 by including unit tests.

CI checks for lint errors

Give the exact era in numeric form under the path `tx/artifacts`

For discussion.

`/staking/ADDRESS` fields `total` and `active` should always have decimal values.

Sometimes /staking/ADDRESS inappropriately returns non-decimal values for total and active. The values for these fields should be decimals.

Investigating Adding Versioned Paths to Sidecar

This Issue is being opened to track an investigation into versioning Sidecar via URL paths. Unversioned requests could be routed to the latest version.

Wrong env variable in README - 'SAS_SUBSTRATE_NODE_WS_URL'

"SAS_SUBSTRATE_NODE_WS_URL" environment variable does not override config when starting up, as the app defaults to 'ws://127.0.0.1:9944', changing it to "SAS_SUBSTRATE_WS_URL" seems to do the trick. *Running via docker, passing in as an env variable

Add endpoint for generic storage queries

Add an endpoint that enables querying any storage item from any pallet. It will need to handle all types of storage items, like values, maps, and double maps. It will also need to handle a block ID input for querying that item at that block ID.

There are a lot of ways to implement this, but perhaps something like:

/storage/<pallet>/<item>?key1=abc&key2=def

Use jest to unit test sanitizeNumbers

Configure Jest in the repo and setup some basic sanity checks for sanitizeNumbers.

Add the field `author: <AccountId>` to `fetchBlock`'s response

For some use cases it may helpful to have an author field for the block endpoint.

Add compatibility notice to readme

Would it make sense to add some compatibility notice to the readme?

Substrate version/commit
Runtime configuration (I reckon it assumes a FRAME runtime with balances pallet?)

I'm afraid the framework nature of Substrate will get lost otherwise and it's pretty confusing to users.

Return vesting info

Endpoint that accepts an address and returns the info from vesting.vesting.

Endpoint to fetch block

There should be an endpoint to retrieve a particular block. If the optional parameter number is provided, this endpoint returns the block at number. Otherwise, it returns the finalized head.

The block response includes

block_hash
extrinsics
number
parent_hash
timestamp

Each extrinsic includes

events
- Used to determine (among other things):
  - the success or failure of the extrinsic, so a status would also suffice
  - the source, dest, and value in a balance transfer
- Currently the data in an event is returned as an array. It would be great if this was a map that provided a key for each data blob or another array was returned with a key that mapped to each data value.
hash
method (module_name + method_name)
is_transaction
- Specifies if the extrinsic is a transaction

If an extrinsic is a transaction, the extrinsic should also have a transaction field which includes

signature
fee, which is the total of the BaseFee, ByteFee, WeightFee, tip, and any Function Fee (like the transfer fee) <-- not sure if function fee part is possible, this is part of an Event from the Balances module, while the other fees (Base, Byte, Weight, tip) come from a System module Event.

{
  "block_hash": "0xca7b3749ea85ae0a604f240b7d1ac7a497f0ec5365398866f6af260579b389e7",
  "extrinsics": [
    {
      "events": [
        {
          "data": {},
          "documentation": [" An extrinsic completed successfully."],
          "method": "system.ExtrinsicSuccess"
        }
      ],
      "hash": "0xde069e59a2363b86dd194b822358a3e9ea54e44b427368f361d5b462d1d7d62f",
      "is_transaction": false,
      "method": "timestamp.set"
    },
    {
      "events": [
        {
          "data": {},
          "documentation": [" An extrinsic completed successfully."],
          "method": "system.ExtrinsicSuccess"
        }
      ],
      "hash": "0x725b26fd3339629bfa297bd9b77af4056513504081e9c78f7c6a7373e81517bf",
      "is_transaction": false,
      "method": "finalityTracker.finalHint"
    },
    {
      "events": [
        {
          "data": {},
          "documentation": [" An extrinsic completed successfully."],
          "method": "system.ExtrinsicSuccess"
        }
      ],
      "hash": "0x9ea9fd936a97deecfb3152b5664e32e23c45d50a36a53da3622ba5bec1942b66",
      "is_transaction": false,
      "method": "parachains.setHeads"
    },
    {
      "events": [
        {
          "data": {
            "address": "ELmaX1aPkyEF7TSmYbbyCjmSgrBpGHv9EtpwR2tk1kmpwvG",
            "account_index": "4SXe4"
           },
          "documentation": [
            " A new account index was assigned.",
            "",
            " This event is not triggered when an existing index is reassigned",
            " to another `AccountId`."
          ],
          "method": "indices.NewAccountIndex"
        },
        {
          "data": {
            "address": "ELmaX1aPkyEF7TSmYbbyCjmSgrBpGHv9EtpwR2tk1kmpwvG",
            "balance": "50000000000000"
          },
          "documentation": [" A new account was created."],
          "method": "balances.NewAccount"
        },
        {
          "data": {
            "from": "F7uBbx4pbZ5u7eRGPExD6SKSA6TVqTsLf7daXYjAeEChcEY",
            "to": "ELmaX1aPkyEF7TSmYbbyCjmSgrBpGHv9EtpwR2tk1kmpwvG",
            "value": "50000000000000",
            "fees": "100000000"
          },
          "documentation": [" Transfer succeeded (from, to, value, fees)."],
          "method": "balances.Transfer"
        },
        {
          "data": {
            "sudo": "true"
          },
          "documentation": [" A sudo just took place."],
          "method": "sudo.Sudid"
        },
        {
          "data": {},
          "documentation": [" An extrinsic completed successfully."],
          "method": "system.ExtrinsicSuccess"
        }
      ],
      "hash": "0x262b47e724c7a5f0dad24695993d42963a48224fa0680ad221010d7817a3b884",
      "method": "sudo.sudo",
      "is_transaction": true,
      "transaction": {
        "signature": {
          "signer": "FcxNWVy5RESDsErjwyZmPCW6Z8Y3fbfLzmou34YZTrbcraL",
          "signature": {
            "Sr25519": "0x861bcb9c505fe8e24763f39dcffa4a600aba856f860c4787d628bf7dd7589f3b90bc9f1c478dde688fae4b54ad631d340981747a05ff12692b5f2016e7471300"
          },
          "era": {
            "MortalEra": "0x0500"
          },
          "nonce": 114,
          "tip": 0
        },
        "fee": "120000000"
      }
    }
  ],
  "number": 650689,
  "parent_hash": "0x5733ed31c4dbf5f04d394f55c845999d5a5fca7e5fe7508a30cf1580f5ff802d",
  "timestamp": 1573657194000
}

Add node info path

Add a node info path. We should design the API such that we can add other node/client endpoints in addition. E.g. /node/{health, version, networking, txpool}. It should return (off the top of my head, let's look deeper):

version
peers
node health

Add fee prediction endpoint

Transaction Payment provides a queryInfo endpoint to predict current transaction fees given the current fee multiplier. Provide an endpoint to call this and get current fee info.

Alternatively: use the latest fee multiplier and calculate this directly in sidecar.

Run yarn start failed

When I update the latest version of the program and run it on a mac, I get this error

yarn run v1.22.0
$ tsc && node ./build/main.js
Running on http://127.0.0.1:18080/
events.js:298
throw er; // Unhandled 'error' event
^

Error: listen EADDRINUSE: address already in use 127.0.0.1:18080
at Server.setupListenHandle [as _listen2] (net.js:1309:16)
at listenInCluster (net.js:1357:12)
at doListen (net.js:1496:7)
at processTicksAndRejections (internal/process/task_queues.js:85:21)
Emitted 'error' event on Server instance at:
at emitErrorNT (net.js:1336:8)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'EADDRINUSE',
errno: -48,
syscall: 'listen',
address: '127.0.0.1',
port: 18080
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I checked, 18080 port is not occupied by other programs

Get Block Should Accept Hash

A lot of systems, including Sidecar's console log, identify blocks by hash rather than number. the block/:number endpoint should be able to accept either an int or a hash and return the block accordingly.

Sample error from Sidecar's console:

Internal Error: Error: Missing extrinsic 2 in block 0x9e49962fbbfca0d6921e422e4fc4e3e5efbf4cfe3318847697c035891e9c0a12

Transaction Construction Endpoint

Make an endpoint that returns everything needed to make transactions in txwrapper. It should take an optional block number (return the finalized head if not provided) and return:

metadata
runtime spec version
checkpoint block number (finalized head or param)
checkpoint block hash

txConstruction(block: number) -> (
    metadata: string,
    specVersion: number,
    blockNumber: number,
    blockHash: string
)

`at` field for `fetchClaimsInfo`

In order to create a more consistent API and give more meaningful responses there should be an at field with a block hash and number for fetchClaimsInfo.

QueryInfo not working for all extrinsics

The queryInfo enpoint does not return fee information for a subset of transactions. Not sure if this is a Substrate issue or Polkadot JS API issue.

Edit:

polkadot v0.8.2 --chain kusama
"@polkadot/api": "^1.17.2"

Block 2603042: Fee error on staking.withdrawUnbonded
Block 2603870: Fee error on staking.withdrawUnbonded
Block 2605012: Fee error on staking.withdrawUnbonded
Block 2605556: Fee error on staking.withdrawUnbonded
Block 2606071: Fee error on staking.withdrawUnbonded
Block 2606490: Fee error on staking.withdrawUnbonded
Block 2606494: Fee error on staking.withdrawUnbonded
Block 2609401: Fee error on electionsPhragmen.reportDefunctVoter
Block 2609891: Fee error on electionsPhragmen.reportDefunctVoter
Block 2609993: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610027: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610041: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610069: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610074: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610079: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610092: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610097: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610131: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610131: Fee error on electionsPhragmen.reportDefunctVoter
Block 2610179: Fee error on electionsPhragmen.removeMember
Block 2610184: Fee error on electionsPhragmen.reportDefunctVoter
Block 2611935: Fee error on democracy.second
Block 2612358: Fee error on democracy.second
Block 2612590: Fee error on staking.withdrawUnbonded
Block 2612797: Fee error on democracy.unlock
Block 2613087: Fee error on staking.withdrawUnbonded
Block 2613416: Fee error on staking.withdrawUnbonded
Block 2617290: Fee error on staking.withdrawUnbonded
Block 2619342: Fee error on staking.withdrawUnbonded
Block 2621370: Fee error on staking.withdrawUnbonded
Block 2625054: Fee error on staking.withdrawUnbonded
Block 2626339: Fee error on staking.withdrawUnbonded

Sidecar log is:

2020-06-06 16:34:00        RPC-CORE: queryInfo(extrinsic: Bytes, at?: BlockHash): RuntimeDispatchInfo:: 2: Unable to query dispatch info.: Error decoding field Call :: Staking.0
Error: 2: Unable to query dispatch info.: Error decoding field Call :: Staking.0
    at RpcCoder._checkError (/home/parity/projects/substrate-api-sidecar/node_modules/@polkadot/rpc-provider/coder/index.js:83:13)
    at RpcCoder.decodeResponse (/home/parity/projects/substrate-api-sidecar/node_modules/@polkadot/rpc-provider/coder/index.js:46:10)
    at WsProvider.value (/home/parity/projects/substrate-api-sidecar/node_modules/@polkadot/rpc-provider/ws/Provider.js:166:90)
    at W3CWebSocket.value [as onmessage] (/home/parity/projects/substrate-api-sidecar/node_modules/@polkadot/rpc-provider/ws/Provider.js:146:153)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/home/parity/projects/substrate-api-sidecar/node_modules/yaeti/lib/EventTarget.js:107:17)
    at W3CWebSocket.onMessage (/home/parity/projects/substrate-api-sidecar/node_modules/websocket/lib/W3CWebSocket.js:234:14)
    at WebSocketConnection.<anonymous> (/home/parity/projects/substrate-api-sidecar/node_modules/websocket/lib/W3CWebSocket.js:205:19)
    at WebSocketConnection.emit (events.js:315:20)
    at WebSocketConnection.processFrame (/home/parity/projects/substrate-api-sidecar/node_modules/websocket/lib/WebSocketConnection.js:554:26)
    at /home/parity/projects/substrate-api-sidecar/node_modules/websocket/lib/WebSocketConnection.js:323:40

Use middleware to catch errors and sanitize response

Fix PaysFee Parsing

Right now, ExtrinsicSuccess[data['paysFee']] = False for all extrinsics because paritytech/substrate#5733 changed pays_fee from bool to enum Pays { Yes, No } and since Yes decodes to 0, it reads as False. Update the parsing of this event to display correctly.

Example extrinsic (see extrinsic.events.5.data.paysFee):

{'method': 'balances.transferKeepAlive',
   'signature': {'signature': '0x841f8717395dcbd5fa2dd3e5dd610dffdc60c97338ff07b9c94c607ee9604d2d742348d5c0f80ac2838c242ce05eca600cb29c0bd9e6bd5d3fa35a0b0eedaf82',
    'signer': '5CAcTqah24vuWeU5YFwGNDukHhRYVTDq6QEKBaJ8LVLxYsEJ'},
   'nonce': '0',
   'args': ['5FnVcw9CKSkMTCT6HwABbW7dQhfitUi6TqBXihcLMTzSh55H',
    '10000000000000'],
   'tip': '0',
   'hash': '0x938199b5481f5be4a8195c90ca23b74a79f81f89e1d793efa6b55a8e55136dc5',
   'info': {'weight': '175000000',
    'class': 'Normal',
    'partialFee': '168827389'},
   'events': [{'method': 'system.NewAccount',
     'data': ['5FnVcw9CKSkMTCT6HwABbW7dQhfitUi6TqBXihcLMTzSh55H']},
    {'method': 'balances.Endowed',
     'data': ['5FnVcw9CKSkMTCT6HwABbW7dQhfitUi6TqBXihcLMTzSh55H',
      '10000000000000']},
    {'method': 'balances.Transfer',
     'data': ['5CAcTqah24vuWeU5YFwGNDukHhRYVTDq6QEKBaJ8LVLxYsEJ',
      '5FnVcw9CKSkMTCT6HwABbW7dQhfitUi6TqBXihcLMTzSh55H',
      '10000000000000']},
    {'method': 'treasury.Deposit', 'data': ['135063165']},
    {'method': 'balances.Deposit',
     'data': ['5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY', '33765792']},
    {'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '175000000', 'class': 'Normal', 'paysFee': False}]}],
   'success': True}

Explicit return types for public ApiHandler functions

Return on_initialize/finalize events in /block

Not all events come from transactions and inherents. Some emit from the on_initialize and on_finalize hooks, for example, democracy enactment, scheduler execution, or both (e.g. a democracy::enact that schedules set_code).

Current behavior:

Block 2201991 returns from the sidecar:

{'number': '2201991',
 'hash': '0x478f359d51519b954a49567ae01afd2d62b315bd06d94e433fba212c8a7eef76',
 'parentHash': '0x2ce221548b02173d885c73c5112908ae810d401704bd278ac551573559f799a4',
 'stateRoot': '0x49cd57fd1e0e111c907cab8369040ecb2eda26ce9ac9c1a89ccc193f704fc755',
 'extrinsicsRoot': '0x2f74dff55ce3d709b7e8f080ff835d9165297ea41b4da122a34157d1f42784e6',
 'logs': [{'type': 'PreRuntime',
   'index': '6',
   'value': ['BABE', '0x0298000000a49fc80f00000000']},
  {'type': 'Seal',
   'index': '5',
   'value': ['BABE',
    '0x6a78c40bd1a966bed24535a4902656930ccde1c4e1dc2fe59df80255261a2b4f3b2ca0c69e7bb46771668b39a948c40278dea24f338803e0e467f6e2df1d2b81']}],
 'extrinsics': [{'method': 'timestamp.set',
   'signature': None,
   'nonce': '0',
   'args': ['1588837848000'],
   'tip': '0',
   'hash': '0x0e375409ed055dd882fbfa297c8946382fbbe227b4e1bfd69be5fd260f26bdf2',
   'info': {},
   'events': [{'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '10000000', 'class': 'Mandatory', 'paysFee': True}]}],
   'success': True},
  {'method': 'finalityTracker.finalHint',
   'signature': None,
   'nonce': '0',
   'args': ['2201988'],
   'tip': '0',
   'hash': '0x5eee0aa66e2cbbaa669e5f3dab105b28b7b1d107f859a3f2a8661cf08325e42d',
   'info': {},
   'events': [{'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '10000000', 'class': 'Mandatory', 'paysFee': True}]}],
   'success': True},
  {'method': 'parachains.setHeads',
   'signature': None,
   'nonce': '0',
   'args': [[]],
   'tip': '0',
   'hash': '0xcf52705d1ade64fc0b05859ac28358c0770a217dd76b75e586ae848c56ae810d',
   'info': {},
   'events': [{'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '1000000000',
       'class': 'Mandatory',
       'paysFee': True}]}],
   'success': True}]}

However, there are events from democracy enactment, which happened on_initialize. So, fetchBlock should also return fields like:

onInitialize: { events: [...] },
onFinalize: { events: [...] }

Block 2064961 doesn't parse and logs:

Internal Error: Error: Missing extrinsic 655360000 in block 0x4b8928d922b8940fe2f8a80d3491ab5992dd610d498dafe28445fa1ba7405f5b
    at ApiHandler.fetchBlock (/home/parity/projects/substrate-api-sidecar/build/ApiHandler.js:76:31)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async /home/parity/projects/substrate-api-sidecar/build/main.js:73:16
    at async /home/parity/projects/substrate-api-sidecar/build/main.js:42:50

Block 1574408 and older show the runtime upgrade events as part of timestamp. (Polkascan has the same error)

{'method': 'timestamp.set',
   'signature': None,
   'nonce': '0',
   'args': ['1584975294000'],
   'tip': '0',
   'hash': '0x60d118d378593d7a739890d7c26d55559cc3c50df8f3a4dce79404b07dbfb9be',
   'info': {},
   'events': [{'method': 'democracy.PreimageUsed',
     'data': ['0x8b98a7dfe17fbca031355eec884f238ae0d4139d7cd1a426580cedf317d181d2',
      'EqeaSHobbxoBkKXdfcxTJwYEsofyu48LMrSeFtiQGcafouM',
      '133635200000000']},
    {'method': 'system.CodeUpdated', 'data': []},
    {'method': 'democracy.Executed', 'data': ['42', True]},
    {'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '10000', 'class': 'Operational', 'paysFee': True}]}],
   'success': True}

Focus should be on fixing the two latest examples (2201991 and 2064961) and adding a field for initialize/finalize events in the block data object.

Endpoint for reward payment destination

Add an endpoint to query a staker by stash address and optional block number and get reward destination info (namely, the destination address and whether or not the funds will be locked). If no block number is provided, then it should return the finalized head.

The API should look like:

stakingPayoutInfo(stash: address, block: number) -> (
    block: number, 
    rewardDestination: rewardDestination, 
    controller: address
)

First, this will have to query staking.payee for the reward destination (enum of {Staked, Stash, Controller}). Then, it will query staking.bonded for the controller address.

Refactor main out into an App class and controller classes

Live reload development server

Currently, whenever a change is made to the code base, the developer must restart the server. In order to improve ergonomics a command like yarn run server:dev should be added that watches for changes to the code base and automatically recompiles + restarts.

Automate versioning

Investigate and implement automated versioning. A service like this may be useful.

Cannot read property 'extrinsicBaseWeight' of undefined

Internal Error: TypeError: Cannot read property 'extrinsicBaseWeight' of undefined

Endpoint for transaction metadata

This endpoint returns the runtime metadata to use for transaction construction. It should take a block number (ideally, hash, and the height is derived from that) as an optional parameter (if not provided, return the finalized head). The return should include the hash of the block along with its height.

Give useful Staking info

The staking endpoint should return a bunch of information that would inform a validator or nominator about whether or not (or when) they should make a transaction.

getStaking(blockId) -> {
  at: { blockNumber, hash },
  ValidatorCount: number, // of validators in the set
  ValidatorSet: Vec<AccountId>, // list of validators for this session
  ActiveEra: number, // ActiveEra.index
  ForceEra: enum, // status of era forcing
  NextEra: number, // block number or unix time of next era
  NextSession: number, // block number or unix time of next session
  UnappliedSlashes: Vec<UnnappliedSlash>, // array of upcoming slashes, keyed by era
  QueuedElected: Option<ElectionResult>, // contains an array of stashes elected
  ElectionStatus: { status, toggle }, // see below
}

The final item, ElectionStatus deserves particular discussion. Certain staking transactions that would affect the outcome of a validator are not allowed while the submission window for election results is open.

staking.EraElectionStatus returns either Closed or Open(BlockNumber) with the block number from which the window has been open.

Todo:

Look up how to predict the end of an era (see Polkadot JS Apps).
Look up when election opens and closes (IIRC, it's open for the last 1/16th of a session on Polkadot and 1/4 of a session on Kusama).

To return:

electionStatus:
  window: 'Open' | 'Closed',
  toggle: blockNumber

Where blockNumber is the future block when the window will change state. Note: as this is based on slots and not blocks, a toggle UnixTime might be more accurate than a block number.

Include block height with balance queries

The client should have an endpoint to query account balance. This endpoint should include the block hash and height at which the node performed the balance fetch;

ex: GET /balance/address1 -> {balance: 10, block_hash: 0x1234...4321, height: 100}

Give message when Sidecar is connected to a known public node

openAPI 3.0 yaml spec for v1

Create an openAPI specification .yaml document to declare the proposed v1 API.

Use OpenAPI specification Version 3.0.3

Return consistent number format in events

All numbers in events should have a consistent type (e.g. numeric or hex).

For example, block 1642150 events have:

"events": [
                {
                    "method": "treasury.Deposit",
                    "data": [
                        8000000000
                    ]
                },
                {
                    "method": "balances.Deposit",
                    "data": [
                        "FiHWU9AjN7z2no8zyiVEXtiTE46izxDgEcF1KM4r2r1GK59",
                        2000000000
                    ]
                },
                {
                    "method": "staking.Unbonded",
                    "data": [
                        "FChhpqk6Er57sXMC71F2aQ7EsNBQZTe3wUGXfKEvLH7W8zE",
                        "0x000000000000000000470de4df820000"
                    ]
                },
                {
                    "method": "system.ExtrinsicSuccess",
                    "data": [
                        {
                            "weight": 400000,
                            "class": "Normal",
                            "paysFee": true
                        }
                    ]
                }
            ],

Validate address prior to calling polkadot.js api

When an address with improper length or non-base58 character is passed as a param an error occurs while polkadot.js tries to createType(AccountId). This results in a 500 error.

This can be avoided by validating the address is properly base58 encoded and has the correct length. If the request does not have a valid address a 4xx error can be thrown to indicate malformed user input and the error message can specify exactly what validation failed.

Endpoint to submit pre-signed transactions

The endpoint should accept pre-signed transactions and inject them into the transaction queue.

If the node returns an error on transaction broadcast, there must be clear documentation describing if the transaction could ever land on-chain. For example, the node should return a successful status if a transaction that sends funds from an account with insufficient balance could be persisted in the node’s transaction queue and land on chain at some point in the future when the account has sufficient balance. When in doubt, return a successful status.

Endpoint for individual validator rewards

The Staking pallet only emits a single event regarding reward amount at the end of an era. Many users are interested in the rewards to each staker.

In a block with a Staking::Reward event, the RPC should be able to compute the reward to each staker based on:

Total validator set size
Total number of points issued in era
Total reward
Reward destinations (Controller vs. Stash)
- If Stash, flag for if adding to stake or not

Given an event, the RPC should return an array of all stakers rewarded and their corresponding reward amount.

Get block 0

Right now we are querying some fee params on parentHash, but of course it can't get them for block 0 because the parent is hardcoded 0x000...000. But there are no txs in block 0, so we should only do fee calculation if blockNumber > 0.

Add ESlint

I would like to advocate for adding ESLint to sidecar. With plans to add significant test coverage (and the necessary refactorizations) in the near future plus a growing, diverse group of contributors, it would behoove sidecar’s maintainers to add a linter. My proposal would be to closely mimic @amaurymartiny ESLint setup in txwrapper because 1) the repos are intended to be used together & 2) since @amaurymartiny is well experienced in JS work at Parity and frequently reviews this repo we could benefit from his consultation in config decisions/ justification.

Benefits would include

Static syntax checking
Style checking to help contributors conform to style guidelines
Hopefully encourage more idiomatic and safer code

I would personally be happy to do the setup of ESLint in the repo. Before starting though I would like to get feedback/ advice from stakeholders and frequent reviewers, including @amaurymartiny, @danforbes, @joepetrowski, & @maciejhirsz.

Show the user the expected config if the validation fails

I just added that in a project of mine and find it rather convenient so here it is :)

In the code, you probably want to benefit from the new GetEnv()method.

Here is how it can be used:

 if (!this.config.Validate()) {
      this.config.Print({ compact: false });
      Logger.Err('Invalid config, we expect something like:');
      Logger.Info(this.config.GenEnv().join('\n'));
      Logger.Err('You may copy the snipper above in a .env file');
      Logger.Err('Invalid config, exiting...');
      process.exit(1);
    } else {
      this.config.Print({ compact: true });
    }

So if the config is OK we show only the compact version. However, if the config is wrong, we show the extended version to help the user understand the issues AND we show how a default config would look like.

Bug: cannot decode events on Kusama - unable to reproduce though

I had sidecar connected to wss://kusama-rpc.polkadot.io/ and I was using the route /block/2577685 and /block/2578001. It was not able to decode any of the events properly, although it had the correct quantity of events. I think the issue was originally thrown in 'fetchEvents` because I was getting the text "Unable to fetch Events, cannot confirm extrinsic status. Check pruning settings on the node."

In order to fix the issue I quit running sidecar and restarted. Since then I have not been able to reproduce the issue.

Below is output from the terminal during the session I was getting these errors with sidecar.

Running on http://127.0.0.1:8080/
2020-06-02 11:02:31          API-WS: disconnected from wss://kusama-rpc.polkadot.io/ code: '1006' reason: 'Socket Error: read ETIMEDOUT'
Unable to decode storage system.events: createType(Vec<EventRecord>):: findMetaEvent: Unable to find Event with index 0x81ec/[129,236]
2020-06-02 11:10:51        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: createType(Vec<EventRecord>):: findMetaEvent: Unable to find Event with index 0x81ec/[129,236]
Unable to decode storage system.events: createType(Vec<EventRecord>):: Vec length 1054882983 exceeds 32768
2020-06-02 11:13:13        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: createType(Vec<EventRecord>):: Vec length 1054882983 exceeds 32768
Unable to decode storage system.events: createType(Vec<EventRecord>):: Vec length 1054882983 exceeds 32768
2020-06-02 11:14:40        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: createType(Vec<EventRecord>):: Vec length 1054882983 exceeds 32768

HTTP request logger

For debugging, it would be helpful to have a request logger. A common express middleware logger is morgan and it could be a good candidate. It can be added with app.use, right under this line app.use(bodyParser.json());

`vesting` in `fetchVesting` is an option that should be unwrapped

Currently, fetchingVesting returns a field vesting that is of type Option<VestingInfo>. In order to give more consistent responses and richer error messages, this option should be checked unwrapped or return an error if it is None.

Update `README.md` to have `wasm-pack` install instructions

This should be a companion PR to #50 that reflects the necessary steps to install wasm-pack prior to running yarn.

Correct lint errors introduced by #110

PR #110 introduced 6 ESLint errors (see below). In an effort to avoid the use of any for all code introduced after #61 was merged, these should be corrected to have proper types.

   30:7   error  Unsafe assignment of an any value                  @typescript-eslint/no-unsafe-assignment
   31:7   error  Unsafe assignment of an any value                  @typescript-eslint/no-unsafe-assignment
   32:7   error  Unsafe assignment of an any value                  @typescript-eslint/no-unsafe-assignment
   34:7   error  Unsafe assignment of an any value                  @typescript-eslint/no-unsafe-assignment
  507:36  error  Invalid type "any" of template literal expression  @typescript-eslint/restrict-template-expressions
  507:44  error  Invalid type "any" of template literal expression  @typescript-eslint/restrict-template-expressions

Update fees for refund and polynomial calculation

We need to update fee calculation for two changes:

First, some functions now return weight consumed and issue a refund if the weight consumed was less than that charged at the beginning. Update sidecar to return the actual fee paid in case of a refund. paritytech/substrate#6024

Second, weight calculation will be based on a polynomial, the coefficients of which will be constants in the runtime. Calculate the fee based on runtime constants, the fee multiplier, encoded length, and weight consumed. The logic will look roughly like:

// constants
Get byte fee
Get polynomial coefficients (integer, fraction, negative, degree)
Get extrinsic base weight

// per-block variables
Get FeeMultiplier (emits in `onInitialize.transactionPayment`)

// per-transaction
Get encoded length
Get weight consumed

fee = byteFee * encodedLength + FeeMultiplier * weightFee(weightConsumed)

Return staking ledger

Endpoint that accepts a controller address and returns info from staking.ledger.

Internal Error: TypeError: Cannot read property 'at' of undefined

When I request 'http://127.0.0.1:8080/balance/EEWyMLHgwtemr48spFNnS3U2XjaYswqAYAbadx2jr9ppp4X/1472950'

The api responses:
{"error":"Interal Error"}

The console log:

Internal Error: TypeError: Cannot read property 'at' of undefined
at ApiHandler.fetchBalance (/Users/chance/Documents/github/blockchain/substrate-api-sidecar/build/ApiHandler.js:102:44)
at process._tickCallback (internal/process/next_tick.js:68:7)

paritytech / substrate-api-sidecar Goto Github PK

substrate-api-sidecar's Introduction

@substrate/api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.

Prerequisites

<= v15.0.0

>= v16.0.0

Table of contents

NPM package installation and usage

Global installation

Local installation

Finishing up

Source code installation and usage

Quick install

Rust development installation

Running

Configuration

Express server

Substrate node

Metrics Server

Custom substrate types

Connecting a modified node template

Logging

Log levels

RPC logging

Prometheus server

Debugging fee and staking payout calculations

Available endpoints

Chain integration guide

Docker

Pull the latest release

Or build from source

Run

Contribute

Notes for maintainers

Commits

Updating polkadot-js dependencies

Maintenance Guide

Hardware requirements

Disk Space

Memory

Running sidecar and a node

Benchmarks

substrate-api-sidecar's People

Contributors

Stargazers

Watchers

Forkers

substrate-api-sidecar's Issues

Recommend Projects

Recommend Topics

Recommend Org