Code Monkey home page Code Monkey logo

srtool-actions's Introduction

SRTOOL Github Action

badge

Introduction

srtool is a docker image that allows building Substrate WASM Runtimes in a deterministic manner. It ensures that all srtool users (and tooling), using the same/latest tag, will produce 100% exactly the same runtime. It enables further trustless verifications.

srtool gh action 256px

This repository contains a custom Github Action that will help you integrate srtool in your Github CI.

Gitlab users are not left behind and can use the 2 first options mentioned below.

Srtool helpers

There are now several ways to use srtool in your project:

  • using srtool via alias: powerful but very verbose and prone to errors. This is for developers. This option is being deprecated and not recommended.

  • using the srtool-cli: much easier to use and removes a whole bunch of potential user’s mistakes. This is for developers.

  • using the srtool-app: the easiest option of all thanks to its GUI. This is good for non-developers.

  • using the Github actions from this repo. This is for your automatic CI.

Requirements

In order to use this Github Action, you will need a Github repository with your Substrate based project. You can learn how to get started with a first Github workflow here.

Before you setup your new workflow, you should gather the following information:

  • name of your chain: ie. polkadot

  • runtime package: ie. runtime-polkadot

  • location of your runtime: ie. runtime/polkadot

If your project uses standard values (such as your runtime package being named xyz-runtime if your chain is xyz), you will be able to skip many of the inputs. If you have a different setup, you can override the defaults to make it work for you.

Sample workflows

Make sure you store the yml files shown below in your repository under .github/workflows.

basic

basic.yml

name: Srtool build

on: push

jobs:
  srtool:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        chain: ["asset-hub-kusama", "asset-hub-westend"]
    steps:
      - uses: actions/checkout@v3
      - name: Srtool build
        id: srtool_build
        uses: chevdor/[email protected]
        with:
          chain: ${{ matrix.chain }}
          runtime_dir: polkadot-parachains/${{ matrix.chain }}-runtime
      - name: Summary
        run: |
          echo '${{ steps.srtool_build.outputs.json }}' | jq . > ${{ matrix.chain }}-srtool-digest.json
          cat ${{ matrix.chain }}-srtool-digest.json
          echo "Runtime location: ${{ steps.srtool_build.outputs.wasm }}"

Report

name: Srtool build

on: push

jobs:
  srtool:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        chain: ["westend"]
    steps:
      - uses: actions/checkout@v3
      - name: Srtool build
        id: srtool_build
        uses: chevdor/[email protected]
        with:
          chain: ${{ matrix.chain }}
      - name: Summary
        run: |
          echo Summary:
          echo - version: ${{ steps.srtool_build.outputs.version }}
          echo - info: ${{ steps.srtool_build.outputs.info }}
          echo - prop: ${{ steps.srtool_build.outputs.proposal_hash }}
          echo - json: ${{ steps.srtool_build.outputs.json }}

Artifacts

name: Srtool build

on: push

jobs:
  srtool:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        chain: ["asset-hub-kusama", "asset-hub-westend"]
    steps:
      - uses: actions/checkout@v3
      - name: Srtool build
        id: srtool_build
        uses: chevdor/[email protected]
        with:
          chain: ${{ matrix.chain }}
          runtime_dir: polkadot-parachains/${{ matrix.chain }}-runtime
      - name: Summary
        run: |
          echo '${{ steps.srtool_build.outputs.json }}' | jq . > ${{ matrix.chain }}-srtool-digest.json
          cat ${{ matrix.chain }}-srtool-digest.json
          echo "Runtime location: ${{ steps.srtool_build.outputs.wasm }}"
      - name: Archive Runtime
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.chain }}-runtime-${{ github.sha }}
          path: |
            ${{ steps.srtool_build.outputs.wasm }}
            ${{ matrix.chain }}-srtool-digest.json

Environmental variables and BUILD_OPTS

name: Srtool build

on: push

jobs:
  srtool:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        chain: ["asset-hub-kusama", "asset-hub-westend"]
    steps:
      - uses: actions/checkout@v3
      - name: Srtool build
        id: srtool_build
        uses: chevdor/[email protected]
        env:
          # optional: will be passed to docker srtool run cmd
          BUILD_OPTS: "--features on-chain-release-build"
        with:
          chain: ${{ matrix.chain }}
          runtime_dir: polkadot-parachains/${{ matrix.chain }}-runtime
      - name: Summary
        run: |
          echo '${{ steps.srtool_build.outputs.json }}' | jq . > ${{ matrix.chain }}-srtool-digest.json
          cat ${{ matrix.chain }}-srtool-digest.json
          echo "Runtime location: ${{ steps.srtool_build.outputs.wasm }}"

Environmental variables and parachain overrides

name: Srtool build

on: push

jobs:
  srtool:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        chain: ["asset-hub-kusama", "asset-hub-westend"]
    steps:
      - uses: actions/checkout@v3
      - name: Srtool build
        id: srtool_build
        uses: chevdor/[email protected]
        env:
          # optional: will override the parachain pallet ID and authorize_upgrade call ID,
          #           which will result in a different parachain_authorize_upgrade_hash
          # the hex values must be quoted
          PARACHAIN_PALLET_ID: "0x1e"
          AUTHORIZE_UPGRADE_PREFIX: "0x02"
        with:
          chain: ${{ matrix.chain }}
          runtime_dir: polkadot-parachains/${{ matrix.chain }}-runtime
      - name: Summary
        run: |
          echo '${{ steps.srtool_build.outputs.json }}' | jq . > ${{ matrix.chain }}-srtool-digest.json
          cat ${{ matrix.chain }}-srtool-digest.json
          echo "Runtime location: ${{ steps.srtool_build.outputs.wasm }}"

Parachain overrides

Similar to subwasm, the parachain pallet ID and the authorize_upgrade call ID can be overriden by PARACHAIN_PALLET_ID and AUTHORIZE_UPGRADE_PREFIX environmental variables, respectively. It will affect the generated proposal hash parachain_authorize_upgrade_hash.

If unset, the two envs will have the following default values:

  • PARACHAIN_PALLET_ID: 0x01

  • AUTHORIZE_UPGRADE_PREFIX: 0x02

Contributing

This project is using asciidoc for its documentation. You should not edit any mardown file (.md) as your changes would be dismissed.

Instead you should be editing the .adoc file and the markdown files will be generated. If you create a PR and do not have any of the required tooling, feel free to only edit the .adoc. files and I will generate the new .md.

Dev notes

Tooling: act

Due to a bug in act, the defaults defined in the action are not applied. That means must pass all the inputs while testing with act.

You can test locally using act:

act -W tests

To make it easier, you can also use just:

just test <test_name>

For instance:

# Run the default dev workflow: tests/dev.yml
just test

# Run the ipfs test workflow
just test ipfs

srtool-actions's People

Contributors

chevdor avatar ermalkaleci avatar fewensa avatar joshorndorff avatar kailai-wang avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

s3krit joshorndorff crystalin acalanetwork tidelabs gilescope 0xbillw kailai-wang interlay fewensa

srtool-actions's Issues

Permissions failed

Hello :)
I updated from v0.4.0 to v0.5.0 (and srtool tag from 1.57.0 to 1.62.0) and the CI is failing with:

Run echo ::group::Pulling the srtool docker image: paritytech/srtool:1.62.0
Pulling the srtool docker image: paritytech/srtool:1.62.0
Run echo ::group::Srtool version
Srtool version
Run echo ::group::srtool info
srtool info
Run echo ::group::Srtool build of chain moonbeam
  echo ::group::Srtool build of chain moonbeam
  CMD="docker run -i --rm -e PACKAGE=moonbeam-runtime -e CARGO_NET_GIT_FETCH_WITH_CLI=true -v /home/$USER/srtool/.ssh:/root/.ssh -v /home/$USER/srtool/entrypoint.sh:/srtool/entrypoint.sh --entrypoint /srtool/entrypoint.sh -e RUNTIME_DIR=runtime/moonbeam -e BUILD_OPTS -e PARACHAIN_PALLET_ID -e AUTHORIZE_UPGRADE_PREFIX -v /home/gh-actions/runner/_work/moonbeam-private/moonbeam-private:/build paritytech/srtool:1.62.0 build --app --json -cM"
  echo ::debug::build::docker_run $CMD
  
  # here we keep streaming the progress and fetch the last line for the json result
  stdbuf -oL $CMD | {
    while IFS= read -r line
    do
      echo ║ $line
      JSON="$line"
    done
    echo ::set-output name=json::$JSON
    echo $JSON | jq .
  
    PROP=`echo $JSON | jq -r .runtimes.compact.prop`
    echo ::set-output name=proposal_hash::$PROP
  
    WASM=`echo $JSON | jq -r .runtimes.compact.wasm`
    echo ::set-output name=wasm::$WASM
  
    Z_WASM=`echo $JSON | jq -r .runtimes.compressed.wasm`
    echo ::set-output name=wasm_compressed::$Z_WASM
  
    IPFS=`echo $JSON | jq -r .runtimes.compact.ipfs`
    echo ::set-output name=ipfs::$IPFS
  }
  
  echo ::endgroup
  shell: /bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    TMSP: 20220708_175500
    RUSTC_VERSION: 1.62.0
    SRTOOL_VERSION: 0.9.21
    SRTOOL_LATEST: 1.62.0
    SRTOOL_TAG: 1.62.0
    SRTOOL_IMAGE: paritytech/srtool:1.62.0
    WORKDIR: /home/gh-actions/runner/_work/moonbeam-private/moonbeam-private
    RUNTIME_DIR: runtime/moonbeam
    PACKAGE: moonbeam-runtime -e CARGO_NET_GIT_FETCH_WITH_CLI=true -v /home/$USER/srtool/.ssh:/root/.ssh -v /home/$USER/srtool/entrypoint.sh:/srtool/entrypoint.sh --entrypoint /srtool/entrypoint.sh
    BUILD_OPTS: 
    PARACHAIN_PALLET_ID: 0x01
    AUTHORIZE_UPGRADE_PREFIX: 0x03
Srtool build of chain moonbeam
  ║ 2022-07-08 17:55:36 - Update APT
  ║ Reading package lists...
  E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
  E: Unable to lock directory /var/lib/apt/lists/
  parse error: Invalid numeric literal at line 1, column 8
  Error: Process completed with exit code 4.

need rustc upgrade for polkadot-v1.0.0 para builds

We currently get this error for srtool build in CI with GHA:

  ║ error: `<C as TrieCacheProvider<H>>::Cache<'_>` does not live long enough
728
  ║ --> /home/builder/cargo/git/checkouts/substrate-7e08433d4c370a21/40e3395/primitives/state-machine/src/trie_backend_essence.rs:305:38
729
  ║ |
730
  ║ 305 | let cache = cache.as_mut().map(|c| c as _);
731
  ║ | ^

pretty sure this will work if this action bumps rustc to rustc 1.71.0-nightly (9d871b061 2023-05-21)

Possibility to override parachain pallet ID

Hi there, I'd like to ask if there's a way to override the parachain pallet ID and authorize_upgrade index in this GHA.

I noticed there's PARACHAIN_PALLET_ID and AUTHORIZE_UPGRADE_PREFIX for subwasm and was wondering if there's a similar thing for srtool-actions. Thanks in advance!

update the docker image to LLVM 10

We have a substrate-runtime dependency that needs LLVM 10. It would be great, if the LLVM version is updated in the docker image.

As rustc 1.53 requires LLVM >=10, this will be a necessary step anyhow.

Is there a way to enable a specific rust compilation feature ?

Hello,

we would like to implement the on-chain-release-build feature for the moonbeam runtime, but I see in the srtool code that this feature is only added for polkadot and kusama.

We use this gh action to release our on-chain runtime, is it possible via this github action to build the runtime by activating this rust compilation feature ?

The RUNTIME_DIR 'runtime' does not look like a Cargo project with version 0.9.0 or later

Switching from v0.8.0 to 0.9.0 or later seems to fail with:

Srtool build of chain creditcoin
  ║ 🧰 Substrate Runtime Toolbox - srtool v0.12.0 🧰
  ║ - by Chevdor -
  info: using existing install for '1.73.0-x86_64-unknown-linux-gnu'
  info: override toolchain for '/build' set to '1.73.0-x86_64-unknown-linux-gnu'
  ║
  ║ 1.73.0-x86_64-unknown-linux-gnu unchanged - rustc 1.73.0 (cc66ad468 2023-10-03)
  ║
  ║ 🏗 Building creditcoin3-runtime as release using rustc 1.73.0 (cc66ad468 2023-10-03)
  ║ ⏳ That can take a little while, be patient... subsequent builds may be faster.
  ║ Since you have to wait a little, you may want to learn more about Substrate runtimes:
  ║ https://docs.substrate.io/learn/architecture/
  toml: No such file or directory (os error 2)
  ║ !!! The RUNTIME_DIR 'runtime' does not look like a Cargo project. Is it pointing to the folder of your runtime crate?
  parse error: Invalid numeric literal at line 1, column 4

Works fine with version 0.8.0.

Full logs at:
https://github.com/gluwa/creditcoin3/actions/runs/7715917088/job/21031526965?pr=195

Latest PR switching from 0.8.0 to 0.9.2 -> gluwa/creditcoin3#195 (there are closed PRs in the same repo trying to upgrade to 0.9.0 and 0.9.1 but logs have probably expired).

Passing empty envvar to the build?

srtool-actions/action.yml

Line 182 in 7091873

CMD="docker run -i --rm -e PACKAGE=${{ env.PACKAGE }} -e RUNTIME_DIR=${{ env.RUNTIME_DIR }} -e BUILD_OPTS -e PARACHAIN_PALLET_ID -e AUTHORIZE_UPGRADE_PREFIX -v ${{ env.WORKDIR }}:/build ${{ env.SRTOOL_IMAGE }} build --app --json -cM" 

-e BUILD_OPTS -e PARACHAIN_PALLET_ID -e AUTHORIZE_UPGRADE_PREFIX

why?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.