Code Monkey home page Code Monkey logo

spine-conductor's Introduction

About spine-conductor

It is a collection of release orchestration & automation scripts, and CI workflows to simplify Spine tool development tasks.

Installation & dependencies

You can install the tools in this repo as usual using pip. If you don't intend to develop the tool, you can use pipx, otherwise the manual installation might be prefered.

Using pipx

  • Installation:
    pipx install git+https://github.com/spine-tools/spine-conductor.git
  • Upgrade:
    pipx upgrade spine-conductor

Manual installation

Manual: git clone <repo> followed by pip install

git clone [email protected]:spine-tools/spine-conductor.git
# or: gh repo clone spine-tools/spine-conductor
cd spine-conductor
python -m venv --prompt conductor .venv
source venv/bin/activate # on Linux/MacOS, see below for Windows
pip install -U pip
pip install -e .

Alternatively, you could also try installing it with pipx from the local repo.

Additional note for Windows

To activate the virtual environment on Windows, depending on the command line shell you are using, you can use either of these:

$ source venv/Scripts/activate # on git-bash
$ .\venv\Scripts\activate.ps1 # on powershell

Updating

Go to the repo on the terminal, then:

git switch master
git pull
source venv/bin/activate
pip install -e .

Dependencies

  • The conduct release command uses Git behind the scenes. So will need to have the git available in the same shell.
  • To use the conduct publish command, you besides Git, you also need GitHub CLI (gh) installed and configured. Please see the upstream docs for details.
  • If you do not want to use gh, you will have to push the tags to GitHub manually, and trigger the publish workflow from GitHub (see below).

Usage

Create the release tags

$ cd /path/to/repo/Spine-Toolbox
$ conduct release --bump patch -c release.toml  # or include in pyproject.toml
Repository: /path/to/repo/Spine-Toolbox
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.6.19 @ 034fb4b
Repository: /path/to/repo/spine-items
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.20.1 @ 5848e25
Repository: /path/to/repo/spine-engine
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.22.1 @ e312db2
Repository: /path/to/repo/Spine-Database-API
## master...origin/master
Select the files to add (comma/space separated list):
Creating tag: 0.29.1 @ d9ed86e

Package Tags summary  ๐Ÿ’พ โžก 'pkgtags.json':
{
  "Spine-Toolbox": "0.6.19",
  "spine-items": "0.20.1",
  "spine-engine": "0.22.1",
  "Spine-Database-API": "0.29.1"
}

Release the packages to PyPI

$ conduct publish --pkgtags pkgtags.json -c release.toml

Alternatively, you could do this step manually as follows:

  1. push the tags to GitHub

    for repo in . /path/to/repo/{Spine-Database-API,spine-{items,engine}}; do
        pushd $repo;
        git push origin master --tags;
        popd
    done

    Or with Powershell on Windows:

    "." , "/path/to/repo/spinedb-api", "/path/to/repo/spine-items", "/path/to/repo/spine-engine" | % {
      pushd $_;
      git push origin master --tags;
      popd;
    }
  2. trigger the workflow

    $ cat pkgtags.json | gh workflow run --repo spine-tools/spine-conductor test-n-publish.yml --json

    You can also trigger the workflow from GitHub. You will have to add the tags manually in that case.

Configuration

All commands are configured using a TOML file. For spine-tools you can use the example shown below:

[tool.conductor]
packagename_regex = "spine(toolbox|(db){0,1}[_-][a-z]+)"  # package name on PyPI

[tool.conductor.dependency_graph]
spinetoolbox = ["spine_items", "spine_engine", "spinedb_api"]
spine_items  = ["spinetoolbox", "spine_engine", "spinedb_api"]
spine_engine = ["spinedb_api"]
spinedb_api  = []

[tool.conductor.repos]
spinetoolbox = "."
spine_items  = "venv/src/spine-items"
spine_engine = "venv/src/spine-engine"
spinedb_api  = "venv/src/spinedb-api"

# # default
# [tool.conductor.branches]
# spinetoolbox = "master"
# spine_items  = "master"
# spine_engine = "master"
# spinedb_api  = "master"

[tool.conductor.workflow]
repo = "spine-tools/spine-conductor"  # gh-org/repo hosting the workflow
file = "test-n-publish.yml"           # workflow file-name/name/id

It can be included in pyproject.toml, or kept in a separate config file. In the latter case, it needs to be provided with the -c/--config option when calling the release tagging script.

Publishing to PyPI

We use the Trusted Publishers feature for releasing to PyPI. First we have to add this repo as the source for releases. Following which, we can trigger the workflow manually or using GitHub CLI.

The release tagging script writes a file called pkgtags.json in the current directory. It lists the repos that were tagged, along with the tags. This can be fed to GitHub CLI to initiate publishing.

$ cat pkgtags.json | gh workflow run --repo spine-tools/spine-conductor test-n-publish.yml --json

The packages may also be publised by triggering the workflow manually from GitHub. In that case, the release tags for the different Spine repos have to be entered manually.

Screenshot of the GH UI to do manual dispatch

spine-conductor's People

Contributors

suvayu avatar soininen avatar pekkasavolainen avatar

Stargazers

 avatar

Watchers

 avatar Erkka Rinne avatar Jody Dillon avatar Manuel Marin avatar  avatar  avatar Maren Ihlemann avatar  avatar

Forkers

suvayu

spine-conductor's Issues

Support version pinning Spine packages

The logic for version pinning is more complex, as we have to check all packages and update the pyproject.toml even if there are no new commits since the last release.

Newlines broken in output pyproject.toml

I ran the conduct release command and it finished successfully. However, when pushing the modified pyproject.toml files to GitHub, I noticed that the Spine-Database-API GitHub actions failed because Python's toml library couldn't parse the files. In PyCharm, pyproject.toml shows extra new line on each line so I suppose conductor somehow messes those up. Manually replacing all newlines by with comes out from Return key in Windows fixes pyproject.toml.

Implement a command to push the git tags to GH and trigger the publish workflow

Now

Currently after the tags are created, you have to manually push the tags to github, and trigger the publishing workflow.

sh-like:

for repo in . venv/src/{spinedb-api,spine-{items,engine}}; do
    pushd $repo;
    git push origin master --tags;
    popd
done

pwsh:

"." , "venv/src/spinedb-api", "venv/src/spine-items", "venv/src/spine-engine" | % {
   pushd $_;
   git push origin master --tags;
  popd;
}

Followed by:

$ cat pkgtags.json | gh workflow run --repo spine-tools/spine-conductor test-n-publish.yml --json

Proposal

original

It is good to keep these steps separate from the step creating the tags. This gives the person making the release a chance to inspect the tags and make a deliberate decision to push. However the above steps maybe combined into a separate command, say publish that takes the json file as argument. Something like this:

$ conduct publish pkgtags.json -c release.toml

We could check for GH CLI. In case it isn't installed, we can print a message pointing to the documentation on how to publish it from GH workflow dispatch menu.

shlex does not work on windows

The shlex library offers, as per the Python documentation, lexical analyzers for simple syntaxes resembling that of the Unix shell. Windows shell does not resemble Unix shell. For example, shlex.split() removes path separators (\) from Windows paths which makes invoking the editor command fail.

Other ways should be found to deal with shell commands on Windows.

Traceback when trying to publish

I was trying to publish new version of Toolbox. While running

conduct publish --pkgtags pkgtags.json -c release.toml

and selecting the remote repository for Spine-Toolbox, I got this:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Traceback (most recent call last) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ spine-conductor\orchestra\__main__.py:69 in publish                   โ”‚
โ”‚                                                                                                  โ”‚
โ”‚    66 ):                                                                                         โ”‚
โ”‚    67 โ”‚   """Push Git tags to GitHub and publish packages to PyPI"""                             โ”‚
โ”‚    68 โ”‚   conf = read_conf(f"{config}")                                                          โ”‚
โ”‚ >  69 โ”‚   publish_tags_whls(conf, pkgtags)                                                       โ”‚
โ”‚    70                                                                                            โ”‚
โ”‚    71                                                                                            โ”‚
โ”‚    72 _xtest_doc_ref = (                                                                         โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚    conf = {                                                                                  โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'packagename_regex': 'spine(toolbox|(db){0,1}[_-][a-z]+)',                     โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'pkgname_re': re.compile('spine(toolbox|(db){0,1}[_-][a-z]+)'),                โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'repos': {'spinetoolbox': '.', 'spine_items': '../spine-items',                โ”‚ โ”‚
โ”‚ โ”‚           'spine_engine': '../spine-engine', 'spinedb_api': '../Spine-Database-API'},        โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'dependency_graph': {'spinetoolbox': ['spine_items', 'spine_engine',           โ”‚ โ”‚
โ”‚ โ”‚           'spinedb_api'], 'spine_items': ['spinetoolbox', 'spine_engine', 'spinedb_api'],    โ”‚ โ”‚
โ”‚ โ”‚           'spine_engine': ['spinedb_api'], 'spinedb_api': []},                               โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'default_branch': 'master',                                                    โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'branches': {                                                                  โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   โ”‚   'spinetoolbox': 'master',                                                  โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   โ”‚   'spine_items': 'master',                                                   โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   โ”‚   'spine_engine': 'master',                                                  โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   โ”‚   'spinedb_api': 'master'                                                    โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   },                                                                             โ”‚ โ”‚
โ”‚ โ”‚           โ”‚   'workflow': {'repo': 'spine-tools/spine-conductor', 'file':                    โ”‚ โ”‚
โ”‚ โ”‚           'test-n-publish.yml'}                                                              โ”‚ โ”‚
โ”‚ โ”‚           }                                                                                  โ”‚ โ”‚
โ”‚ โ”‚  config = WindowsPath('release.toml')                                                        โ”‚ โ”‚
โ”‚ โ”‚     ctx = <click.core.Context object at 0x000001D2ADAAAB90>                                  โ”‚ โ”‚
โ”‚ โ”‚ pkgtags = WindowsPath('pkgtags.json')                                                        โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ spine-conductor\orchestra\publish.py:95 in publish_tags_whls          โ”‚
โ”‚                                                                                                  โ”‚
โ”‚    92 โ”‚   for _, repo_path in config["repos"].items():                                           โ”‚
โ”‚    93 โ”‚   โ”‚   repo = Repo(repo_path)                                                             โ”‚
โ”‚    94 โ”‚   โ”‚   push_tags(repo, tags[remote_name(repo)])                                           โ”‚
โ”‚ >  95 โ”‚   res = dispatch_workflow(pkgtags)                                                       โ”‚
โ”‚    96 โ”‚   if isinstance(res, subprocess.CalledProcessError):                                     โ”‚
โ”‚    97 โ”‚   โ”‚   console.print(res.stderr.decode())                                                 โ”‚
โ”‚    98 โ”‚   โ”‚   return                                                                             โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚         _ = 'spinedb_api'                                                                    โ”‚ โ”‚
โ”‚ โ”‚    config = {                                                                                โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'packagename_regex': 'spine(toolbox|(db){0,1}[_-][a-z]+)',                   โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'pkgname_re': re.compile('spine(toolbox|(db){0,1}[_-][a-z]+)'),              โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'repos': {'spinetoolbox': '.', 'spine_items': '../spine-items',              โ”‚ โ”‚
โ”‚ โ”‚             'spine_engine': '../spine-engine', 'spinedb_api': '../Spine-Database-API'},      โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'dependency_graph': {'spinetoolbox': ['spine_items', 'spine_engine',         โ”‚ โ”‚
โ”‚ โ”‚             'spinedb_api'], 'spine_items': ['spinetoolbox', 'spine_engine', 'spinedb_api'],  โ”‚ โ”‚
โ”‚ โ”‚             'spine_engine': ['spinedb_api'], 'spinedb_api': []},                             โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'default_branch': 'master',                                                  โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'branches': {                                                                โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   โ”‚   'spinetoolbox': 'master',                                                โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   โ”‚   'spine_items': 'master',                                                 โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   โ”‚   'spine_engine': 'master',                                                โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   โ”‚   'spinedb_api': 'master'                                                  โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   },                                                                           โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'workflow': {'repo': 'spine-tools/spine-conductor', 'file':                  โ”‚ โ”‚
โ”‚ โ”‚             'test-n-publish.yml'}                                                            โ”‚ โ”‚
โ”‚ โ”‚             }                                                                                โ”‚ โ”‚
โ”‚ โ”‚   pkgtags = WindowsPath('pkgtags.json')                                                      โ”‚ โ”‚
โ”‚ โ”‚      repo = <git.repo.base.Repo 'Spine-Database-API\\.git'>   โ”‚ โ”‚
โ”‚ โ”‚ repo_path = '../Spine-Database-API'                                                          โ”‚ โ”‚
โ”‚ โ”‚      tags = {                                                                                โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'Spine-Toolbox': '0.7.2',                                                    โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'spine-items': '0.21.3',                                                     โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'spine-engine': '0.23.2',                                                    โ”‚ โ”‚
โ”‚ โ”‚             โ”‚   'Spine-Database-API': '0.30.3'                                               โ”‚ โ”‚
โ”‚ โ”‚             }                                                                                โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ spine-conductor\orchestra\publish.py:65 in dispatch_workflow          โ”‚
โ”‚                                                                                                  โ”‚
โ”‚    62 โ”‚   workflow = CONF["workflow"]["file"]                                                    โ”‚
โ”‚    63 โ”‚   CMD = CMD_FMT.format(repo=ghrepo, workflow=workflow)                                   โ”‚
โ”‚    64 โ”‚   try:                                                                                   โ”‚
โ”‚ >  65 โ”‚   โ”‚   res = subprocess.run(                                                              โ”‚
โ”‚    66 โ”‚   โ”‚   โ”‚   shlex.split(CMD),                                                              โ”‚
โ”‚    67 โ”‚   โ”‚   โ”‚   input=pkgtags_json.read_bytes(),                                               โ”‚
โ”‚    68 โ”‚   โ”‚   โ”‚   check=True,                                                                    โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚            _ = None                                                                          โ”‚ โ”‚
โ”‚ โ”‚          CMD = 'gh workflow run --json --repo spine-tools/spine-conductor                    โ”‚ โ”‚
โ”‚ โ”‚                test-n-publish.yml'                                                           โ”‚ โ”‚
โ”‚ โ”‚       ghrepo = 'spine-tools/spine-conductor'                                                 โ”‚ โ”‚
โ”‚ โ”‚       kwargs = {}                                                                            โ”‚ โ”‚
โ”‚ โ”‚ pkgtags_json = WindowsPath('pkgtags.json')                                                   โ”‚ โ”‚
โ”‚ โ”‚     workflow = 'test-n-publish.yml'                                                          โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ subprocess.py:548 in run          โ”‚
โ”‚                                                                                                  โ”‚
โ”‚    545 โ”‚   โ”‚   kwargs['stdout'] = PIPE                                                           โ”‚
โ”‚    546 โ”‚   โ”‚   kwargs['stderr'] = PIPE                                                           โ”‚
โ”‚    547 โ”‚                                                                                         โ”‚
โ”‚ >  548 โ”‚   with Popen(*popenargs, **kwargs) as process:                                          โ”‚
โ”‚    549 โ”‚   โ”‚   try:                                                                              โ”‚
โ”‚    550 โ”‚   โ”‚   โ”‚   stdout, stderr = process.communicate(input, timeout=timeout)                  โ”‚
โ”‚    551 โ”‚   โ”‚   except TimeoutExpired as exc:                                                     โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚ capture_output = True                                                                        โ”‚ โ”‚
โ”‚ โ”‚          check = True                                                                        โ”‚ โ”‚
โ”‚ โ”‚          input = b'{\r\n    "Spine-Toolbox": "0.7.2",\r\n    "spine-items": "0.21.3",\r\n    โ”‚ โ”‚
โ”‚ โ”‚                  "spine-engin'+52                                                            โ”‚ โ”‚
โ”‚ โ”‚         kwargs = {'stdin': -1, 'stdout': -1, 'stderr': -1}                                   โ”‚ โ”‚
โ”‚ โ”‚      popenargs = (                                                                           โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   [                                                                       โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   'gh',                                                               โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   'workflow',                                                         โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   'run',                                                              โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   '--json',                                                           โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   '--repo',                                                           โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   'spine-tools/spine-conductor',                                      โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   โ”‚   'test-n-publish.yml'                                                โ”‚ โ”‚
โ”‚ โ”‚                  โ”‚   ],                                                                      โ”‚ โ”‚
โ”‚ โ”‚                  )                                                                           โ”‚ โ”‚
โ”‚ โ”‚        timeout = None                                                                        โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ subprocess.py:1026 in __init__    โ”‚
โ”‚                                                                                                  โ”‚
โ”‚   1023 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   self.stderr = io.TextIOWrapper(self.stderr,                           โ”‚
โ”‚   1024 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   encoding=encoding, errors=errors)                             โ”‚
โ”‚   1025 โ”‚   โ”‚   โ”‚                                                                                 โ”‚
โ”‚ > 1026 โ”‚   โ”‚   โ”‚   self._execute_child(args, executable, preexec_fn, close_fds,                  โ”‚
โ”‚   1027 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   pass_fds, cwd, env,                                       โ”‚
โ”‚   1028 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   startupinfo, creationflags, shell,                        โ”‚
โ”‚   1029 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   p2cread, p2cwrite,                                        โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚               args = [                                                                       โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   'gh',                                                               โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   'workflow',                                                         โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   'run',                                                              โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   '--json',                                                           โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   '--repo',                                                           โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   'spine-tools/spine-conductor',                                      โ”‚ โ”‚
โ”‚ โ”‚                      โ”‚   'test-n-publish.yml'                                                โ”‚ โ”‚
โ”‚ โ”‚                      ]                                                                       โ”‚ โ”‚
โ”‚ โ”‚            bufsize = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚            c2pread = 4                                                                       โ”‚ โ”‚
โ”‚ โ”‚           c2pwrite = Handle(956)                                                             โ”‚ โ”‚
โ”‚ โ”‚          close_fds = True                                                                    โ”‚ โ”‚
โ”‚ โ”‚      creationflags = 0                                                                       โ”‚ โ”‚
โ”‚ โ”‚                cwd = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚           encoding = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚                env = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚             errors = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚            errread = 5                                                                       โ”‚ โ”‚
โ”‚ โ”‚           errwrite = Handle(916)                                                             โ”‚ โ”‚
โ”‚ โ”‚         executable = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚       extra_groups = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚                  f = <_io.BufferedReader name=5>                                             โ”‚ โ”‚
โ”‚ โ”‚                gid = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚               gids = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚              group = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚            p2cread = Handle(948)                                                             โ”‚ โ”‚
โ”‚ โ”‚           p2cwrite = 3                                                                       โ”‚ โ”‚
โ”‚ โ”‚           pass_fds = ()                                                                      โ”‚ โ”‚
โ”‚ โ”‚           pipesize = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚         preexec_fn = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚      process_group = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚    restore_signals = True                                                                    โ”‚ โ”‚
โ”‚ โ”‚               self = <Popen: returncode: None args: ['gh', 'workflow', 'run', '--json',      โ”‚ โ”‚
โ”‚ โ”‚                      '--repo',...>                                                           โ”‚ โ”‚
โ”‚ โ”‚              shell = False                                                                   โ”‚ โ”‚
โ”‚ โ”‚  start_new_session = False                                                                   โ”‚ โ”‚
โ”‚ โ”‚        startupinfo = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚             stderr = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚              stdin = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚             stdout = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚               text = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚                uid = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚              umask = -1                                                                      โ”‚ โ”‚
โ”‚ โ”‚ universal_newlines = None                                                                    โ”‚ โ”‚
โ”‚ โ”‚               user = None                                                                    โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ subprocess.py:1538 in             โ”‚
โ”‚ _execute_child                                                                                   โ”‚
โ”‚                                                                                                  โ”‚
โ”‚   1535 โ”‚   โ”‚   โ”‚                                                                                 โ”‚
โ”‚   1536 โ”‚   โ”‚   โ”‚   # Start the process                                                           โ”‚
โ”‚   1537 โ”‚   โ”‚   โ”‚   try:                                                                          โ”‚
โ”‚ > 1538 โ”‚   โ”‚   โ”‚   โ”‚   hp, ht, pid, tid = _winapi.CreateProcess(executable, args,                โ”‚
โ”‚   1539 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚    # no special security                            โ”‚
โ”‚   1540 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚    None, None,                                      โ”‚
โ”‚   1541 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   โ”‚    int(not close_fds),                              โ”‚
โ”‚                                                                                                  โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚ โ”‚                     args = 'gh workflow run --json --repo spine-tools/spine-conductor        โ”‚ โ”‚
โ”‚ โ”‚                            test-n-publish.yml'                                               โ”‚ โ”‚
โ”‚ โ”‚           attribute_list = {'handle_list': [916, 956, 948]}                                  โ”‚ โ”‚
โ”‚ โ”‚                  c2pread = 4                                                                 โ”‚ โ”‚
โ”‚ โ”‚                 c2pwrite = Handle(956)                                                       โ”‚ โ”‚
โ”‚ โ”‚                close_fds = False                                                             โ”‚ โ”‚
โ”‚ โ”‚            creationflags = 0                                                                 โ”‚ โ”‚
โ”‚ โ”‚                      cwd = None                                                              โ”‚ โ”‚
โ”‚ โ”‚                      env = None                                                              โ”‚ โ”‚
โ”‚ โ”‚                  errread = 5                                                                 โ”‚ โ”‚
โ”‚ โ”‚                 errwrite = Handle(916)                                                       โ”‚ โ”‚
โ”‚ โ”‚               executable = None                                                              โ”‚ โ”‚
โ”‚ โ”‚              handle_list = [916, 956, 948]                                                   โ”‚ โ”‚
โ”‚ โ”‚         have_handle_list = False                                                             โ”‚ โ”‚
โ”‚ โ”‚                  p2cread = Handle(948)                                                       โ”‚ โ”‚
โ”‚ โ”‚                 p2cwrite = 3                                                                 โ”‚ โ”‚
โ”‚ โ”‚                 pass_fds = ()                                                                โ”‚ โ”‚
โ”‚ โ”‚               preexec_fn = None                                                              โ”‚ โ”‚
โ”‚ โ”‚                     self = <Popen: returncode: None args: ['gh', 'workflow', 'run',          โ”‚ โ”‚
โ”‚ โ”‚                            '--json', '--repo',...>                                           โ”‚ โ”‚
โ”‚ โ”‚                    shell = False                                                             โ”‚ โ”‚
โ”‚ โ”‚              startupinfo = <subprocess.STARTUPINFO object at 0x000001D2AE581A10>             โ”‚ โ”‚
โ”‚ โ”‚               unused_gid = None                                                              โ”‚ โ”‚
โ”‚ โ”‚              unused_gids = None                                                              โ”‚ โ”‚
โ”‚ โ”‚     unused_process_group = -1                                                                โ”‚ โ”‚
โ”‚ โ”‚   unused_restore_signals = True                                                              โ”‚ โ”‚
โ”‚ โ”‚ unused_start_new_session = False                                                             โ”‚ โ”‚
โ”‚ โ”‚               unused_uid = None                                                              โ”‚ โ”‚
โ”‚ โ”‚             unused_umask = -1                                                                โ”‚ โ”‚
โ”‚ โ”‚          use_std_handles = True                                                              โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
FileNotFoundError: [WinError 2] The system cannot find the file specified

My release.toml:

[tool.conductor]
packagename_regex = "spine(toolbox|(db){0,1}[_-][a-z]+)"  # package name on PyPI

[tool.conductor.dependency_graph]
spinetoolbox = ["spine_items", "spine_engine", "spinedb_api"]
spine_items  = ["spinetoolbox", "spine_engine", "spinedb_api"]
spine_engine = ["spinedb_api"]
spinedb_api  = []

[tool.conductor.repos]
spinetoolbox = "."
spine_items  = "../spine-items"
spine_engine = "../spine-engine"
spinedb_api  = "../Spine-Database-API"

[tool.conductor.workflow]
repo = "spine-tools/spine-conductor"  # gh-org/repo hosting the workflow
file = "test-n-publish.yml"           # workflow file-name/name/id

Publish action to process a single repository

I ran into some problems with the Publish GitHub action. Publishing spinedb_api, spine_engine and spine_items went well, but spinetoolbox got into a limbo. For some reason, the build step produced a wrong version (0.8.0-xxx instead of 0.7.1) which failed (luckily!) to get uploaded to PyPI. This was probably because the version update commit was not successfully pushed to GitHub, see #16. In the end I built and uploaded the package manually, so no problem there. However, it would be nice if it was possible to retry the publish process for a single repository at a time, e.g. Spine-Toolbox in this case.

Rollback on failure

Try to rollback to original state if creating a release w/ conduct release fails.

Possible reasons:

  • unanticipated failure,
  • user error, or
  • bug.

Traceback when including a repo that doesn't need to be published in tool.conductor.repos

I had this situation in my hands:

spinetoolbox - commits since publication
spine_items - commits since publication
spine_engine - HEAD at latest version tag
spinedb_api = ,commits since publication

I tried to use my release.toml without modifications but when running conduct release --bump patch -c release.toml, I get a Traceback when selecting files to commit for spine_engine. The gist is, expectedly, fatal: tag '0.23.3' already exists.

Instead of Traceback, I wish conductor would notice that spine_engine does not need to be published this time and would continue as if nothing happened. Perhaps other option would be to reset changes to the other repositories.

Make users notice errors in publishing process

I had some issues when trying to make a Toolbox release using the Conductor. I think it all started with the Git push to Spine-Toolbox repo failing due to some other commits being pushed there. I did not observe the output of Conductor careful enough to spot that this happened. Perhaps Conductor should bail out early if git push fails? At least the documentation could remind that checking command success is a must.

Document conventions and terminology

  • what is a package name? PyPI package name
  • how is package version handled? git tag is source of truth, but existing code relies on the old version, so there's a small conversion from the git tag based version to the legacy version. we can phase it out gradually
  • others?

Run test suite before publishing

Currently the testing step on the built wheel is disabled. Figure out how to ensure the test suite runs on the wheels instead of the repo.

EDIT: partially implemented in 15aa214

Document partial release

Describe the changes in the workflow in the case of a partial release, i.e. only some of the Spine packages have a new git tag.

Current docs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.