floatingpurr / sync_with_poetry Goto Github PK

Sync .pre-commit-config.yaml repos starting from poetry.lock

License: MIT License

Python 100.00%

pre-commit pre-commit-hook python package-manger poetry poetry-python

sync_with_poetry's Introduction

Sync with poetry

A .pre-commit hook for keeping in sync the repos rev in .pre-commit-config.yaml with the packages version locked into poetry.lock. Check out pre-commit.com for more about the main framework.

Do you rely on PDM? See this equivalent sync repo: sync_with_pdm.

What problem does this hook help us solve?

When it comes to Python dependency management, Poetry is one of the modern solutions to handle project dependencies. Sometimes, you might want to install dev dependencies locally (e.g., black, flake8, isort, mypy, ...) to make your IDE (e.g., VS Code) play nicely with dev packages. This approach usually turns on a live feedback as you code (e.g., suggestions, linting, formatting, errors highlighting). Poetry does not differentiates anymore between dev and production packages inside poetry.lock. Now, this info is managed by dependency groups in pyproject.toml (see #26 for more).

This hook updates the rev of each repo in .pre-commit-config.yaml with the corresponding package version stored in poetry.lock.

E.g., starting from the following files:

# poetry.lock
[[package]]
name = "black"
version = "21.12b0"
description = "The uncompromising code formatter."
optional = false
python-versions = ">=3.6.2"

# .pre-commit-config.yaml
repos:
  # black - formatting
  - repo: https://github.com/psf/black
    rev: 21.11b1
    hooks:
      - id: black

this hook will bump black in .pre-commit-config.yaml as follows:

# .pre-commit-config.yaml
repos:
  # black - formatting
  - repo: https://github.com/psf/black
    rev: 21.12b0
    hooks:
      - id: black

Usage

Excerpt from a .pre-commit-config.yaml using an example of this hook:

- repo: https://github.com/floatingpurr/sync_with_poetry
  rev: "" # the revision or tag to clone at
  hooks:
    - id: sync_with_poetry
      args: [] # optional args

Args

  --skip [SKIP ...]  Packages to skip
  --config CONFIG    Path to a custom .pre-commit-config.yaml file
  --db PACKAGE_LIST  Path to a custom package list (json)
  --allow-frozen     Trust `frozen: xxx` comments for frozen revisions.

Usually this hook uses only dev packages to sync the hooks. Pass --all, if you want to scan also the main project packages.

Pass --skip <package_1> <package_2> ... to disable the automatic synchronization of the repos such packages correspond to.

Pass --config <config_file> to point to an alternative config file (it defaults to .pre-commit-config.yaml).

Pass --db <package_list_file> to point to an alternative package list (json). Such a file overrides the mapping in db.py.

Pass --allow-frozen if you want to use frozen revisions in your config. Without this option SWP will replace frozen revisions with the tag name taken from poetry.lock even if the frozen revision specifies the same commit as the tag. This options relies on frozen: xxx comments appended to the line of the frozen revision where xxx will be the tag name corresponding to the commit hash used. If the comment specifies the same revision as the lock file nothing is changed. Otherwise the revision is replaced with the expected revision tag and the frozen: xxx comment is removed.

Supported packages

Supported packages out-of-the-box are listed in db.py:

autopep8
bandit
black
commitizen
flake8
flakeheaven
isort
mypy
pyupgrade

You can create your very own package list, passing a custom json file with the arg --db. Such a file specifies how to map a package to the corresponding repo, following this pattern:

{
  "<package_name_in_PyPI>": {
    "repo": "<repo_url_for_the_package>",
    "rev": "<revision_template>"
  }
}

Sometimes the template of the version number of a package in PyPI differs from the one used in the repo rev. For example, version 0.910 of mypy in PyPI (no pun intended) maps to repo rev: v0.910. To make this hook aware of the leading v, you need to specify "v${rev}" as a "<revision_template>". Use "${rev}" if both the package version and the repo rev follow the same pattern.

Please, do not open PRs to extend db.py anymore. Use your personal package list instead.

Contributing

See CONTRIBUTING.md.

Credits

This hook is inspired by pre-commit autoupdate.

sync_with_poetry's People

Contributors

Stargazers

Watchers

Forkers

vergenzt connor-berg joshuata berk-tosun jgopel m-roberts real-yfprojects mayankagg44

sync_with_poetry's Issues

Poetry 1.5.0 removes category field.

With v1.5.0 poetry no longer writes a category field to poetry.lock.
This results in the following error:

Traceback (most recent call last):
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/bin/swp", line 8, in <module>
    sys.exit(main())
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/lib/python3.8/site-packages/sync_with_poetry/swp.py", line 170, in main
    retv |= sync_repos(filename, args.all, args.skip, args.config, mapping)
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/lib/python3.8/site-packages/sync_with_poetry/swp.py", line 99, in sync_repos
    poetry_items = PoetryItems(content["package"], all, skip, db)
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/lib/python3.8/site-packages/sync_with_poetry/swp.py", line 58, in __init__
    if ((not all) and package["category"] != "dev") or package["name"] in skip:
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/lib/python3.8/site-packages/tomlkit/items.py", line 1007, in __getitem__
    return self._value[key]
  File "/home/user/.cache/pre-commit/repo6cjh4d2_/py_env-python3.8/lib/python3.8/site-packages/tomlkit/container.py", line 553, in __getitem__
    raise NonExistentKey(key)
tomlkit.exceptions.NonExistentKey: 'Key "category" does not exist.'

Alternative solution

Instead of a sync hook, can't you just use the poetry environment for hooks that have a CLI (via language: system)? Like with mypy, we do

  - repo: local
    hooks:
      - id: mypy
        name: mypy
        language: system
        entry: poetry run mypy .
        types_or: [ python, pyi ]
        require_serial: true
        pass_filenames: false

This also saves space by not creating two environments. What are your thoughts on that approach?

Run sync_with_poetry after changing the .pre-commit.config.yaml

I noticed that sync_with_poetry only runs after changing and committing the poetry.lock file. Since in our project there are more frequent changes within .pre-commit.config.yaml, it would be great that a commit of this file would also trigger "sync_with_poetry". Is this possible?

FR: Strip `frozen: x.x.x` comments

I like to regularly update my pre-commit hooks with pre-commit autoupdate --freeze. Afterwards I run SWP to pin those hooks that are also configured via poetry. (Some hooks aren't python packages so I can't manage all of them via poetry)

This result in lines like the following

rev: 1.7.5 # frozen: 1.7.5

rev: 22.10 # frozen: 22.12

I like that SWP preserves comments (in contrast to pre-commit autoupdate). But I would like it to remove frozen: xxx comments still.

Autoupdate for flake8 does not work

Hi,
I tried this pre-commit-hook. It seems like something is not working properly (at least for me).

I have the following setup:

.pre-commit-config.yaml

- repo: https://gitlab.com/pycqa/flake8
    rev: 3.9.0
    hooks:
      - id: flake8
        additional_dependencies: [
            'pycodestyle==2.7.0',  # E,W
            'pyflakes==2.3.0',  # F
            'mccabe==0.6.1',  # C
            'flake8-bugbear==21.3.2',  # B
            'flake8-builtins==1.5.3',  # A
            'flake8-comprehensions==3.3.1',  # C4
            'flake8-debugger==4.0.0',  # T1
            'flake8-logging-format==0.6.0',  # G
            'flake8-print==4.0.0',  # T0
        ]
        args: [
            '--select=E,W,F,C,B,A,C4,T1,G,T0',
            '--ignore=W503,B305,A003,G003,G004',
            '--max-complexity=10',
            '--max-line-length=120',
        ]
- repo: https://github.com/floatingpurr/sync_with_poetry
    rev: 0.2.0
    hooks:
      - id: sync_with_poetry
        name: sync pre-commit dependencies with poetry
        args: [
            '--config', 'src/python/.pre-commit-config.yaml',
            '--all',  # Scan all dependencies in poetry.lock (main and dev)
        ]

poetry.lock

[[package]]
name = "flake8"
version = "4.0.1"
description = "the modular source code checker: pep8 pyflakes and co"
category = "dev"
optional = false
python-versions = ">=3.6"

[package.dependencies]
mccabe = ">=0.6.0,<0.7.0"
pycodestyle = ">=2.8.0,<2.9.0"
pyflakes = ">=2.4.0,<2.5.0"

When running the pre-commit hook, flake8 won't update. Other dependencies like black do update.
First I thought it might be because of a major version jump. But the issue also remains if the rev in pre-commit-config is 4.0.0. It won't update to 4.0.1 (from poetry.lock).

Any ideas?

[Question] Does this work with plugins?

If this tool were to ping the version of flake8, would it also detect and add, as well as pin the versions of any plugins that flake8 uses?

Matching seems to fails when revision has double quotes

If .pre-commit-config.yaml contains:

- repo: https://github.com/psf/black
  rev: "22.3.0"
  hooks:
  - id: black

then sync_with poetry exists with code 1, but changes nothing.

I added a print there, it told me that pre_commit_repo.rev is 22.3.0while match[4] is 22.3.0"

Sure enough, the regex apparently accepts a double quote at the beginning but not at the end.

From an engineering point of view, I'd say think using regex here is quite brittle. You'd probably be much better off with a style-preserving yaml editor such as ruamel.yaml, which lets you load a yaml, modify it and save it back while preserving empty lines, comments etc.

That said, I see that pre-commit autoupdate uses more or less the same code. And autoupdate does work with quoted revs.

In our case, the main issue is that the check should be with the yaml-loaded version stored in the variable pre_commit_data and not with a regex match.

Feature request: static checks dependencies

When using pre-commit with a static type checker (mypy, pyright, ...), one often needs to add dependencies to the hook so that the type checker knows how to navigate the project (example)

It would be nice if sync_with_poetry would also sync those. For example, we could bind a hook name with a poetry group name in the sync_with_poetry args, and it would get all of the deps defined in the poetry group, put them in the expected format and write them back in the yaml ?

Allow passing in custom db

It would be very useful to be able to define a file with mappings that could be passed as an argument/config option to the hook so that a user could define their own outside of those shipped with the hook

Request: Sync rev of sync_with_poetry and poetry

First of all, great hook!

The idea behind the hook is to have a single source of truth for all our packages, right? The rev of all my pre-commit hooks are managed in the pyproject.toml except for sync_with_poetry and poetry itself. I have to adjust my .pre-commit-config.yaml manually, for these two.

Perhaps, sync of these two special cases can be covered by sync_with_poetry as well?

I have tried using a custom db.json which includes:

    "poetry": {
        "repo": "https://github.com/python-poetry/poetry",
        "rev": "${rev}"
    },

It did not work.

Relevant part of .pre-commit-config.yaml:

- repo: https://github.com/floatingpurr/sync_with_poetry
    rev: 0.4.0
    hooks:
      - id: sync_with_poetry
        args: [--db, .sync-with-poetry-db.json]
- repo: https://github.com/python-poetry/poetry
    rev: 1.2.0  # sync_with_poetry fails
    hooks:
      - id: poetry-check
      - id: poetry-lock
      - id: poetry-export
        args: ["-f", "requirements.txt", "-o", "requirements.txt"]