# duties.py
MULTIRUN = os.environ.get("MULTIRUN", "0") == "1"


def pyprefix(title: str) -> str:  # noqa: D103
    if MULTIRUN:
        prefix = f"(python{sys.version_info.major}.{sys.version_info.minor})"
        return f"{prefix:14}{title}"
    return title

# scripts/multirun.sh
export MULTIRUN=1

# TODO: remove echo statements

It then looks like:

% make check
> Currently selected Python version: python3.9
✓ (python3.7)   Checking code quality
✓ (python3.7)   Type-checking
✓ (python3.7)   Building documentation
✓ (python3.8)   Checking code quality
✓ (python3.8)   Type-checking
✓ (python3.8)   Building documentation
✓ (python3.9)   Checking code quality
✓ (python3.9)   Type-checking
✓ (python3.9)   Building documentation
✓ (python3.10)  Checking code quality
✓ (python3.10)  Type-checking
✓ (python3.10)  Building documentation
✓ (python3.11)  Checking code quality
✓ (python3.11)  Type-checking
✓ (python3.11)  Building documentation
> Restored previous Python version: python3.9
✓ Exporting dependencies as requirements
✓ Checking dependencies

or

% make test
✓ (python3.7)   Running tests
✓ (python3.8)   Running tests
✓ (python3.9)   Running tests
✓ (python3.10)  Running tests
✓ (python3.11)  Running tests

• Mike or new plugin
• Banners: squidfunk/mkdocs-material#5803

Following #4, I'd like to show a graph of dependencies (maybe just the runtime ones) in the credits or development page.
The data can be fetched from PDM's lock file again, and rendered using pydeps and markdown-exec.

In devbox.json:

{
  "packages": [
    "pdm@latest"
  ],
  "shell": {
    "init_hook": []
  }
}

Verify compliance between the licenses of all the project's dependencies. I know there are online services allowing to do that, but as always, I'm more of a self-contained, autonomous, all-in-one, offline, no vendor lock-in project person.

We would get the list of dependencies, their licenses, and check their compliance using something like https://pypi.org/project/liccheck/, and/or a Python port of https://github.com/librariesio/license-compatibility.

It would improve UX: make lock setup instead of pdm lock && make setup.

There is a spec to improve this whole licensing subject: https://reuse.software/

It could be integrated into this template. As scripts/post_generation.py you can then just run reuse download --all to download all used licenses.

• blacken-docs
• darker
• doctest

Opening for visibility. See upstream issue: python/mypy#10633

Thanks for the cool template! I have forked my own (https://github.com/hoangthienan95/copier-pdm/) and added my own variable to be templated in pyproject.toml.

However, when I run copier gh:hoangthienan95/copier-pdm test, I don't see my python_requirement variable being asked and also the pyproject.toml requires-python field is still ">=3.7". Any idea how to fix this? New to copier and pdm :)

I know I recently simplified it, but look at that: https://axgkl.github.io/docutools/about/credits/. Ins't this page great? I want the same thing 😤

I just published version 0.8.0 which brings actual support for Copier v6. While there shouldn't be any issue when generating new projects and updating them, an issue can arise when you try to update a pre-existing project from pre 0.8.0 to 0.8.0+ while your installed Copier version is very recent, like 6.0.0a9. In that case, just downgrade your Copier version to 6.0.0a5 (maybe a6 and a7 work too, didn't try), run the update, and you can upgrade Copier again. In my case, I use pipx:

% copier -f update
...
UnboundLocalError: local variable 'use_precommit' referenced before assignment
...
subprocess.CalledProcessError: Command 'python scripts/post_generation.py' returned non-zero exit status 1.

% pipx install -f copier==6.0.0a0
% pipx inject copier 'markupsafe<2.1'
...
% copier -f -r 0.10.0 update
...  # OK
% pipx uninstall copier
% pipx install copier
...  # upgrade again to latest

The explanation is that since a7 or a8 (commit copier-org/copier@d86c88d), Copier stopped automatically setting the previous Jinja envops ([[, ]], [%, etc.) when _min_copier_version is not present in copier.yml. Copier does set these settings if you have _min_copier_version: 5.1.0 in copier.yml, which I didn't have. My bad for not catching it before!

/cc @yajo as it could be of interest to you 🙂

Great repo, much less overhead than Hypermodem and cookiecutter.

Getting started with the repo and noticed that if a make format is done, it changes the docs/credits.md file contents which mkdocs then fails. In credits.md there is this (python exec="yes" on the open fence):

--8<-- "scripts/gen_credits.py"

which when make format is run with blacken-docs changes the format to:

--8 < --"scripts/gen_credits.py"

with spaces changes. make check-docs will then error with:

 WARNING -  markdown_exec: Execution of python code block exited with errors

  Code block is:

    --8 < --"scripts/gen_credits.py"

  Output is:

    Traceback (most recent call last):
...

I looked into duty and the lazy callable for blacken_docs has an exclude parameter, but it looks like it's never used. I'll continue to pursue there, but has anyone see this error and was able to correct?

Thanks for sharing a great resource!

I have found an issue when using the command "pdm run doc"

> pdm run doc
INFO    -  Building documentation...
INFO    -  Cleaning site directory
../README.md ['C:\\Users\\Steve\\SynologyDrive\\development\\python\\test_app1\\docs']
ERROR   -  Error reading page 'index.md': Snippet at path '../README.md' could not be found

I believe this is caused by the nesting of the documentation folder "docs" inside a folder of the same name, and then attempting to reference README.md outside the root folder, docs.

The solution I have used is to update mkdocs.yml as follows

markdown_extensions:
  ...
  - pymdownx.snippets:
      check_paths: true
      restrict_base_path: false

I could create my 1st pull request if you would prefer that route.
Steve

To help implementing #4, #5 and #6 without spaghetti code, maybe I should create a new project that reads PDM dependencies from both pyproject.toml and pdm.lock and queries various information from PyPI indexes or other online databases to extract as much information as possible about Python packages, and cache it locally (a given package's metadata never changes, supposedly).

Useful information:

• name
• summary
• version
• URLs
• license (SPDX identifier?)
• direct/indirect production dependency?
• direct/indirect development dependency? (not mutually exclusive with production deps)
• dependencies
• parent dependencies if any (maybe hard to keep a clean cache)

A config file should also allow to add items manually, such as Python, PDM, safety and copier-pdm itself.

All these metadata could then be used to generate the credits file, check for dependency confusion, license compatibility issues, and even CVEs using safety/skjold.

Issue on Windows

copier copy --trust "gh:pawamoy/copier-pdm" /path/to/your/new/project

results in invalid path 'project/docs/{% if author_username == "pawamoy" %}.overrides{% endif %}/main.html.jinja' .

 
Solved by
Changing "pawamoy" to 'pawamoy' in {% if author_username == "pawamoy" %}.overrides{% endif %}

 
as documented here: "On Windows, double-quotes are not valid characters in file and directory paths." Copier-docs

Thanks a lot for the great template 🙂

The check is the same on every Python version, so it's not useful to run it multiple times.

Add a duty that reads dependencies from pdm.lock and yields instance of a Dependency class which has an is_internal method that the user can override to implement their own logic (like checking if its name is in a set, or if it matches a regular expression). Then for every internal dependency, the duty asserts that it is absent from a given PyPI index (overridable as well). This way we ensure an internal dependency has no public counter-part that could lead to dependency confusion. Of course if public indexes are not reachable, the duty would be disabled.

Still not sure about implementing this though, as it is only useful in private contexts (which I have no interest in supporting in an opensource project).

It is a bit more work, but we spare some resources and it should make things faster.

@duty
def check_quality(ctx, files=PY_SRC_LIST):
    """
    Check the code quality.

    Parameters:
        ctx: The context instance (passed automatically).
        files: The files to check.
    """
    from flake8.main.cli import main as flake8

    ctx.run(flake8, args=[["--config=config/flake8.ini", *files]], title="Checking code quality", pty=PTY)

@duty
def check_docs(ctx):
    """
    Check if the documentation builds correctly.

    Parameters:
        ctx: The context instance (passed automatically).
    """
    from mkdocs.__main__ import cli as mkdocs

    Path("htmlcov").mkdir(parents=True, exist_ok=True)
    Path("htmlcov/index.html").touch(exist_ok=True)
    ctx.run(mkdocs, args=[["build", "-s"]], title="Building documentation")

@duty  # noqa: WPS231
def check_types(ctx):  # noqa: WPS231
    """
    Check that the code is correctly typed.

    Parameters:
        ctx: The context instance (passed automatically).
    """
    from mypy.main import main as mypy

    class LazyStdout(StringIO):
        def __repr__(self) -> str:
            return "stdout"
        def write(self, value):
            return sys.stdout.write(value)

    class LazyStderr(StringIO):
        def __repr__(self) -> str:
            return "stderr"
        def write(self, value):
            return sys.stderr.write(value)

    os.environ["MYPY_FORCE_COLOR"] = "1"
    ctx.run(
        mypy,
        kwargs={
            "args": ["--config-file", "config/mypy.ini", *PY_SRC_LIST],
            "stdout": LazyStdout(),
            "stderr": LazyStderr(),
            "clean_exit": True,
        },
        title="Type-checking",
        command=f"mypy --config-file config/mypy.ini {PY_SRC}",
    )

It will require pawamoy/failprint#16, pawamoy/failprint#15, and pawamoy/failprint#14.

• add a docs macro that outputs a JSON badge:

  @env.macro
  def get_coverage():
      data = json.load(Path("htmlcov/coverage.json").read_text())
      value = int(data["totals"]["percent_covered"])
      for threshold, hex in (
          (95, "#4c1"),  # brightgreen
          (90, "#97CA00"),  # green
          (75, "#a4a61d"),  # yellowgreen
          (60, "#dfb317"),  # yellow
          (40, "#fe7d37"),  # orange
          (0, "#e05d44"),  # red
      ):
          if value >= threshold:
              color = hex
              break
      return {"value": value, "color": color}

• add a templated file coverage_badge.json to the docs:

  {% with get_coverage() as coverage %}
  {"schemaVersion": 1, "label": "coverage", "message": "{{ coverage.value }}", "color": "{{ coverage.color }}"}
  {% endwith %}
  

• add a badge to the readme pointing to that published JSON file:

  [![coverage](https://img.shields.io/endpoint?url=https://namespace.github.io/project/coverage_badge.json)](https://namespace.github.io/project/coverage)

Hi there!

The error is triggered when executing make docs:

$ make docs
✗ Building documentation (1)
  > mkdocs build
  INFO     -  Cleaning site directory
  INFO     -  Building documentation to directory: /Users/fperez/dev/llull/site
  Traceback (most recent call last):
    File "/Users/fperez/dev/llull/__pypackages__/3.7/bin/mkdocs", line 8, in <module>
      sys.exit(cli())
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/click/core.py", line 1128, in __call__
      return self.main(*args, **kwargs)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/click/core.py", line 1053, in main
      rv = self.invoke(ctx)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/click/core.py", line 1659, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/click/core.py", line 1395, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/click/core.py", line 754, in invoke
      return __callback(*args, **kwargs)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/mkdocs/__main__.py", line 187, in build_command
      build.build(config.load_config(**kwargs), dirty=not clean)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/mkdocs/commands/build.py", line 282, in build
      files = config['plugins'].run_event('files', files, config=config)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/mkdocs/plugins.py", line 102, in run_event
      result = method(item, **kwargs)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/mkdocs_gen_files/plugin.py", line 33, in on_files
      runpy.run_path(file_name)
    File "/Users/fperez/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
      pkg_name=pkg_name, script_name=fname)
    File "/Users/fperez/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
      mod_name, mod_spec, pkg_name, script_name)
    File "/Users/fperez/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "docs/gen_credits.py", line 62, in <module>
      fd.write(get_credits())
    File "docs/gen_credits.py", line 58, in get_credits
      return jinja_env.from_string(template_text).render(**template_data)
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/jinja2/environment.py", line 1291, in render
      self.environment.handle_exception()
    File "/Users/fperez/dev/llull/__pypackages__/3.7/lib/jinja2/environment.py", line 925, in handle_exception
      raise rewrite_traceback_stack(source=source)
    File "<template>", line 24, in top-level template code
  jinja2.exceptions.UndefinedError: 'more_credits' is undefined

make: *** [docs] Error 1

It is due to the fact that the variable more_docs is not defined by the jinja2 template if author_username is not "pawamoy". I will submit a PR to fix it.

If I add a module such as requests to a project via pdm add, what is the proper way to ensure it gets added to all the versions of python when running make tests? I see in the GH workflow how it's being done, but not when developing locally.

Is this something pdm should do, pytest, or some other process?

Love the plugin, just had an issue where my virtual environments were named after their python versions (e.g., 3.8.17, 3.9.17) and I found that I kept getting muddled between venvs and PEP 582 style environments because I'd forget to add the -e flag to my commands.

Suggesting a new feature where there is an environment variable, PDM_MULTIRUN_USE_VENVS which when set to 1 will use virtual environments by default.

See snok/flake8-type-checking#34

The template arrangement is super well executed - so transition from a productive project based on copier.poetry was really really smooth [1].

And: Yes, pdm rocks...
This has to be the future of python packaging - so glad you invested the work in supporting it, so well aligned with copier.poetry.

Cheers & Thanks again!!,
G.

PS: Again, a little problem to justify the issue here:
Took me quite a while (had not all brains available) to find out that optional dev.dependencies are not yet supported in current pipx installed pdm version. Imho that would be worth a note or a comment in the generated pyproject.toml file, to not frustrate others when first trying copier.pdm. Not sure when this is supported.

After thinking about it, I don't really have a use-case for dependency locking. I only use the lock file to install dependencies under multiple Python versions. I don't track it, I don't have reproducible builds in CI (tradeoff for other maintenance benefits). So I could definitely switch to something lighter like Hatch, which simply installs dependencies using pip IIUC. Hatch also provides environments isolation, which is very interesting to me. My dev-deps pollute the version resolution of my prod-deps, while most of the time, my dev-deps work statically, so do not require having the prod-deps installed (pytest is the exception).

So, todo:

• try Hatch in an existing project, see if the migration is smooth (articles? guides?)
• see how Hatch environments fit my workflow, if their verbosity in pyproject.toml is maintainable
• see if Hatch's output is configurable: I want to retain my very minimalist output, it's non-negociable 🙂
• see the perfs: installing deps, updating deps, running check/test commands
• see if Hatch's recent ability to install compiled Python versions fits my workflow

There are a few problems creating a new project from scratch:

1. If there is no python 3.7, I get errors like this:

make check-types
[NoPythonVersion]: No Python interpreter matching python3.7 is found.
make: *** [Makefile:52: check-types] Error 1

2. the template should prompt for the range of python versions we want

After the project was created, I updated pyproject.toml like this:

requires-python = ">=3.8,<3.12"
keywords = []
dynamic = ["version"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Documentation",
    "Topic :: Software Development",
    "Topic :: Utilities",
    "Typing :: Typed",
]

3. After this, I get errors like this:

make check
[NoPythonVersion]: No python is found meeting the requirement python >=3.8,<3.12
make: *** [Makefile:43: check] Error 1

Where is this version check coming from?

4. Looking at pdm-multirun, I wonder whether the Makefile should set PDM_MULTIRUN_VERSIONS

I tried it manually:

pdm multirun -i 3.11 duty check-types
Traceback (most recent call last):
  File "/home/nfr/.local/bin/duty", line 5, in <module>
    from duty.cli import main
ModuleNotFoundError: No module named 'duty'

This is surprising because I see in pyproject.toml the following:

[tool.pdm.dev-dependencies]
duty = ["duty>=0.10"]

Any suggestions on addressing these problems?

See pdm-project/pdm#2310.

In .envrc:

# Automatically sets up your devbox environment whenever you cd into this
# directory via our direnv integration:

eval "$(
    devbox generate direnv --print-envrc ||
    echo echo Install Devbox: https://www.jetpack.io/devbox/docs/installing_devbox/#install-devbox
)"

# check out https://www.jetpack.io/devbox/docs/ide_configuration/direnv/
# for more details