at-gmbh / at-python-template Goto Github PK

It's important to create the documentation at the same time as you code your project, otherwise you won't ever do it or you'll die trying.

To create a template for documentation, MkDocs or Sphinx could be a solution.

MkDocs
Sphinx

In my opinion, it does not make sense to have a __main__.py file if create_cli is set to "no". Since __main__.py is used to run python code from the command line, it makes most sense as the entrypoint to an app's CLI.

The alternative use case is that the app does exactly one thing, in which case the user might not want the overhead of a fully-fledged CLI with typer. I would argue that this overhead is worth it in order to encourage good practice. Besides, even if an app starts off by just doing a single thing with zero options, there's no guarantee it'll stay that way!

During setup with cookiecutter, the yes/no (1/2) order for answering setup questions is different from the rest in one option ('Select use_notebook'):

Consider changing the order to be same for all yes/no questions. This is where one often chooses the wrong option when using the template regularly. ;)

Cheers

...since we dropped Python 3.5 compatibility a long time ago

@klamann As we found out in the delivery.bootcamp 1:

1. In order to run 'pytest tests', the module code (src/) should be installed locally first, not later in the Notebooks section.
   If the package manager is pip, this can be done by running 'pip install -e .' (Needs to be adapted accordingly for poetry and conda).

2. I think another dependency that was not specified was jupyter lab.

3. Lastly, at least for PyCharm, the required Python interpreter setup should be mentioned. At least when using conda or pip, PyCharm does not know about the newly created virtual env (It might do when running poetry install but not tested).

There is a lot of unneccessary clutter in the current Dockerfile for poetry and the two-stage build is not really required. Let's create a nice and easy single stage Dockerfile.

OmegaConf is a YAML based hierarchical configuration system, with support for merging configurations from multiple sources (files, CLI argument, environment variables) providing a consistent API regardless of how the configuration was created. OmegaConf also offers runtime type safety via Structured Configs.

https://omegaconf.readthedocs.io/

With OmegaConf, we can implement features in YAML that we only used to have in HOCON so far, like config inheritance or variable substitution. In fact, OmegaConf handles these things far more consistently than pyhocon currently does.

Let's keep the "yaml" option, but replace the current PyYAML implementation with OmegaConf.

Unfortunately the blueprint for the ci-pipeline is not created even though I selected Select ci_pipeline: 2 - gitlab.
All the selections I made when using the template:

The created project structure:

Hi,

I have noticed that in the Contributing section of the Readme the links to Confluence and the Teams Channel are not up to date.
When trying to open the latter, this message is shown:

There's a bunch of small issues with the Readme we should try to solve

• the pip install and conda install commands are hard to copy-paste, because there's a weblink behind them. maybe let's remove that thing
• we should encourage users to use cookiecutter>=2.0
• some documentation is missing, e.g. regarding Docker
• the list of contributors should be updated (or removed?)

After running cookiecutter https://github.com/at-gmbh/at-python-template in my conda environment and after configuration, I get following error:

Error: failed to remove files - [WinError 2] The system cannot find the file specified: 'environment-dev.yml'
ERROR: Stopping generation because post_gen_project hook script didn't exit successfully
Hook script failed (exit status: 1)

Python version 3.11.5
Cookiecutter version 2.1.1 / 2.2.3 / 2.3.1 (I tried multiple versions)

My configuration was:

full_name [Jane Doe]: Michael Oberhauser
company_name []:
email [[email protected]]: [email protected]
project_name [My Project]: Name
project_slug [name]:
module_name [name]:
project_short_description [A short summary of the project]:
Select package_manager:
1 - conda
2 - pip
3 - poetry
Choose from 1, 2, 3 [1]: 3
Select use_notebooks:
1 - no
2 - yes
Choose from 1, 2 [1]: 2
Select use_docker:
1 - no
2 - yes
Choose from 1, 2 [1]: 2
Select ci_pipeline:
1 - none
2 - gitlab
Choose from 1, 2 [1]: 2
Select create_cli:
1 - no
2 - yes
Choose from 1, 2 [1]: 1
Select config_file:
1 - none
2 - hocon
3 - yaml
Choose from 1, 2, 3 [1]: 3
Select code_formatter:
1 - none
2 - black
Choose from 1, 2 [1]: 2
Select editor_settings:
1 - none
2 - pycharm
3 - vscode
Choose from 1, 2, 3 [1]: 3

What do I do wrong?

We should check that a compatible Python interpreter is used and that the right cookiecutter version is installed on the user's system instead of letting them run into cryptic error messages. This accounts for a lot of the support requests we get.

We should switch to micromamba for faster and smaller docker builds, and use mambaorg/micromamba as base image.

Also, maybe remove the need to use conda run before every command by modifying the default shell, see https://pythonspeed.com/articles/activate-conda-dockerfile

We know that some of our users are using Windows Subsystem for Linux and should test in that environment as well. WSL seems to be available in GitHub Actions: actions/runner-images#50 (comment)

We should use a consistent style for import statements. Proposed config:

repos:
  - repo: https://github.com/pycqa/isort
    rev: 5.10.1
    hooks:
      - id: isort
        args: ['--py', '38', '--line-length', '100', '--multi-line', '3']

This is really just a minor issue.

Instructions say that project name should be the "pretty" name, which can easily contain whitespaces, but then in the README

{{ py_command }} -m ipykernel install --user --name="{{ cookiecutter.project_name }}"

wants to create an ipykernel with a name that would contain whitespaces, which is prohibited by ipykernel (even explained in the error message).

Quick fix: use the project_slug in this command template.

I will make the PR myself (unless there are objections), just created this ticket for backlog-purpose. :)

Depending on the choice of config file type (yaml / hocon / none), different locations for the respecitve config files are used.
The hocon choice uses

{{cookiecutter.project_slug}}/src/{{cookiecutter.module_name}}/res/default.conf

while the yaml choice uses

{{cookiecutter.project_slug}}/config/config.yml

The hocon choice has the benefit that this path lies within the sourcepath of the module, meaning that functions like {{ cookiecutter.module_name }}.util.get_resource_string can fetch config files regardless of module location / installation / etc.

I suggest moving the config.yml also to the res/ directory and while we're at it, also specifying the default path to the config.yml in util.load_config, as it is the case when choosing the hocon option.

we provide a bunch of commands in setup.py (dist, test, testcov) that don't always work as expected. Let's remove those and document this stuff properly in the readme.

I tried to set up the cookiecutter template and I get the folowing error after entering the Project Name.

cookiecutter https://github.com/at-gmbh/at-python-template
full_name [Jane Doe]: xyz
company_name []: Test Company
email [[email protected]]: [email protected]
project_name [My Project]: Test Project 
Traceback (most recent call last):
  File "/usr/bin/cookiecutter", line 11, in <module>
    load_entry_point('cookiecutter==1.6.0', 'console_scripts', 'cookiecutter')()
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cookiecutter/cli.py", line 120, in main
    password=os.environ.get('COOKIECUTTER_REPO_PASSWORD')
  File "/usr/lib/python2.7/dist-packages/cookiecutter/main.py", line 82, in cookiecutter
    context['cookiecutter'] = prompt_for_config(context, no_input)
  File "/usr/lib/python2.7/dist-packages/cookiecutter/prompt.py", line 216, in prompt_for_config
    val = render_variable(env, raw, cookiecutter_dict)
  File "/usr/lib/python2.7/dist-packages/cookiecutter/prompt.py", line 170, in render_variable
    template = env.from_string(raw)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 880, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 591, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateAssertionError: no filter named 'slugify'

see https://keepachangelog.com/ and https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62 for inspirations

There's lots of places where whitespace is used inconsistently. This stuff is hard to notice, since we use a lot of jinja templates. Let's make use of cookiecutter-server to catch as many of these issues as possible.

Add nice, human-readable propmpts to the cookiecutter setup dialogue. Those were introduced in cookiecutter 2.2.

Make sure that

• the pre-commit hooks work as expected
• that all files of a new repo pass the pre-commit hook conditions (no errors on the first commit)

This turned out to be harder than expected, because pre-commit will only run when we have a git repo and all files we want to check have been added to the git repo, which is pretty slow, considering that we want to do this in a loop on a lot of repos.

related PR: #6

When creating a conda project (probably also for pip), the language_version configured for Black in the pre-commit config is not harmonized with the python version defined in the environment.yml.

Hi!
I tried using the template in the bash console in VS Code running in WSL (Ubuntu), but got the following error when execting the commmand cookiecutter https://github.com/at-gmbh/at-python-template:

Select package_manager:
1 - conda
2 - pip
Choose from 1, 2 [1]: 
Select use_notebooks:
1 - yes
2 - no
Choose from 1, 2 [1]: 1
Select use_docker:
1 - no
2 - yes
Choose from 1, 2 [1]: 1
Select create_cli:
1 - no
2 - yes
Choose from 1, 2 [1]: 1
Select config_file:
1 - yaml
2 - hocon
3 - none
Choose from 1, 2, 3 [1]: 1
Select code_formatter:
1 - none
2 - black
Choose from 1, 2 [1]: 2
Select editor_settings:
1 - none
2 - vscode
3 - pycharm
Choose from 1, 2, 3 [1]: 2
Traceback (most recent call last):
 File "/tmp/tmprob9p_0q.py", line 167, in <module>
   handle_editor_settings()
 File "/tmp/tmprob9p_0q.py", line 119, in handle_editor_settings
   _rename_files('.vscode__editor', '__editor', '')
 File "/tmp/tmprob9p_0q.py", line 141, in _rename_files
   path.rename(path.with_name(path.name.replace(old, new)))
 File "/home/lbr/miniconda3/envs/tshack/lib/python3.8/pathlib.py", line 1358, in rename
   self._accessor.rename(self, target)
PermissionError: [Errno 13] Permission denied: '.vscode__editor' -> '.vscode'
ERROR: Stopping generation because post_gen_project hook script didn't exit successfully
Hook script failed (exit status: 1)

I retried it with Select editor_setting:None and it worked. So this maybe a VS Code running in WSL problem. Not really very important because it works now but maybe you find the time to fix it or tell me what I did wrong. Thanks!

I believe it would be great if, as an alternative to pip and conda, this template also supported poetry.

Advantages

1. Easy project isolation
2. Automatic dependency resolution and reproducible builds
3. Only one config file (pyproject.toml) reduces clutter

Disadvantages

1. This will increase the complexity of the template
2. It will make the template less opinionated, which makes it less helpful as a tool to bootstrap new python projects

In my opinion, poetry is wholly superior to the traditional approach of pip, setup.py and requirements.txt. Thus, if increasing complexity is a concern, we could try something like:

1. Implement poetry for a specified trial period (e.g. 6 months)
2. At the end of the period, if a few projects have used and liked poetry, remove (or deprecate) the pip option in the template
3. If no projects have used poetry, or people ran into difficulties or disliked it, remove poetry from the template

Required Work

Here's my assessment of what needs to be done:

1. Add poetry option to cookiecutter.json
2. Add pyproject.toml template and update handle_package_manager() accordingly
3. Create Dockerfile__poetry template and update handle_docker() posthook accordingly
4. Update docker-compose.yml template (behaviour for poetry should be the same as for pip)

Since the editor config files do not point to python executables, there's no need to change them.

I may have missed something; let me know if you notice anything! Thanks 😄

The two stage build was created to reduce image size overhead due to caching, but with pip --no-cache-dir and a proper .dockerignore, this is basically negligible and not worth the downsides of a two stage build (more complicated docker caching, longer image build time).

Let's create a nice and easy single stage Dockerfile.

At the end of the interactive project setup, there's a bunch of deprecation warnings

C:\Users\...\AppData\Local\Temp\tmpj_ps3wk2.py:19: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if StrictVersion(platform.python_version()) < StrictVersion("3.6.0"):
C:\Users\...\AppData\Local\Temp\tmpj_ps3wk2.py:25: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if StrictVersion(cookiecutter.__version__) < StrictVersion('1.7.2'):

this is part of our code, we should find an alternative solution for this stuff

Our UI has become moch more user friendly over the years, we should update the terminal session recording in the Readme. I suggest we use https://asciinema.org/ this time instead of an animated gif.

I have noticed that when the project slug has forbidden characters, it is necessary to start the script again and unfortunately all the provided selections are lost.

I have a suggestion: it is possible to checked the project slug directly after the user has entered it, if there are errors, then the appropriate message will be shown, and a user has an opportunity to correct the error.

@klamann If that's fine, I could work on this issue.

Currently, when create_cli is answered with "yes", then the app() function will contain the logging setup. However, when the entry point is not the command line but a notebook, then using something like

from <project>.util import logger
logger.info("Test")

in a notebook cell uses the logger in its default configuration.

Likewise, when you have a function in the project library such as

from <project>.util import logger

def some_func():
    logger.info("Test")

and you import and call this function from a notebook, the logger will be in its default configuration, instead of the config in config/config.yml.

One fix that I have found is adding these two lines to the __init__.py:

from .util import logging_setup, load_config
logging_setup(load_config("../config/config.yml"))

This configures the logger on every import of the project library.

Perhaps this is worth including in the template unless there are some reasons for not putting code in the __init__.py.

When create_cli is answered with "no" it may still be nice to show how to configure and use the util.logger in the main.py as a showcase.

...due to a breaking change in badges/shields#8671

Now that we have support for GitLab CI (#26), let's add another template for GitHub Actions.

We already have workflows in personio-py that can serve as a starting point.

Hey,

I got the same error as already reported in Issue 9.

I am on Microsoft Windows 11 Pro (Version 10.0.22000 Build 22000), I set up a new conda environment, installed cookiecutter and tried the command cookiecutter https://github.com/at-gmbh/at-python-template.

The commands I executed:

conda create --name cookiecutter-env
conda activate cookiecutter-env
conda install -c anaconda python=3.9.*
conda install -c conda-forge "cookiecutter==1.7.*"
cookiecutter https://github.com/at-gmbh/at-python-template  # using always the default option

I also created a new conda environment with a newer version cookiecutter version 2.1.1, but I got the same error.

Here is the file when you run the cookiecutter command with --debug-file: debug.log

To check whether our template is used (throughout the company) it could be useful to save the traffic insights beyond the 2 week period. This could be done via an Github Action: https://github.com/sangonzal/repository-traffic-action

Where to store the data? Options:

• private git repo (it's only csv's)
• S3 company tennant

We should test using

• pip
• conda

on

• Linux
• Windows
• Mac

with Python 3.6 and all later versions up to the current stable version (right now: Python 3.8).
This way we may detect some of the platform-dependent issues a bit earlier.

Add CI pipeline templates for a few popular CI systems (GitHub Actions, Azure Pipelines, GitLab CI, Travis CI, CircleCI, ...?) and add an option to cookiecutter.json to select your CI provider.

At the moment .DS_Store is tracked by all changes. Since it does not need to be tracked, it should be included in the .gitignore file.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

• chore(deps): update python docker tag to v3.12

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update endbug/add-and-commit action to v9.1.3
• chore(deps): update dependency cookiecutter to ~=2.5.0
• chore(deps): update dependency typer to ~=0.9.0
• chore(deps): update pre-commit hook pre-commit/pre-commit-hooks to v4.5.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-python action to v5
• chore(deps): update conda-incubator/setup-miniconda action to v3
• chore(deps): update condaforge/mambaforge docker tag to v23
• chore(deps): update dependency pre-commit to v3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

currently we're using pkg_resources to load resource files like the default.conf. Since Python 3.7, there is the importlib.resources module in the standard library, which provides a more convenient and efficient way to access resources.

Only problem is that we're currently compatible with Python 3.6; should we make >=3.7 a requirement or leave this for now? The benefit is not huge, but support for Python 3.6 will end this year anyway.

We can probably replace the current implementation with

from importlib.abc import Traversable
from importlib.resources import files

def get_resource_string(path: str) -> str:
    """
    Load a package resource (i.e. a file from within this package)

    :param path: the path, starting at the root of the current module (e.g. 'res/default.conf').
           must be a string, not a Path object!
    :return: the contents of the resource file as string
    """
    return get_resource_path(path).read_text()


def get_resource_bytes(path: str) -> bytes:
    """
    Load a package resource (i.e. a file from within this package)

    :param path: the path, starting at the root of the current module (e.g. 'res/image.png').
           must be a string, not a Path object!
    :return: the contents of the resource file as bytes
    """
    return get_resource_path(path).read_bytes()


def get_resource_path(path: Union[str, Traversable]) -> Traversable:
    """Get a reference to a package internal path object"""
    module_name = __name__.split('.')[0]
    return files(module_name).joinpath(path)

The jupyter and ipykernel dependencies are not installed when you choose to work with poetry AND notebooks in the cookiecutter prompts