Code Monkey home page Code Monkey logo

goodconf's Introduction

Goodconf

https://github.com/lincolnloop/goodconf/actions/workflows/test.yml/badge.svg?branch=main&event=push pre-commit.ci status

A thin wrapper over Pydantic's settings management. Allows you to define configuration variables and load them from environment or JSON/YAML file. Also generates initial configuration files and documentation for your defined configuration.

Installation

pip install goodconf or pip install goodconf[yaml] / pip install goodconf[toml] if parsing/generating YAML/TOML files is required.

Quick Start

Let's use configurable Django settings as an example.

First, create a conf.py file in your project's directory, next to settings.py:

import base64
import os

from goodconf import GoodConf, Field
from pydantic import PostgresDsn

class AppConfig(GoodConf):
    "Configuration for My App"
    DEBUG: bool
    DATABASE_URL: PostgresDsn = "postgres://localhost:5432/mydb"
    SECRET_KEY: str = Field(
        initial=lambda: base64.b64encode(os.urandom(60)).decode(),
        description="Used for cryptographic signing. "
        "https://docs.djangoproject.com/en/2.0/ref/settings/#secret-key")

    class Config:
        default_files = ["/etc/myproject/myproject.yaml", "myproject.yaml"]

config = AppConfig()

Next, use the config in your settings.py file:

import dj_database_url
from .conf import config

config.load()

DEBUG = config.DEBUG
SECRET_KEY = config.SECRET_KEY
DATABASES = {"default": dj_database_url.parse(config.DATABASE_URL)}

In your initial developer installation instructions, give some advice such as:

python -c "import myproject; print(myproject.conf.config.generate_yaml(DEBUG=True))" > myproject.yaml

Better yet, make it a function and entry point so you can install your project and run something like generate-config > myproject.yaml.

Usage

GoodConf

Your subclassed GoodConf object can include a Config class with the following attributes:

file_env_var
The name of an environment variable which can be used for the name of the configuration file to load.
default_files
If no file is passed to the load method, try to load a configuration from these files in order.

It also has one method:

load
Trigger the load method during instantiation. Defaults to False.

Use plain-text docstring for use as a header when generating a configuration file.

Environment variables always take precedence over variables in the configuration files.

See Pydantic's docs for examples of loading:

Fields

Declare configuration values by subclassing GoodConf and defining class attributes which are standard Python type definitions or Pydantic FieldInfo instances generated by the Field function.

Goodconf can use one extra argument provided to the Field to define an function which can generate an initial value for the field:

initial
Callable to use for initial value when generating a config

Django Usage

A helper is provided which monkey-patches Django's management commands to accept a --config argument. Replace your manage.py with the following:

# Define your GoodConf in `myproject/conf.py`
from myproject.conf import config

if __name__ == '__main__':
    config.django_manage()

Why?

I took inspiration from logan (used by Sentry) and derpconf (used by Thumbor). Both, however used Python files for configuration. I wanted a safer format and one that was easier to serialize data into from a configuration management system.

Environment Variables

I don't like working with environment variables. First, there are potential security issues:

  1. Accidental leaks via logging or error reporting services.
  2. Child process inheritance (see ImageTragick for an idea why this could be bad).

Second, in practice on deployment environments, environment variables end up getting written to a number of files (cron, bash profile, service definitions, web server config, etc.). Not only is it cumbersome, but also increases the possibility of leaks via incorrect file permissions.

I prefer a single structured file which is explicitly read by the application. I also want it to be easy to run my applications on services like Heroku where environment variables are the preferred configuration method.

This module let's me do things the way I prefer in environments I control, but still run them with environment variables on environments I don't control with minimal fuss.

Contribute

Create virtual environment and install package and dependencies.

pip install -e ".[tests]"

Run tests

pytest

Releasing a new version to PyPI:

export VERSION=X.Y.Z
git tag -s v$VERSION -m v$VERSION
git push --tags
rm -rf ./dist
hatch build
hatch publish --user __token__
gh release create v$VERSION dist/goodconf-$VERSION* --generate-notes --verify-tag

goodconf's People

Contributors

apollo13 avatar ipmb avatar maribedran avatar pre-commit-ci[bot] avatar smileychris avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

goodconf's Issues

Generate config with subclasses

Hi and thanks for this project! I am slowly migrating my configuration management to this wonderful library.

I would like to generate a yaml or toml configuration file, but I am having trouble on how to do it using the initial callable function on a nested class.

Here is a minimal example:

class AppConfig(GoodConf):
    "Main configuration file"

    class AppDaemon(BaseModel):
        "Configuration class for AppDaemon"
        latitude: float = 0
        longitude: float = 0
        elevation: float = 30
        time_zone: str = "Europe/Berlin"
   
    appdaemon: AppDaemon = Field(description="AppDaemon",  initial=lambda:AppConfig.AppDaemon())

config = AppConfig()

toml_config = config.generate_toml()

I am trying to obtain an output similar to the following:

[appdaemon]
latitude = 0
longitude = 0
elevation = 30
time_zone = "Europe/Berlin"

However I obtain the following error:

../../.pyenv/versions/3.10.8-debug/envs/appdaemon/lib/python3.10/site-packages/goodconf/__init__.py:197: in generate_toml
    toml_str = tomlkit.dumps(cls.get_initial(**override))
../../.pyenv/versions/3.10.8-debug/envs/appdaemon/lib/python3.10/site-packages/tomlkit/api.py:51: in dumps
    data = item(dict(data), _sort_keys=sort_keys)
../../.pyenv/versions/3.10.8-debug/envs/appdaemon/lib/python3.10/site-packages/tomlkit/items.py:181: in item
    val[k] = item(v, _parent=val, _sort_keys=_sort_keys)

ValueError: Invalid type <class 'appdaemon.config.AppConfig.AppDaemon'>

It seems to me that the toml serialization can only handle basic Python types?

Deprecation Warning depending on ruamel version

Thanks for this module , super useful quickly making configs.

So nothing is broken just getting deprecation warnings when running code under pytest. I get the following:

PendingDeprecationWarning: safe_load will be removed, use

yaml=YAML(typ='safe', pure=True)
yaml.load(...)

Maybe can do setup based based on ruamel version ? Details in project description: https://pypi.org/project/ruamel.yaml/

Also unrelated, if none of the "default_files" are found there is no error and FileNotFoundError is not thrown, is that intended ?

Setting `Field(default=None)` results in a `pydantic` error

Version

goodconf==3.0.1
pydantic==1.10.11

Issue

There is a bug present in pydantic<2 which makes it impossible to use a None value in a pydantic dataclass. I believe this is causing an issue when creating a configuration Field with default=None like so, resulting in the following error:

# config.py
class AppConfig(GoodConf):
  FOO  = Field(default=None, help="foo")

...
File "/my/config.py", line 11, in <module>
    class AppConfig(GoodConf):
  File "pydantic/main.py", line 221, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 546, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 578, in pydantic.fields.ModelField._set_default_and_type
pydantic.errors.ConfigError: unable to infer type for attribute "FOO"

Potential Fix

The fix seems to be to upgrade to pydantic>2. The goodconf package lists a requirement of pydantic<2.

Automatically generate and expose settings - even less setup code!

Thanks for this solid tool! I wonder if we can take it further to meet my madcap schemes!

Or maybe this is already possible but I just haven't understood yet.

Here's my idealised solution:

Firstly, no need for two files: conf.py goes away and settings.py remains. I preferably only want to look at a single file for all the configuration declarations (goodconf actually makes it feasible to have this file declarative, short and easy to grok, woot woot).

Then, in my project-name/settings.py, I define (names not fixed, bikeshed as necessary):

from goodconf import GoodConf, generate_settings

class Base(GoodConf): # django core settings ...
class Dev(Base): # my dev specific settings ...
class Stage(Base): # my staging specific settings ...
class Prod(Base): # my production specific settings

generate_settings(  # `GoodConf` API more or less
    load=True,
    file_env_var='DJANGO_GOODCONF',
    default_class=Dev,
    settings_directory='./settings/',
)

So,

  • If no DJANGO_GOODCONF is supplied, we generate a settings/dev.yml
  • If DJANGO_GOODCONF=Stage, we generate a settings/stage.yml
  • We load the settings/<ENV>.yml and programmatically make all settings available for Django (no need to X = config.X boilerplate) through some globals trick or something

The reason we need generate_settings (or something that isn't an environment specific subclass of GoodConf, like, Dev) is so that we can put the logic for the selection of the environment and generation of settings into goodconf core and not leave it for the user to configure. The DJANGO_GOODCONF env var is the only thing the user needs to consider when deploying to anything that isn't the localhost (Dev used by default otherwise).

Thoughts? Happy to get stuck in on implementation if we're in agreement!

PS. A lot of this comes from my past experiences with https://github.com/jazzband/django-configurations. I think if we join these two approaches, we have a really flexible method for Django configuration.

Generate comments when writing toml files

When you generate a yaml config, the docstring and description for each field will be used as comments in the file.

if cls.__doc__:
dict_from_yaml.yaml_set_start_comment("\n" + cls.__doc__ + "\n\n")
for k in dict_from_yaml.keys():
if cls.__fields__[k].field_info.description:
dict_from_yaml.yaml_set_comment_before_after_key(
k, before="\n" + cls.__fields__[k].field_info.description
)

It looks like tomlkit has the same functionality, so we should use that.

Export typing information when packaging

Whenever I import Goodconf I get the next mypy error:

Skipping analyzing "goodconf": found module but no type hints or library stubs

It would be nice if goodconf exposed the typing information at package level, probably adding the py.typed file would be enough

Release 3.0.0

  • update CHANGES.rst (can be converted to markdown if desired) (@apollo13)
  • Remove zest.releaser (@ipmb)
  • Bump version and tag (@ipmb)
  • Use hatch to build/publish (@ipmb)
  • Bump version to dev (@ipmb)
  • Document release process (@ipmb)

Support `env_prefix`

Hi, I'm trying to use the BaseSettings env_prefix Config property to preprend a string on the environment variables without success. A simple snippet that shows the desired behaviour is:

import os

from goodconf import GoodConf


class AppConfig(GoodConf):
    "Configuration for My App"
    DEBUG: bool = False

    class Config:
        env_previx = "GOOD_"


os.environ["GOOD_DEBUG"] = "True"

config = AppConfig(load=True)

assert config.DEBUG

Using os.environ['DEBUG'] = "True" works though

Allow config saving

Right now we can only load the settings, but it would be nice to have a save method to save changed configuration values. It should keep the user comments on the file

Official support for 3.11

I'd like to see support for 3.11, most likely it is just a matter of adding it to the tests. Finding a supported Django version for the tests is more fun then :D If we were to drop 3.7 as well (EOL in 5 months) we could increase Django for the tests to 4.1. Otherwise we could make the tests conditional somehow. This would be okay to do together with #14. Let me know which approach you'd prefer

Generate .env file

We can currently generate yml, json, and toml, but .env style files are probably the most popular option for storing environment variables for local development. It'd be nice if we could generate a .env (with field descriptions as comments) in addition to the other options.

Allow the initialization of values

Hi, first thanks for your awesome project :)

To build the test cases, I want to create a GoodConf instance with some values that are different from the default, for example the DATABASE_URL attribute. It will be nice that we could do something like:

AppConfig(DATABASE_URL='fake://address')

I know BaseSettings from pydantic supports it, so maybe we can call to super().__init__ in the GoodConf`` init`. What do you think?

Doc incorrect

Hi, thanks for goodconf!

In the example in the docs a Config object is initialised and assigned to a variable config. However the load method returns nothing. Therefore the config var is None.

Also in the modified manage.py I had to include this line for it to work normally:

if __name__ == '__main__':
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')
    settings.config.django_manage()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.