dynaconf / dynaconf Goto Github PK

Users of Django apps have to specify a 'DYNACONF_SETTINGS' environment variable to specify where their settings file is located. Application writers would benefit from being able to set a SETTINGS config in their settings.py which would provide a default location for a settings file. If the file is not present, dynaconf should log this, and allow django to start up normally.

from flask import Flask
from dynaconf.utils.flask import DynaFlask

app = Flask(__name__)
DynaFlask(app)

app.run()

Then, app.config will be replaced by DynaFlask which is a config class able to read from dynaconf

Currently, all loaders are executed again every time we

• switch namespaces using settings.using_namespace or settings.namespace
• when a variable is marked as always_fresh
• when using `settings.get('var', fresh=True)

For the fresh=True and always_fresh it makes sense to execute loaders again as it is demanded to get the fresh variable (lets say from redis or another database).

The problem is that it may cause overhead (I guess not too much) but in any case we can improve by.

• writing more namespace tests (with more switching and fresh examples)
• implement a cache on execute_loaders so files and databases are not accessed twice when we are only switching namespaces and not wanting to load all again.

Part of this is already implemented in loaded_by_loaders dictionary, we just need to use this as a cache point.

A Django extension when enabled should replace django.conf.settings with Dynaconf instance

I tried to convert README.md to README.rst using m2r

the code is on setup.py

does not works, pypi page is bas formatted: https://pypi.python.org/pypi/dynaconf

I need help to convert README to proper RsT or even rewrite README in valid .rst and drop .md.

Just like vault and redis
for

https://github.com/coreos/etcd

SSIA

I am wondering if there might be a method to define default values for configuration parameters for some values because they might not need to be required in some namespaces.

For example , I have a settings.yml file

DYNACONF:

DEVEL:
  MONGODB_DATABASE: devel
  MONGODB_HOST: localhost
  MONGODB_PORT: 27000

PRODUCTION:
  MONGODB_DATABASE: production_stuff
  MONGODB_HOST: mongodb.example.com
  MONGODB_PORT: 10980
  MONGODB_USERNAME: efe6398127928f1b2e9ef3207fb82663
  MONGODB_PASSWORD: 1f048e5fb80c559a4ab4c6e79d940708

And the username and password fields are not required in the devel namespace (can be None).

If I attempt to get the MONGODB_USERNAME parameter from the production namespace, it works as expected but if I attempt to get it from the DEVEL namespace, it raises an AttributeError and I need to hack around it with getattr as shown.

print(settings.MONGODB_HOST)
print(settings.MONGODB_PORT)
print(settings.MONGODB_DATABASE)
print(getattr(settings, "MONGODB_USE_SSL", False))
print(getattr(settings, "MONGODB_USERNAME", None))
print(getattr(settings, "MONGODB_PASSWORD", None))

I understand if this is not a desired feature but it might come in really handy in some situations when we need some parameters in only one environment.

As of today, importing dynaconf configures the handlers in the logging root logger, which renders the basicConfig useless:

This function does nothing if the root logger already has handlers configured for it.

Example:

>>> import logging
>>> logging.basicConfig(format="[%(asctime)s|%(levelname)s] %(message)s",
...                     datefmt="%Y-%m-%dT%H:%M:%S")
>>> logging.error("Something")
[2018-06-23T18:21:03|ERROR] Something

The collateral effect of importing dynaconf:

>>> import logging
>>> import dynaconf
>>> logging.basicConfig(format="[%(asctime)s|%(levelname)s] %(message)s",
...                     datefmt="%Y-%m-%dT%H:%M:%S")
>>> logging.error("Something")
2018-06-23 18:21:59,723 - root - ERROR - Something

Depends on #84

Traversal of nested dictionaries should also be possible in validators.

having settings.yaml

DEFAULT:
    REDIS:
        HOST: localhost
        PORT: 1234

It is currently possible to write validator to ensure:

REDIS key is set and it is a dict and it has the keys HOST and PORT being HOST a str and PORT as int

from dynaconf import settings, Validator

settings.validators.register(
    Validator(
        'REDIS', 
        must_exist=True, 
        is_type_of=dict.
        condition=lambda v: (
            'HOST' in v and 'PORT' in v
        ) and (
            isinstance(v.HOST, int) and isinstance(v.PORT, int)
        )
    ),
)

RFC: Allow the above validation to be written using dot notation

settings.validators.register(
    Validator('REDIS',  must_exist=True,  is_type_of=dict),
    Validator('REDIS.HOST', must_exist=True, is_type_of=str),
    Validator('REDIS.PORT', must_exist=True, is_type_of=int),
)

Also the above validation can be translated to toml validator file as in https://dynaconf.readthedocs.io/en/latest/guides/validation.html#cli-and-dynaconf-validators-toml

Targeting plugin based applications take as example:

Configuration for general program is in export SETTINGS_FOR_DYNACONF=/etc/program/settings.ext (ext being any of the supported formats yaml, json, ini, toml etc)

That same program has plugins and plugin stores its settings under different path like /etc/plugin_name/settings.yaml

RFC: Allow include of settings files in different formats.

Then /etc/program/settings.yaml

DEFAULT:
    DYNACONF_INCLUDE:
        - /etc/plugin_name/settings.yaml
        - /etc/other_plugin_name/settings.toml
        - /etc/settings_folder/* 

or toml

[default]
dynaconf_include = ["path1", "path2"]

or json

{
   "default": {
        "dynaconf_include": ["path1", "path2"]
    }
}

then All the loaders will take the special DYNACONF_INCLUDE key then load values from each of the files listed using the normal pipeline.

Requirements

• Backwards compatibility
• Multiple [envs] should be considered
• Preserve the ordering
• Nested includes should have a maximum recursion level
• When the path is a directory, glob should be used and load in alpha order
• All file formats should be supported, the functionality should be in BaseLoader

For flask and django the dynaconf cli should discover the global_env FLASK_ or DJANGO_ to be able to list, write or init correctly

Would be nice to support a more complex settings structure like:

PRODUCTION:
    DATABASE:
        HOST: ....
        PASSWORD: ...

And after use it like: settings.DATABASE.HOST
Currently you get a dict at DATABASE

look: https://github.com/disqus/gutter

Dynaconf should have a simplified version

from dynaconf import settings
if settings.is_active('some feature'):
    # do something

and then in dynaconf.conf features will be defined with conditions

[some feature]
object = user
attribute = age
condition = >=45

Features should also be read from REDIS

Dynaconf requires Flask by default, is that by mistake or is it intentionally?

 File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/__init__.py", line 5, in <module>        
   from dynaconf.contrib import FlaskDynaconf        
 File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/contrib/__init__.py", line 1, in <module>        
   from dynaconf.contrib.flask_dynaconf import FlaskDynaconf, DynaconfConfig  # noqa        
 File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/contrib/flask_dynaconf.py", line 2, in <module>        
   from flask.config import Config        
ModuleNotFoundError: No module named 'flask'

Run from project folder, admin to inspect config data

having a dynaconf.toml file

dynaconf validate

must run validators

Write a lot of tests

This library should support Python 3 only

Hey!

I tracked a bug in the namespace with YAML.
The problem is basically that the environment variable DYNACONF_NAMESPACE is loaded in the execute_loaders(self, namespace=None, silent=None, key=None) AFTER the YAML as been already parsed in module_loader(obj, settings_module=None, namespace=None, silent=False).
And for some reason the YAML loader isn't rexecuted in the execute_loaders, not sure why.

Thanks ;)

Implement loaders for:

• https://www.vaultproject.io/
• https://docs.docker.com/engine/swarm/secrets/

Based on https://www.reddit.com/r/Python/comments/3h1tau/dynamic_config_load_for_python_load_settings_from/

If we have two projects using Dynaconf in the same environment we need a way to set different variables, so we need namespaces.

A namespace should be defined by __ double underline (also known as dunder)

so

$DYNACONF_DATABASE

will parse to

settings.DATABASE

and

$DYNACONF__PROJECTA_DATABASE

or

$DYNACONF__PROJECTB_DATABASE

will be accessed by settings the namespace

from dynaconf import settings
settings.DATABASE

# accessing namespace vars

settings.namespace('PROJECTA')
settings.DATABASE

settings.namespace('PROJECTB')
settings.DATABASE

I haven't had time to find all the Python 3 incompatibilities but there's one I just ran into here. basestring is not a thing in Python 3 anymore.

dynaconf is properly showing the settings I put into /etc/pulp/settings.py

(pulp) [vagrant@pulp3 pulp]$ dynaconf list -k DATABASES
Django app detected
Working in development environment 
DATABASES: {'default': {'CONN_MAX_AGE': 0,
             'ENGINE': 'django.db.backends.dummy',
             'HOST': 'postgres',
             'NAME': 'pulp',
             'PORT': '5432',
             'USER': 'pulp'}}

However, django is not using the setting. It seems like Django is loading the settings for the database before dynaconf is loaded.

SSIA

Load settings from YAML file

• This is a good example to start explaining Dynaconf https://rocket.rs/guide/configuration/

Usually I create a virtualenv for my python-applications,
and I collect all config-files in an etc-directory to avoid polluting the project-directory, like this:

/opt/application-name/bin
/opt/application-name/etc/
/opt/application-name/lib
/opt/application-name/src/git-checkout

So I propose to look for config-files not only in the parent-directories, walking all the way up to root,
but also check the existence for an etc-subdirectory.

In bigger projects with lots of components, we split up the etc-dir into
.../etc/uwsgi
.../etc/supervisord
.../etc/dynaconf

As bonus we also support the global etc-directory:
/etc/settings.yml
/etc/dynaconf/settings.yml

I couldn't find any documentation for this project.
I suggest to generate (with sphinx, using autodoc plugin?) some documentation for the end user and for future people that are interested in contributing to this project (for example I couldn't find some hints or guides on how to create a loader from scratch).

It's very common in production to store settings in a NoSQL database like DynamoDB.
Would be nice to support these kind of sources.

1. Change the use of term namespace to use env so variable names like NAMESPACE_FOR_DYNACONF will be changed to ENV_FOR_DYNACONF, also functions like using_namespace will be called using_env and a compat layer must be added to not break. The reason is that it meets the use of ENV in other frameworks.

2. Rename the base DYNACONF env to DEFAULT, currently, the base env is called DYNACONF it is the default namespace for variables and must be renamed to DEFAULT (this will be a breaking change)

3. Change the order of loading setting .toml as the preferred config format.

4. Add to CLI #2 a command called init which will add a new settings.* file (defautls to toml) containing the scaffolding.

So with this change, an application config file will look like this.

settings.toml

[default]
server = 'defaulserver.com'
# value in [default]  env will serve as default when it is not defined in specific env
# default is optional and can also be empty or ommited

# values from other [envs] will overrride existing default values

[development]
server = 'devserver.com'

[staging]
server = 'stagingserver.com'
user = 'staging user'

[testing]
server = 'ciserver.com'

[production]
server = 'prodserver.com'

[custom]
server = 'customserver.com'

# If it is needed to override all values in every namespace declare a [global] env
[global]
server = 'itoverrideseverything.com'

IN a .secrets file there is no reason to have different envs as easch environment must have its own secrets file.
.secrets.toml

[default]
password = 'SuperSecret'

The variable ENV_FOR_DYNACONF must default to DEVELOPMENT and when it is time to change a variable must be defined.

export ENV_FOR_DYNACONF=PRODUCTION

That is all

Programatically switching namespaces may be useful for testing and also to use Dynaconf as a Feature flag system so to switch:

with settings.using_env('staging'):
    assert settings.USERNAME == 'staging user'
    assert settings.SERVER == itoverrideseverything.com'  # override by [global]

The higher precedence is always for the DYNACONF env it works as a global and is named DYNACONF instead of GLOBAl to avoid conflicts with existing envvars.

To overwrite values in any of the namespaces do:

export DYNACONF_SERVER='ThisOverwritesEverything.com'

• always load toml as the first in list
• install toml by default

a dynaconf cli to be use used on shell to validate and --help as documentation

setup.py specifies packages=find_packages(), which will include tests and example.

Perhaps packages=find_packages(include=['dynaconf']) would be better?

Hi, I have this folder structure:

test_dynaconf
└── subfolder
    ├── main.py
    └── settings.toml

with main.py:

#main.py
from dynaconf import settings

print(settings["KEY1"])

and settings.toml:

#settings.toml
[default]
KEY1="hi"

If I run from subfolder, everything works:

➜  subfolder python3 main.py 
hi

But if I run from test_dynaconf or another path, it breaks:

test_dynaconf python3 subfolder/main.py
Traceback (most recent call last):
  File "subfolder/main.py", line 3, in <module>
    print(settings["KEY1"])
  File "/usr/local/lib/python3.5/dist-packages/dynaconf/utils/functional.py", line 208, in inner
    return func(self._wrapped, *args)
  File "/usr/local/lib/python3.5/dist-packages/dynaconf/base.py", line 193, in __getitem__
    raise KeyError('{0} does not exists'.format(item))
KeyError: 'KEY1 does not exists'

In my oppionion, there should not be a constraint to define the default DYNACONF namespace when the application doesn't need it. There should be a way to avoid loading the default namespace when changing it inside the application.

My suggestion is the to load the desired configuration when dynaconf.namespace(...) is called or load the default configuration only when there was an attempt to read a variable from dynaconf.settings with no previous dynaconf.namespace() call.

One use-case might be choosing the desired environment from command-line or from other sources.

from dynaconf import settings


if sys.argv[1] == "prod":
   settings.namespace("production")
elif sys.argv[1] == "devel":
   settings.namespace("devel")
else:
   raise RuntimeError("Invalid environment specified")

print(settings.MONGODB_HOST)
print(settings.MONGODB_PORT)
print(settings.MONGODB_DATABASE)

Using pytest-xdist or even threading and multiprocessing directly

we need to write a test which will run on travis to ensure.

• Dynaconf is thread safe
• There are no conflicts with variables
• There are no conflicts when multiple threads swaps namespaces
• There are no conflicts reading the same config file at the same time

If settings.py is not present, try settings.yml instead of throwing an error. This will allow projects using settings.yml on root without having to specify a env var nor calling configure function.

Lets take a settings.yaml

DEFAULT:
    REDIS:
        HOST: localhost
        PORT: 1234

To access the port you need to use one of:

settings.REDIS['PORT']        # may raise KeyError
settings['REDIS']['PORT']      # may raise KeyError
settings.REDIS.PORT           # possible because it is a Box(), but may raise AtributeError
settings.get('REDIS').PORT  # dict like get method
settings('REDIS')['PORT']      # the same as calling .get
settings.get('REDIS').get('PORT')  # may raise AttributeError

But what if you are not sure that the REDIS value is defined in settings.yaml and you don't wanna fail?

one way is relying on the .get default argument (dict like)

settings.get('REDIS', {}).get('PORT', 1234)

RFC: Allow traversing

settings('REDIS.PORT')   # would access the settings['REDIS']['PORT'] or raise KeyError.

settings('REDIS.PORT', 1234)   # if value cannot be accessed returns 1234 by default

The . should be parsed on LazySettings.get method and then perform the traversing.

Requirements

• Avoid regressions (keep backwards compatibility)
• Allow it to be disabled opt-out
• Ensure it does not include overhead by recursion and traversing
• Document it
• Write test cases
• Write examples/

https://github.com/lbolla/yamlenv

Hi Bruno,
I am trying to load configuration from multiple files in different folders
This is the folder structure that i use:
my_project
├── config1
│   ├── init.py
│   └── settings.toml
├── config2
│   ├── init.py
│   └── settings.toml
└── main.py
in each settings.toml i have 2 environment development and production ad VAR1 for config1 and VAR2 for config2

[development]
VAR1 = 'var1_dev'
[production]
VAR1 = 'var1_prod'

In the main

import os
from dynaconf import settings

from my_project.config1 import __file__ as config1_file
from my_project.config2 import __file__ as config2_file

f1 = os.path.join(os.path.dirname(config1_file), 'settings.toml')
f2 = os.path.join(os.path.dirname(config2_file), 'settings.toml')

os.environ['SETTINGS_MODULE_FOR_DYNACONF'] = "{},{}".format(f1, f2)
                                                        
print('development')
print(settings.VAR1)
print(settings.VAR2)

print('production')
settings.setenv('production')
print(settings.VAR1)
print(settings.VAR2)

For default and development it works but when i try to setenv to production i have this error :

development
var1_dev
var2_dev

production
Traceback (most recent call last):
  File "/home/hch/PycharmProjects/dynaconf/iia/my_project/main.py", line 13, in <module>
    settings.setenv('production')
  File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 459, in setenv
    self.execute_loaders(env=env, silent=silent, filename=filename)
  File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 572, in execute_loaders
    filename=filename)
  File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/loaders/__init__.py", line 53, in settings_loader
    settings_module = settings_module or obj.settings_module
  File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 413, in settings_module
    if settings_module != self.SETTINGS_MODULE:
AttributeError: 'Settings' object has no attribute 'SETTINGS_MODULE'

I have a settings written in linux, with utf-8, however when running on windows it is being read as cp1252 which shows invalid characters. :(

I get this error when I run an exe generated from pyinstaller :

>dummy.exe
Hello World !!!
Traceback (most recent call last):
  File "dummy.py", line 4, in <module>
  File "site-packages\dynaconf\base.py", line 87, in __getattr__
  File "site-packages\dynaconf\base.py", line 113, in _setup
  File "site-packages\dynaconf\base.py", line 166, in __init__
  File "site-packages\dynaconf\base.py", line 575, in execute_loaders
  File "site-packages\dynaconf\base.py", line 550, in loaders
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dynaconf.loaders.env_loader'
[3256] Failed to execute script dummy

Here is my source script dummy.py :

import dynaconf

print('Hello World !!!')
print(dynaconf.settings.var1)

I use the latest 64 bits anaconda on a windows 7 64 bits and :

pip.exe install pyinstaller==3.3.*
pip.exe install pyyaml==3.12.* dynaconf[yaml]==1.0.*

I try to build a folder or a single exe with pyinstaller (both give the same error) :
pyinstaller.exe dummy.py
or
pyinstaller.exe --onefile dummy.py

Yes it is possible and https://coveralls.io/github/rochacbruno/dynaconf lists all files missing coverage.

Lets write more tests!!!

Currently, if I have FOO = False in settings.pyand I usesettings.get('FOO', cast='@Bool')`, I get:

AttributeError: 'bool' object has no attribute 'lower'

I would suggest changing https://github.com/rochacbruno/dynaconf/blob/master/dynaconf/utils/parse_conf.py#L9 to:

'@bool': lambda value: True if str(value).lower() in true_values else False,

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Hi!
First of all, you rock. Great project.

Bug description:

Environment: Flask Application

When you use yaml files, the settings.yml(yaml) and .secret.yml(yaml) are no loaded in the settings
When you use toml both are loaded, e.g.:

|- other/stuff
|- .env                     
|- manage.py
|- settings.yml(yaml)(toml)  # -->  Both settings are loaded from toml
|- .secrets.yml(yaml)(toml)  #      No settings are loaded from yaml/yml

Raises:

(...) dynaconf/base.py", line 100, in __getattr__
    return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'ATTRIBUTE'

Question: is there any way to use settings.toml and .secrets.yml?

Hi Bruno

I am reading about dynaconf and it looks quite good, however, i dont know if it will fit in our use case, so i am looking for your confirmation on that

I would like to use a config server for python to retrieve external configurations (eg - a yaml file in a git repository) that i will use that later for my application. That configuration can change at some point, so it should be dinamically reloaded

I have seen it works well when pulling a configuration from a redis database or similar, but i dont know if it would work to pull a file from a git repository?

Thanks