beetbox / confuse Goto Github PK

Configurations should be immutable. Compositions should preserve immutability and return new configurations. Yea or nay? I don't regard this as an important feature since most of the time clients don't need to keep around multiple configurations, but it might come in handy down the line and generally doesn't cost that much. Additionally, it is possible to provide both mutating and non-mutating versions of compositions, so one option is to have both.

In the documentation, Confit mentions taking configurations from directories. I'm guessing that means it expects a file of a prescribed name in those directories. I would rather just have a function that takes a path and returns a configuration that I can pass to override (#9). Then, composing a colon-separated path of configuration files in precedence order turns into this:

reduce(confit.override, map(confit.from_file, PATH.split(':')))

I think there is a good opportunity for directory based configuration, though. Many projects (like the users of this other confit) want to use different sets of defaults based on different environments. Common environments include "development", "qa" or "staging", and "production". It would be convenient if they could have a directory of multiple configuration files named after the environments.

def from_file(env, path): # env comes first for partial application
  file = path
  if not file.is_file():
    file = path / env
  if not file.is_file():
    file = path / (env + '.yml')
  ... # more guesses
  with open(file, 'r') as stream:
    return yaml.load(stream.read())

It seems that dumping the config to a file has been removed with this commit. Please update documentation or re-introduce this option.

Hi,

I'd like to suggest to add some basic and advanced examples to the docs.
I had to dig into the issues to find how to add a secondary source config file via set_file, and its precedence order. This improvement would save some time to newcomers.

It might simply look like:

import confuse

# Instanciates config. Confuse searches for a config_default.yaml
config = confuse.Configuration('MyGreatApp')
# Add config items from specific file
config.set_file('subdirectory/default_config.yaml')
# Add config items from a second file. If some item were already defined, they will be overwritten (new file precedes the previous ones)
config.set_file('subdirectory/local_config.yaml')

# Fetch the value
foo_bar_value = config['foo']['bar'].get()
# Fetch the value of specific type, exception will be raised if type doesn't match
foo_bar_int_value = config['foo']['bar'].get(int)

Good job with this great tool !

Dear Adrian,

first of all I would like to say Thank you for this awesome package.
It is really great.

Maybe I was not enough thorough but 'import_write': confuse.OneOf([bool, 'ask', 'skip']), allows me to use such a values like import_write: asdfasdfasdf in config_default.yaml.

The example is the example of package so you can reproduce it by yourself.

What do I do wrongly?

Thanks for your help in advance.

Forgive the possibly stupid question here..

I'm adding Confuse to a project that I'm working on to handle the yaml config files, and looking through the docs, I see a reference to debug levels in the example file, but I'm not quite sure if debug level functionality is something that is built in to Confuse, or if I need to implement debugging myself and can utilize Confuse to get the levels from the command line/yaml file.

Build: 1.1.0

For each iteration through .keys(), I set() values. They get reversed at the end.
set() doesn't look like preserving order, or it reverses it.

config.yaml:

items:
  foo1: bar1
  foo2: bar2
  foo3: bar3

code:

config = confuse.Configuration('MyGreatApp')
config.set_file("config.yaml")
config.set_file("config2.yaml") # Doesn't override items

for item in config['items'].keys():
  pprint(config['items'].keys()) #A
  config['items'].set(config['items'].as_str_seq())
  pprint(config['items'].keys()) #B, consistent with A

Output:
A: ['foo1', 'foo2', 'foo3']
B: ['foo1', 'foo2', 'foo3']
A: ['foo2', 'foo1', 'foo3']
B: ['foo2', 'foo1', 'foo3']
A: ['foo3', 'foo2', 'foo1']
B: ['foo3', 'foo2', 'foo1']

Workaround:
Initial iteration through items are made reversed(). It doesn't feel safe or reliable. It only works because I override every value.

config = confuse.Configuration('MyGreatApp')
config.set_file("config.yaml")
config.set_file("config2.yaml") # Doesn't override items

for item in reversed(config['items'].keys()):
  pprint(config['items'].keys()) #A
  config['items'].set(config['items'].as_str_seq())

Output:
A: ['foo3', 'foo2', 'foo1']
A: ['foo2', 'foo3', 'foo1']
A: ['foo1', 'foo2', 'foo3']

Hi! I'd like this module to be in PyPI so I can tell people to pip install it. Another guy and I are ready to volunteer some cycles to help. Do you have a punchlist of problems to solve / things to finish before you consider confit fit to upload?

Potential minor hitch: unfortunately it looks like there's already a "confit" project in PyPI: https://pypi.python.org/pypi/confit/0.0.0

If it encounters an integer, it will convert it to an integer type validator

If it encounters a float, it throws an error

  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1665, in as_template
    return MappingTemplate(value)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1209, in __init__
    subtemplates[key] = as_template(typ)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1665, in as_template
    return MappingTemplate(value)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1209, in __init__
    subtemplates[key] = as_template(typ)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1665, in as_template
    return MappingTemplate(value)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1209, in __init__
    subtemplates[key] = as_template(typ)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1693, in as_template
    raise ValueError(u'cannot convert to template: {0!r}'.format(value))
ValueError: cannot convert to template: 0.5

Would it be possible to tag 2d46b5f as v0.3.0, as well as to create a v0.3.1 tag (and subsequent PyPI release) for confit?

I tried creating a PR for the v0.3.0 tag but it turns out GitHub doesn't recognise tags as actual changes!

Thanks! 😄

In #2, we made a significant change to how Confit is used: extensible templates have replaced ad-hoc validation. This needs documentation.

One nice observation about the current system: you can fluidly trade off the granularity at which you do validation. You can validate the entire configuration at once and use only valid values later, or you can validate every option individually (as in "old" Confit).

I really like this project first off, the one thing I think is missing is a method to pull in config.yml file for your current project. Example of what I am trying to convey:

Project path to config.yml
Foo/config/config.yml

from Foo import config as cfg_path
config = confuse.Configuration("Foo", cfg_path)

This I think would allow user the ability to work within the confines of the project without the
worries of setting enviornment variables, the problem this would solve is the reduction of boiler plate code to set environment varibles for various Unix distrobution ( Tru64 / Solaris ).

Thoughts?

It would be much prettier to be able to do config.something[0].name.get() instead of using string keys for everything. I realize there's the concern of keys being masked by object attributes, but I feel like it's worth it (you can still use strings in those cases). And that conflict could be partially be alleviated by adding an API reference so people will be able more aware of keys that will be masked.

I am not sure I fully understand the concept of views and combining them. Here is my use case:

I am creating a config using this code and yaml file:

config = LazyConfig('qti-scoring-engine', "app")

Here is an example of the config in the yaml:

uvicorn:
  host: 127.0.0.1
  port: 8080
  workers: 1

Then augmenting it with the arguments:

 config.set_args(args, dots=True)

the arguments I am passing are

parser.add_argument('--uvicorn.host', help='a host', required=False)
parser.add_argument('--uvicorn.port', help='a port', type=int, required=False)

When I then look up the configuration using the below line it returns only the entries from the second view created by the config.set_args:

config['uvicorn'].get()

How could I get all the values returned by merging both views?

When I examine the config in the debugger it has two views, and config.flatten() returns the properly merged configuration.

Any help is very much appreciated.

When I do this for a key that doesn't exist in config_default.yaml, I would expect the key to contain None by default.

config['something'].get({
    'key': None
})

But instead, I get:

confuse.NotFoundError: something.key not found

This is because of this line in as_template:

What I would propose:

# throws NotFoundError - '...' is often used to denote a missing value
config['something'].get({
    'key': ...
})

# returns None
config['something'].get({
    'key': None
})

Would it be possible to add type hints/type comments/type stubs (one of those) to support MyPy, PyCharm, etc?

hi,
I have not found a way to have a list of dicts and validate it.

this is the config

cars:
  - manufacturer: volvo
    color: red
    modell: typec
  - manufacturer: toyota
    color: blue
    modell: typeb
  - manufacturer: volvo
    color: green
    modell: typea

example of not working validation:

cars: [
    {
      "manufacturer": confuse.String(default=""),
      "color": confuse.String(default=""),
      "modell": confuse.String(default="")
    }
  ]
}

in the end i just validated it by:
cars: list

but when i run config.dump() i get back the config in yaml but the list as an json/dict

something: value
something2: value
cars:  [{"manufacturer":"volvo","color":"red","modell":"typec"},{"manufacturer":"toyota","color":"blue","modell":"typeb"},{"manufacturer":"volvo","color":"green","modell":"typea"}]

I read in other issues the opinion that there should be one configuration file, and the rest be data. I have a counterexample to that.

I currently need user-switchable configurations for what the app will be doing as the main configuration file. There will be sample configurations for different tasks that users can easily copy. Independently of that, I want database configuration that will not be modified that often. Consequently, I expect to have config.yaml and db.yaml in my .config/APPNAME directory. So currently I have something like this:

config = confuse.Configuration(APPNAME, __name__)
try:
    _db_config_name = os.path.join(config.config_dir(), 'db.yaml')
    config['db'].set(confuse.ConfigSource(confuse.load_yaml(_db_config_name), _db_config_name))
except confuse.ConfigReadError:
    pass

However, it feels like I'm doing something dirty. :) Is there a better way to do something like this?

confuse tries to make some smart guesses about where config files are stored on the system, but it can be somewhat confusing and sometimes I would rather just hardcode specific locations that I can direct people to with certainty.

Basically, I'd like to be able to just do this:

config = confuse.LazyConfig('mypkg', paths='~/.mypkg/config.yml')

I apologize for just pasting a bunch of code, but I figured I'd illustrate the change. I was planning on submitting a pull request, but I figured I would open an issue first to discuss.

Basically, I decoupled the auto-config detection from the the class definition and now they live in 2 utility functions. Now you have the option to override the default config paths and the Configuration object is simpler and less dependent on the config location assumptions.

class Configuration(RootView):
    def __init__(self, appname, modname=None, paths=None, defaults=None, read=True):
        super(Configuration, self).__init__([])
        self.appname = appname
        self.modname = modname
        self._env_var = '{}DIR'.format(self.appname.upper())
        self._package_path = _package_path(modname) if modname else None

        # set/find config files - this preserves the original behavior by default
        self.paths = _as_list(paths) or self.user_config_files()[:1]
        self.defaults_paths = _as_list(defaults) or self.pkg_config_files()

        if read:
            self.read()

    def user_config_files(self):
        return find_user_config_files(self.appname, self._env_var)

    def pkg_config_files(self):
        return find_pkg_config_files(self._package_path)

    def read(self, user=True, defaults=True):
        if user:
            for filename in self.paths:
                self.set_file(filename)
        if defaults:
            for filename in self.defaults_paths:
                self.set_file(filename, default=True)


    def set_file(self, filename, default=False, ignore_missing=False):
        filename = os.path.abspath(filename)
        if os.path.isdir(filename): # provided directory, find file
            fname = DEFAULT_FILENAME if default else CONFIG_FILENAME
            filename = os.path.join(filename, fname)

        if ignore_missing and not os.path.isfile(filename):
            return
        self.set(ConfigSource(load_yaml(filename), filename, default=default))

    def dump(self):
        ...


def find_user_config_files(appname, env_var, config_fname=CONFIG_FILENAME):
    # If environment variable is set, use it.
    if env_var in os.environ:
        appdir = os.path.abspath(os.path.expanduser(os.environ[env_var]))
        if os.path.isfile(appdir): # allow user to set config explicitly
            cfgfiles = [appdir]
        else:
            cfgfiles = [os.path.join(appdir, config_fname)]
    else:
        # Search platform-specific locations. If no config file is
        # found, fall back to the first directory in the list.
        cfgfiles = [os.path.join(d, appname, config_fname) for d in config_dirs()]
        cfgfiles = [f for f in cfgfiles if os.path.isfile(f)] or cfgfiles[:1]

    # Ensure that the directory exists.
    for f in cfgfiles:
        os.makedirs(os.path.dirname(f), exist_ok=True)
    return cfgfiles

def find_pkg_config_files(package_path, config_fname=DEFAULT_FILENAME):
    return [os.path.join(package_path, config_fname)] if package_path else []

def _as_list(x):
    return (
        x if isinstance(x, list) else 
        list(x) if isinstance(x, tuple) else
        [x] if x else [])

I have run into a limitaion or misconfiguration with confuse in regards to nested element lookup within a yaml file when using the following structure:

config.yaml

logging:
  log_dir: '/tmp/foo'

config_test.py

config = confuse.Configuration('Cli_Test', config.__name__)
parser = argparse.ArgumentParser()
parser.add_argument("-lf", "--log_files", dest='log_dir', help="Path specified for log files.")
args = parser.parse_args()
config.set_args(args)
print(config['logging']['log_dir'].get())  # <-- confuse can not find the second level element.

Is this a known issue or did I mis-use the config object?

There should be a function to generate a configuration from environment variables. Overriding configuration in the environment is handy for

• scripts that cannot create a configuration file (or cannot edit an existing one) and cannot change the command line, and
• modules that want to read configuration but do not control sys.argv (e.g. logging and security modules).

I believe Confit already supports an optional environment variable naming a configuration path. That's good, but I'd extend it to be a colon-separated path (like PATH, CLASSPATH, or PYTHONPATH) of configurations to load in order.

Background

I wrote a configuration library in Python for my employer before discovering Confit. They share many of the same features, but I think there are some things mine does better than Confit. Instead of spawning a new project, I'd like to contribute to Confit, but not without finding a consensus on philosophy, features, and implementation. I'm submitting a few different proposals as separate Github issues.

Assumptions

I want to start by discussing a few assumptions I used.

1. A configuration consists of variable bindings. In each new project, a configuration is born only when some variable holds a magic value (e.g. hostname of external service, port to bind, path to temporary directory, path to credentials). The magic value is moved to the configuration as a default, and the variable's value is loaded from the configuration. There is no need for structure richer than a mapping of names to values.
2. Each variable has a type. Equivalently, each variable can be validated.
3. Each type can be serialized to and deserialized from a string. Configuration files, environment variables, and command line arguments are just strings. We have to be able to represent values as strings.

Can we adopt these assumptions? Do we have real-world counterexamples that cannot be conveniently expressed with these assumptions in place? I'd like to hear your thoughts.

Surprisingly often, I find myself needing to check a configuration value that is either null or another template. An Option template could compose with other templates to make this convenient.

We use confuse for pandas-profiling. Multiple users have reported the same Permission Denied issue:
ydataai/ydata-profiling#214. This issue is blocking usage of the package effectively.

The directory creation is not surrounded with a try/except block:
https://github.com/beetbox/confuse/blob/master/confuse.py#L996
When it attempts to create a global directory, it might not be permitted to do so. It could fall back to the current directory for example.

Moreover, #61 fixes a problem where comments cannot be inline, which is also problematic, but non-blocking.

I started writing a solution for #32 when I came across this. It seems that confuse doesn't recognise argparse.Namespace objects as collections, which whilst never appearing normally should still be understood by confuse.

Here's an example of the issue:

import argparse
import confuse


args = argparse.Namespace(
    first="Hello",
    nested=argparse.Namespace(
        second="World"
    )
)

configuration = confuse.Configuration('example', __name__)
configuration.set_args(args)

print(configuration['nested']['second'].get())

And the stack trace from the resulting error:

Traceback (most recent call last):
  File "example.py", line 15, in <module>
    print(configuration['nested']['second'].get())
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 385, in get
    return as_template(template).value(self, template)
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 980, in value
    if view.exists():
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 183, in exists
    self.first()
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 175, in first
    return iter_first(pairs)
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 59, in iter_first
    return next(it)
  File "/usr/local/lib/python2.7/site-packages/confuse-0.4.0-py2.7.egg/confuse.py", line 507, in resolve
    self.parent.name, type(collection).__name__
confuse.ConfigTypeError: nested must be a collection, not Namespace

I know we shouldn't really explicitly add argparse.Namespace as a collection, so maybe our collection detection logic needs some revamping? It might be an idea to try duck typing it first and then throwing a confuse.ConfigTypeError if that fails instead of just refusing the type in it's entirety.

I think StrSeq should guaranty that the sequence is indeed of strings. Right now this raises no error:

test._root({'strseq': [1, 2, 3]}).get(confit.StrSeq())

Im currently setting up a new project and i am trying to use confuse to parse .yaml config files.

I am trying to make my project modular in somewhat way and im having massive issues referring to the config file.

I want the user/app to be able to check if there is an config.yaml file inside the root folder of the project, if that does not exist then i want confuse to use default_config.yaml within the modules folder.

Currently writing on a mac osx but i guess this should be able to work accross all types of OS's but when i print config.user_config.path() then i get /Users/$USER/Library/Application Support/ili/config.yaml

I have searched EVERYWHERE for a place to change the root folder or where the confuse module fetches the config.yaml file from but it seems it is hardcoded within ..._DIR variables set at the top of the confuse.py file, is there any method or "config" to change that, it is not looking in env-variables cus i have tried to change that too and it doesn't work...

Best regards!

So this is a thought I've been having for a while, but I wanted to get it on paper. I realize that it's not trivial and I understand if there is resistance to the thought.

In ansible, you can use jinja2 templating to substitute variables into your config. Those variables could be overwritten either by another config or via command line.

Here's my thought:

On the python side:

config = confuse.Config() # no interpolation - same as now
config = confuse.Config(vars_from='vars') # use any vars relative to `vars` key
config = confuse.Config(vars_from='.') # use vars relative to root
config.set({'model_version': v4})

Now in the config you can do:

vars:
  model_version: v3
# or just (w/ vars_from='.')
model_version: v3

model:
  path: /path/to/model_{{ model_version }}.tf
  output: /output/path/embedding_{{ model_version }}.csv

upload:
  upload_file: '{{ model.output }}'

(In order to mimic ansible, the jinja would need to be evaluated when resolving the key, not when loading the config file, btw)

Under Python 2, running python2 -m nose gives:

...................E................EEEE
======================================================================
ERROR: Failure: ValueError (Attempted relative import in non-package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/makepkg/python-confuse/src/python2/test/test_dump.py", line 6, in <module>
    from . import _root
ValueError: Attempted relative import in non-package

======================================================================
ERROR: Failure: ValueError (Attempted relative import in non-package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/makepkg/python-confuse/src/python2/test/test_valid.py", line 13, in <module>
    from . import _root
ValueError: Attempted relative import in non-package

======================================================================
ERROR: Failure: ValueError (Attempted relative import in non-package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/makepkg/python-confuse/src/python2/test/test_validation.py", line 12, in <module>
    from . import _root
ValueError: Attempted relative import in non-package

======================================================================
ERROR: Failure: ValueError (Attempted relative import in non-package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/makepkg/python-confuse/src/python2/test/test_views.py", line 8, in <module>
    from . import _root
ValueError: Attempted relative import in non-package

======================================================================
ERROR: Failure: ValueError (Attempted relative import in non-package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/makepkg/python-confuse/src/python2/test/test_yaml.py", line 6, in <module>
    from . import TempDir
ValueError: Attempted relative import in non-package

----------------------------------------------------------------------
Ran 40 tests in 0.073s

FAILED (errors=5)

Under Python 3 python3 -m nose gives:

...................E................EEEE
======================================================================
ERROR: Failure: ImportError (attempted relative import with no known parent package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/makepkg/python-confuse/src/python3/test/test_dump.py", line 6, in <module>
    from . import _root
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: Failure: ImportError (attempted relative import with no known parent package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/makepkg/python-confuse/src/python3/test/test_valid.py", line 13, in <module>
    from . import _root
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: Failure: ImportError (attempted relative import with no known parent package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/makepkg/python-confuse/src/python3/test/test_validation.py", line 12, in <module>
    from . import _root
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: Failure: ImportError (attempted relative import with no known parent package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/makepkg/python-confuse/src/python3/test/test_views.py", line 8, in <module>
    from . import _root
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: Failure: ImportError (attempted relative import with no known parent package)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/makepkg/python-confuse/src/python3/test/test_yaml.py", line 6, in <module>
    from . import TempDir
ImportError: attempted relative import with no known parent package

----------------------------------------------------------------------
Ran 40 tests in 0.071s

FAILED (errors=5)

Running tests on the code without first installing it as a package is needed for testing during packaging, where tests are run after the build stage (ie., python{2,3} setup.py build has been run) but before the packaging/install stage.

In beets we noticed (see beetbox/beets#3264) that when confuse can't read the default config file it silently ignores the problem:

This would have been a lot easier to debug if confuse threw an exception or something to indicate that the file was missing. Maybe some users of confuse don't need a default config file, but for those that do it seems that they would always expect it to be present.

Compositions of configurations should be defined independent of the sources of those configurations.

Let us define a composition operator for configurations: override. override takes two configurations, top and bottom, and returns a new configuration with a union of every variable in the two configurations with the values from top taking precedence. In general, clients will start with a configuration of defaults and apply a sequence of overrides.

override may optionally throw, warn, or ignore overrides that do not exist in bottom. Ignore is a good option for migrating configurations across client versions. If a client updates from version 1 to 2 and drops configuration variable "x", a good migration tool will silently drop overrides of "x" in old configurations. Warn and throw are good for clients of varying pedantry.

override may optionally type check overrides. Some clients may want to delay validation until use, to avoid validating unused variables. Some may want pedantic validation of all overrides.

The current system with .get() is nice, but sometimes it's better to do all the checks once and then forget about types and all that.

To quote @sampsyo:

This is the biggest problem with the current implementation of Confit: I originally opted for on-the-fly validation (i.e., x.get(int) checks that x is an int before returning it) because this avoided weird problems where a schema is difficult to get right ahead of time. But since then, I’ve found that I’ve needed to duplicate a lot of code to use this API (i.e., sometimes I’d like to say once that x is an int and then use it blindly elsewhere).

Here's how I handle it in my config project.
Basically, we have an object just like the config where we replace values with tests.
I also use the _ key to allow overriding error messages (like so).

PS: Sorry for the delay, I was waiting to get out of school!

It seems that you've finalized the changelog for v1.1.0, as per the latest commit. It'd be really helpful if you publish the update to PyPI, since I'd like to make use of the Path template :)

See https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

It should fall back to $XDG_CONFIG_DIRS when there is no local config. It should likely also expose $XDG_DATA_HOME, but i'm not sure how

I want to use confuse for configuring applications that use TensorFlow/Keras for deep learning. Part of that is reading various numerical values from the config file. For most types, this is painless, but I encountered the following issue.

Consider a config.yaml that looks like this:

appName: MyMLApp

...

arch:
  lr: 1e-4

In python, the following will happen:

import confuse

config = confuse.Configuration('MyMLApp', __name__)
model_params = config['arch'].get()
assert isinstance(model_params['lr'], float)

----
Traceback (most recent call last):
  File "<pyshell#66>", line 5, in <module>
    assert  isinstance(model_params['lr'], float)
AssertionError

The expected behavior is of course that this assertion does not fail.

OS and info:
Ubuntu 16.04
Python 3.8.2
confuse 1.1.0

I've encountered this pattern where I'm having an optional configuration block:

foo:
  x: false
  y: 42

In my template, I'm currently writing confuse.OneOf([None, dict]), but it's not really exactly right. This still requires foo: null in my config. What I want is an optional template where I can skip the definition of foo entirely. How would I do that?

Is there a way to create a yaml config file from a defined template dict like the one in /eample/__init__.py

The use case is to generate a new config file to start with if there is no configuration file defined yet.

config.dump() only dumps the currently loaded config, but I can't find an option how to dump a template dict.

The changelog does not appear on https://confuse.readthedocs.io/en/latest/.

We make use of the PyYaml library ability to instantiate a python object from a yaml declaration. In order to do so, I patch Loader to derive from yaml.Loader instead of yaml.SafeLoader. I get that you don't necessarily want that to be the default. But it would be awesome to have the ability to be part of the unpatched library.

It will be helpful for users if an application can spit out its final configuration annotated with the origins of each value, e.g. source code, configuration file, environment variable, or command line argument. That way they can be sure no unknown overrides are sneaking into the configuration. Configurations would need to carry with them not just the values of variables but their origins, so that they survive compositions.

We apply a number of YAML hacks on top of PyYAML for Confit. As pointed out in beetbox/beets#396 (comment), the ruamel.yaml fork takes care of a few of them:

• Round-tripping comments (much better than our current incomplete hacks).
• Preservation of ordered dictionaries.

Is there possible to import OS environment variable on yaml config file?

I've read the docs but don't think composition is possible. Say I want to define paths in terms of a previous path, base_dir.

base_dir: /opt/my_app
data_dir: $base_dir/data

Will confuse expand previously-defined variables?

I realise I can affect this sort of behaviour at the YAML level, with anchors - but since the YAML spec doesn't do concatenation I'd have to define paths as the lists of their segments and recombine them in Python, which isn't ideal:

base_dir: &base_dir /opt/my_app
data_dir:
  - *base_dir
  - /data

Any suggestions?

The Problem

When you specify confuse.Filename('/default/path/to/file') you would expect that, if no config value was found, it would default to '/default/path/to/file'. Instead, it throws exceptions.

When I set my_path: null in my config_default.yaml, it gives me:

  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1218, in value
    out[key] = typ.value(view[key], self)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1218, in value
    out[key] = typ.value(view[key], self)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1586, in value
    True
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1158, in fail
    u'{0}: {1}'.format(view.name, message)
confuse.ConfigTypeError: rsa_key: must be a filename, not NoneType

and when I comment out the key entirely, I get:

  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1218, in value
    out[key] = typ.value(view[key], self)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1218, in value
    out[key] = typ.value(view[key], self)
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 1581, in value
    path, source = view.first()
  File "/usr/local/lib/python3.7/dist-packages/confuse.py", line 201, in first
    raise NotFoundError(u"{0} not found".format(self.name))
confuse.NotFoundError: rsa_key not found

The Culprit

Template implements the value method which is what normally checks the default value:

Filename(Template) overrides value and does nothing with the value of default.

One option, though I admittedly a bit messy, would be to do what's done in OneOf and set template as an attribute and perform the logic in convert.

Each type needs a deserializer. Most types have a constructor (e.g. int, float, bool), and that is enough. Clients should be able to provide alternatives, though, because a deserializer also encompasses validation.

Some clients may want to serialize configurations. Most types have a __str__ method, and that is enough. Clients should be able to provide alternatives, though, in case __str__ does not exist or match the deserializer.

I saw the issue on Confit templates, and the idea is very similar. Going back to my assumptions (#7), though, I think it needs to be mandatory, not optional. A simple dict will be enough to define both types and defaults, however, so it should not be burdensome.

The documentation for quick use is not straight forward. Someone will need a lot of time to understand how this library works.

It seems that the Filename and Path classes don't support the use of default values as it is possible for example with confuse.String() or confuse.Integer().

I tried to define defaults in the template dict like:

template = {
    "TEMP_PATH": confuse.Filename(default="/tmp"),
}

But if I check the configuration file with config.get(template) , an error is raised confuse.NotFoundError: TEMP_PATH not found (if it's not defined in the config.yaml).

Instead I expected that TEMP_PATH will be set to "/tmp".

Is the "default" option not implemented for Filename or do I use it incorrectly?

Once we nail down a model, I'd like to see implementations in Java and C++. The model I'm working toward is general, not platform or language specific.