Option to load many config files about toml HOT 9 CLOSED

Can we reopen this issue? Or a new one?

from toml.

The only problem I have is that Python dict1.update(dict2) is not working with multilevel dicts, and that can be built into toml parser if it will load many config files at once.

Now I load many config files and join them with:
if 'config.ini:

testing = false
[logging]
log_level = "info"
prefix = "my_prefix"

and config_dev.ini:

testing = true
[logging]
log_level = "debug"
tester = "Olo"

Python script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import collections
import sys
import toml


def merge_dict(d1, d2):
    """
    Modifies d1 in-place to contain values from d2.  If any value
    in d1 is a dictionary (or dict-like), *and* the corresponding
    value in d2 is also a dictionary, then merge them in-place.
    """
    for k, v2 in d2.items():
        v1 = d1.get(k)  # returns None if v1 has no value for this key
        if (isinstance(v1, collections.Mapping) and
                isinstance(v2, collections.Mapping)):
            merge_dict(v1, v2)
        else:
            d1[k] = v2


def main():
    """
    Main program
    """
    with open('config.ini') as conf_file:
        config = toml.loads(conf_file.read())
    try:
        with open('config_dev.ini') as conf_dev_file:
            config_dev = toml.loads(conf_dev_file.read())
        merge_dict(config, config_dev)
        # config.update(config_dev)  # Won't work => KeyError: 'prefix'
    except:
        pass

    print 'Config:', config
    print 'Testing:', config['testing']
    print 'Log level:', config['logging']['prefix']

    return 0        # success


if __name__ == '__main__':
    #Start Program
    STATUS = main()
    sys.exit(STATUS)

If you uncomment config.update(config_dev) and comment merge_dict(config, config_dev) you will get KeyError: 'prefix'.

Best regards,
Dobrosław Żybort

from toml.

Maybe I write too much for my request, short version:

Can you add option to load and merge many config files?

config = toml.load_files(['file1.timl', 'file2.toml', 'file3.toml', ])

from toml.

As far as I know, I have nothing to do with python's built-in ConfigParser. None of my methods operate on file objects, the current preferred way for reading from a single config file (which can be fairly easily modified to concatenate multiple files together) is something like:

import toml

with open("config.toml") as configfile:
    confstr = configfile.read()
config = toml.loads(confstr)

I think the toml spec says that one is not supposed to define the same keygroup in multiple places, which could be the source of the error that you are experiencing. If you want to be able to safely concatenate multiple configuration files, you might want to file an issue there.

unrelated: apparently github's reply by email is broken :(

from toml.

As far as I can tell, toml is loading the config files one at a time. It has no way of knowing that you intend to combine the two config files. Your problem is that python dictionaries don't like using update when multilevel dictionaries are involved? The merge dict method from StackOverflow seems to be working fine for your needs, although I may be mistaken. I don't see any bugs in toml since your configurations are being loaded properly.

from toml.

Whoops, I didn't mean to close it, just include the commit in the issue.

I'm kind of wary of including a merge dict method in toml. Is there a better solution possible?

from toml.

Nice, still throwing exception if file not exists but otherwise joining dicts seems to work OK.
I'm not aware of better method.

from toml.

Could this be a regression? I don't see any reference to the fix for this issue in the current code. It looks like the current version uses the python update which breaks the initially requested behaviour. At least in my own code I see a simple key overwriting happening in v0.10.2

from toml.

@machielg I was curious about this but couldn't find any explanation why it was removed besides the commit message from ca3d32e, from 2016.

Unless I missed a note somewhere, maybe a note about this incomplete merge behavior could be added, especially since the module did use a different merge strategy in the past.

from toml.