Support for configuration files about typed-argument-parser HOT 21 CLOSED

Hi again,
I found a solution based on your hacky solution that still works with help. It seems to work:

from tap import Tap


class ArgsConfig(Tap):
    # Add all the arguments here
    config_file: str = None
    arg_int: int = 1
    arg_str: str = 'some_string'

    def add_arguments(self):
        self.add_argument('-i', '--arg_int')

class Args(ArgsConfig):
     # This inherits all the params and just parses the config
    def add_arguments(self):
        config_file = ArgsConfig().parse_args(known_only=True).config_file
        if config_file is not None:
            self.load(config_file)
        super().add_arguments()

args = Args()
args = args.parse_args(known_only=True)

What do you think?

from typed-argument-parser.

Hello,

I think that we already have functionality pretty close to that:
https://github.com/swansonk14/typed-argument-parser#saving-and-loading-arguments

What are you looking for in addition to it? Support for the same config formats like yaml and ini?

Thank you for the interest.

from typed-argument-parser.

Hi @martinjm97,
Thank you for your quick response!
The format of the config file doesn't really matter. It could be nice to support other formats but that isn't the issue at all.
What I am suggesting is a way to pass the path to the config file as a command-line argument.
Let's say I have several experiments with different configurations, I can save files (for example in json format) for each experiment and then call the main function with: --config_file
The expected result would be that the parser would read the config_file.
I guess I could implement somthing like:

"""main.py"""

from tap import Tap

class MyTap(Tap):
    config_file: str = None # path to json config file
    package: str = None
    is_cool: bool = True
    stars: int = 5

args = MyTap().parse_args() 
if args.config_file is not None
    args.load(args.config_file) # parse config file

I would also expect that if in addition to the config file argument, some other argument is passed that it would override the config file. That would probably mean something like:

args = MyTap().parse_args()
if args.config_file is not None
    args.load(args.config_file)  # parse config file
    args.arge_parse()  # override with command line.

This implementation should work, but the only problem would be with positional arguments that appear in the config file, since the first MyTap().parse_args() will fail.

Does this make any sense?
Thanks again!

from typed-argument-parser.

Well, I tried my proposal above (and some other variations) and it didn't work since arge_parse() overrides all of the set arguments back to default.
May need to find a more complex solution

from typed-argument-parser.

Right now, the (hacky) solution is:

from tap import Tap


class TapConfig(Tap):
    config_file: str


class Args(Tap):
    arg: int
    
    def add_arguments(self):
        self.load(TapConfig().parse_args(known_only=True).config_file)

args = Args()
args = args.parse_args(known_only=True)

Clearly, this is not ideal. A potential improvement is to internalize the TapConfig and then use a sentinel type to indicate that the specific flag is a string corresponding to config file that should be loaded prior to other arguments being parse (to account for positional and required arguments as you pointed out). For example, consider the proposal:

from tap import Tap, TapConfig  # TapConfig isn't a thing yet 

class Args(Tap):
    arg: int
    config_file: TapConfig = 'my_default_path'

args = Args()
args = args.parse_args()

What do you think of this proposal? Is this what you were looking for? Any other suggestions?

Thanks again for the interest and interesting use case!

from typed-argument-parser.

Hi,
This is exactly what I am looking for and it would be great to have such a solution.
I'll try the hacky solution in the meantime, I'll let you know how it goes.
Thanks again!

from typed-argument-parser.

Hi again!
So I tried the hacky solution and it works with one drawback,
The help string doesn't work anymore. The first TapConfig().parse_args(known_only=True) catches the --help argument and outputs the help message:

usage: parameters.py [--config_file CONFIG_FILE] [-h]

optional arguments:
  --config_file CONFIG_FILE
                        (str, default=None)
  -h, --help            show this help message and exit

Process finished with exit code 0

is there any way to tell TapConfig to ignore --help/-h?

Thanks!

from typed-argument-parser.

Hello,

Thank you so much for taking a look at the hack and for the follow up! I think that your addition is a nice improvement and might be the best solution we have for now. I'll keep this issue up and hopefully work on implementing the real solution some time soon. Thank you again for raising this interesting issue and for your interest in Tap!

from typed-argument-parser.

Hi,
Do you think that proper support for configuration files could be implemented?

from typed-argument-parser.

Wouldn't the loading from a config file be counter to the main thrust of Tap? Loading from a file is inherently a dynamic event, available only at runtime, so you lose the typing support in your IDE that you get from the static typing.

from typed-argument-parser.

Well no, just as using '--arg value' in command line doesn't lose the typing support.
It is just a more convenient way to keep a lot of '--args' in some form of textual file and not have to pass all of them in command line. Something in the form of: https://pypi.org/project/ConfigArgParse/ f

from typed-argument-parser.

My mistake. I had thought you were using the files for the Parser definitions, not the values. Then yes, that would be a nice feature.

from typed-argument-parser.

Hi @arbellea,

@swansonk14 and I have implemented configuration files in this release. It only support very simple config file formats, but this may easily be extended in the future. Let us know what you think!

from typed-argument-parser.

Hi,

I just tried out the multiple json loading functionality with a toy example, but it throws an error.
But apart from this minor issue I really like Tap, thank you for creating it! 😄

Code:

from tap import Tap

class MyFavParser(Tap):
    cat: int = 4
    mouse: float = 0.2

json_path = "./test.json"
opt = MyFavParser().parse_args(['--cat', '2'])
print(opt)
opt.save(json_path)
opt = opt.load(json_path)  # Works like a charm
print(opt)
opt = MyFavParser(config_files=[json_path]).parse_args()  # Has some issues

Outcome:

Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tap import Tap
>>> class MyFavParser(Tap):
...     cat: int = 4
...     mouse: float = 0.2
...
>>> json_path = "./test.json"
>>> opt = MyFavParser().parse_args(['--cat', '2'])
>>> print(opt)
{'cat': 2, 'mouse': 0.2}
>>> opt.save(json_path)
>>> opt = opt.load(json_path)
>>> print(opt)
{'cat': 2, 'mouse': 0.2}
>>> opt = MyFavParser(config_files=[json_path]).parse_args()
usage: [--mouse MOUSE] [--cat CAT] [-h]
: error: unrecognized arguments: { "cat": 2, "mouse": 0.2, "reproducibility": { "command_line": "python ", "git_has_uncommitted_changes": true, "git_root": "/CenterNet", "git_url": "https://github.conti.de/uib58005/CenterNet/tree/4e0bbcf00de96f0abbc2576ea11d950a4b07df80", "time": "Thu Dec 10 13:04:11 2020" } }

from typed-argument-parser.

Hi @hoboc,

Great to hear you're a happy Tapper! Right now, we only support configuration files in the format of

--arg1 value1
--arg2 value2

We don't support JSON config files. I think that's the problem. It shouldn't be too difficult to support loading JSON config files in the future though! In fact, most of the action would happen here:

Let us know if it works for you with files of the right format and if loading from JSON is important to you.

from typed-argument-parser.

Hi @martinjm97 ,

Sorry, I misunderstood the concept. Thanks for the quick reply! Actually this format might also be fine for my use-case.

from typed-argument-parser.

@hoboc,
No worries :). Great to hear that the system as-is works for you! Happy Tapping!

from typed-argument-parser.

Looks like it's working. Additional functionality support for config file formats can be new issues (enhancements).

from typed-argument-parser.

@martinjm97 I think it would be really nice if you supported loading json or yaml files! As well as loading config files specified as an argument theirselves.

Now I'm doing something like this, which is obviously less nice.

parser = argparse.ArgumentParser()
parser.add_argument('--cfg', type=str)
tmp_cfg = parser.parse_args()

with open(tmp_cfg.cfg, 'r') as f:
    dict_cfg = yaml.load(f, Loader=yaml.FullLoader)

cfg = TapConfig()
cfg.from_dict(dict_cfg)
cfg.process_args()

from typed-argument-parser.

Hi!
Sorry for the late reply :)
Thanks for your addition, do you think that it will be possible to add the config path as an argument and not as a hard coded path?
Thanks

from typed-argument-parser.

@arbellea and @rubencart,

As you mentioned in #39, that's where that issue lives. We'll hopefully work on it soon.

--Kyle and Jesse

from typed-argument-parser.