minydra ๐ฆ
Minimal Python command-line parser inspired by Facebook's Hydra + dot-accessible nested dictionaries.
Easily parse arbitrary arguments from the command line without dependencies:
pip install minydra
minydra
is tested on Python 3.7
, 3.8
and 3.9
.
Getting Startedย ย โขย Forcing typesย ย โขย MinyDictย ย โขย Save configย ย โขย Prevent typosย ย โขย Use default configsย ย โขย Examples
Getting Started
from minydra.parser import Parser
if __name__ == "__main__":
parser = Parser(
verbose=0, # print received args
allow_overwrites=False, # allow repeating args in the command-line
warn_overwrites=True, # warn repeating args if they are allowed
parse_env=True, # get environment variable
warn_env=True, # warn if an environment variable is specified but not found
defaults=None, # path to a MinyDict-loadable dictionary of default values for the args
strict=True, # if `defaults` is provided, whether to allow new keys in the command-line
# or restrict to `defaults`' keys
keep_special_kwargs=True, # `defaults` and `strict` can be set from the command-line
# with `@defaults=` and `@strict=`. This argument decides if
# you want to keep those keys in the final arguments.
)
args = parser.args.pretty_print().resolve().pretty_print() # notice .resolve() transforms dotted.keys into nested dicts
from minydra import resolved_args
if __name__ == "__main__":
args = resolved_args()
args.pretty_print()
examples/demo.py
examples/demo.json
from minydra import MinyDict, resolved_args
from pathlib import Path
if __name__ == "__main__":
# parse arbitrary args in 1 line
args = resolved_args()
# override default conf
if args.default:
args = MinyDict.from_json(args.default).update(args)
# protect args in the rest of the code execution
args.freeze()
# print the args in a nice orderly fashion
args.pretty_print()
# access args with dot/attribute access
print(f'Using project "{args.log.project}" in {args.log.outdir}')
# save configuration
args.to_json(Path(args.log.outdir) / f"{args.log.project}.json")
import minydra
from minydra.dict import MinyDict
@minydra.parse_args(verbose=0, allow_overwrites=False) # Parser's init args work here
def main(args: MinyDict) -> None:
args.resolve().pretty_print()
if __name__ == "__main__":
main()
Parsing
- Simple strings are parsed to
float
andint
automatically. - A single keyword will be interpreted as a positive flag.
- A single keyword starting with
-
will be interpreted as a negative flag. - If
parse_env
isTrue
, environment variables are evaluated.
$ python examples/decorator.py outdir=$HOME/project save -log learning_rate=1e-4 batch_size=64
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ batch_size : 64 โ
โ learning_rate : 0.0001 โ
โ log : False โ
โ outdir : /Users/victor/project โ
โ save : True โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
- dotted keys will be resolved to nested dictionary keys:
$ python examples/decorator.py server.conf.port=8000
โญโโโโโโโโโโโโโโโโโโโโโฎ
โ server โ
โ โconf โ
โ โ โport : 8000 โ
โฐโโโโโโโโโโโโโโโโโโโโโฏ
- Using
ast.literal_eval(value)
,minydra
will try and parse more complex values for arguments as lists or dicts. Those should be specified as strings:
$ python examples/decorator.py layers="[1, 2, 3]" norms="{'conv': 'batch', 'epsilon': 1e-3}"
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ layers : [1, 2, 3] โ
โ norms : {'conv': 'batch', 'epsilon': 0.001} โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Forcing types
Adding ___<type>
to a key will force this type to the value. Notice how 01
is parsed to an integer 1
but 04
is parsed to a string (as specified) "04"
, and hello
is parsed to a list
, not kept as a string
$ python examples/decorator.py n_jobs___str=04 job=01 chips___list=hello
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ chips : ['h', 'e', 'l', 'l', 'o'] โ
โ job : 1 โ
โ n_jobs : 04 โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Known types are defined in Parser.known_types
and the separator (___
) in Parser.type_separator
In [1]: from minydra import Parser
In [2]: Parser.known_types
Out[2]: {'bool', 'float', 'int', 'str'}
In [3]: Parser.type_separator
Out[3]: '___'
Command-line configuration
You can configure the Parser
from the command-line using special @
arguments. In other words, all __init__(self, ...)
arguments can be set from the command-line with @argname=new_value
.
In particular if you run python examples/decorator.py @defaults=./examples/demo.json
you will see:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ @defaults : ./examples/demo.json โ
โ log โ
โ โlogger โ
โ โ โlog_level : DEBUG โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/path โ
โ โproject : demo โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
But if you add @strict=false @keep_special_kwargs=false
you will now have:
$ python examples/decorator.py @defaults=./examples/demo.json @strict=false @keep_special_kwargs=false
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ log โ
โ โlogger โ
โ โ โlog_level : DEBUG โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/path โ
โ โproject : demo โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
(you need to have @strict=false
since @keep_special_kwargs
is unknown in demo.json
. It would not be the case if strict=false
had been used in the script itself (but it can be overridden from the command-line!))
MinyDict
Minydra's args are a custom lightweight wrapper around native dict
which allows for dot access (args.key
), resolving dotted keys into nested dicts and pretty printing sorted keys in a box with nested dicts indented. If a key does not exist, it will not fail, rather return None (as dict.get(key, None)
).
a MinyDict
inherits from dict
so usual methods work .keys()
, .items()
etc.
In [1]: from minydra.dict import MinyDict
In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).pretty_print(); args
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ foo : bar โ
โ yes.no.maybe : idontknow โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Out[2]: {'foo': 'bar', 'yes.no.maybe': 'idontknow'}
In [3]: args.resolve().pretty_print(); args
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ foo : bar โ
โ yes โ
โ โno โ
โ โ โmaybe : idontknow โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Out[3]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [4]: args.yes.no.maybe
Out[4]: "idontknow"
In [5]: "foo" in args
Out[5]: True
In [6]: "rick" in args
Out[6]: False
In [7]: args.morty is None
Out[7]: True
In [8]: args.items()
Out[8]: dict_items([('foo', 'bar'), ('yes', {'no': {'maybe': 'idontknow'}})])
Dumping/Loading
You can save and read MinyDict
to/from disk in 3 formats: json
and pickle
without dependencies, yaml
with the PyYAML
dependency (pip install minydra[yaml]
).
Methods to_pickle
, to_json
and to_yaml
have 3 arguments:
file_path
as astr
orpathlib.Path
which is resolved:- expand env variable (
$MYDIR
for instance) - expand user (
~
) - make absolute
- expand env variable (
return_path
which defaults toTrue
. If those methods return the path of the created fileallow_overwrites
which defaults toTrue
. IfFalse
andpath
exists, aFileExistsError
will be raised. Otherwise creates/overwrites the file atfile_path
verbose
which defaults to0
. If>0
prints the path of the created object
Note:
to/from_yaml
will fail with aModuleNotFoundError
ifPyYAML
is not installed.- the
json
standard does not accept ints as keys in dictionaries so{3: 2}
would be dumped -- and therefore loaded -- as{"3": 2}
.
In [1]: from minydra.dict import MinyDict
In [2]: args = MinyDict({"foo": "bar", "yes.no.maybe": "idontknow"}).resolve(); args
Out[2]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [3]: json_file_path = args.to_json("./args.json")
In [4]: yaml_file_path = args.to_yaml("./args.yaml")
In [5]: pkl_file_path = args.to_pickle("./args.pkl")
In [6]: _ = args.to_json("./args.json", verbose=1) # verbose argument prints the path
Json dumped to: /Users/victor/Documents/Github/vict0rsch/minydra/args.json
In [7]: MinyDict.from_json("args.json")
Out[7]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}
In [8]: assert (
MinyDict.from_yaml(yaml_file_path)
== MinyDict.from_json(json_file_path)
== MinyDict.from_pickle(pkl_file_path)
== args
)
python examples/dumps.py path="./myargs.pkl" format=pickle cleanup
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ cleanup : True โ
โ format : pickle โ
โ path : ./myargs.pkl โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Dumped args to /Users/victor/Documents/Github/vict0rsch/minydra/myargs.pkl
Cleaning up
Strict Mode
To prevent typos from the command-line, the MinyDict.update
method has a strict mode: updating a MinyDict
with another one using strict=True
will raise a KeyError
if the key does not already exist:
from minydra import MinyDict, resolved_args
if __name__ == "__main__":
# parse arbitrary args in 1 line
args = resolved_args()
# override default conf
if args.default:
path = args.default
# delete otherwise it will be used to update the conf which does not have
# "default" as a key, therefore raising a KeyError in strict mode
del args.default
args = MinyDict.from_json(path).update(args, strict=True)
args.pretty_print()
No typo:
$ python examples/strict.py default=./examples/demo.json log.logger.log_level=INFO
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ log โ
โ โlogger โ
โ โ โlog_level : INFO โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/path โ
โ โproject : demo โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Typo:
$ python examples/strict.py default=./examples/demo.json log.logger.log_leveel=INFO
Traceback (most recent call last):
File "/Users/victor/Documents/Github/vict0rsch/minydra/examples/strict.py", line 13, in <module>
args = MinyDict.from_json(path).update(args, strict=True)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
self[k].update(v, strict=strict)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
self[k].update(v, strict=strict)
File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 100, in update
raise KeyError(
KeyError: 'Cannot create a non-existing key in strict mode ({"log_leveel":INFO}).'
Using default configurations
The minydra.Parser
class takes a defaults=
keyword argument. This can be:
- a
str
or apathlib.Path
to ajson
yaml
orpickle
file thatminydra.MinyDict
can load (from_X
) - a
dict
or aminydra.MinyDict
- a
list
of the above types, in which case the resulting defaults will be the result of sequential updates from those defaults, enabling hierarchical defaults (first defaults are the starting point, then each subsequent defaults updates it)
When defaults
is provided, the resulting minydra.MinyDict
serves as a reference for the arguments parsed from the command-line:
- If you setup the parser with
strict=True
, arguments from the command-line will still have a higher priority but they will have to be present in thedefaults
to prevent typos or unknown arguments (see strict mode) - arguments not present in the command-line with fallback to values in
defaults
defaults
can actually be a list
and the update order is the same as the list's. For instance:
In [1]: from minydra import Parser
In [2]: Parser(defaults=["./examples/demo.json", "./examples/demo2.json"]).args.pretty_print();
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ log โ
โ โlogger โ
โ โ โlog_level : INFO โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/other/path โ
โ โproject : demo โ
โ new_key : 3 โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
If you need to set defaults from the command-line, there's a special @defaults
keyword you can use:
$ python examples/decorator.py @defaults=./examples/demo.json
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ @defaults : ./examples/demo.json โ
โ log โ
โ โlogger โ
โ โ โlog_level : DEBUG โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/path โ
โ โproject : demo โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
$ python examples/decorator.py @defaults="['./examples/demo.json', './examples/demo2.json']"
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ @defaults : ['./examples/demo.json', './examples/demo2.json'] โ
โ log โ
โ โlogger โ
โ โ โlog_level : INFO โ
โ โ โlogger_name : minydra โ
โ โoutdir : /some/other/path โ
โ โproject : demo โ
โ new_key : 3 โ
โ verbose : False โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
pretty_print
Prints the MinyDict
in a box, with dicts properly indented. A few arguments:
indents
, which defaults to2
: the amount of indentation for nested dictionariessort_keys
, which defaults toTrue
: whether or not to alphabetically sort the keys before printing
to_dict
To produce a native Python dict
, use args.to_dict()
Protected attributes
MinyDict
's methods (including the dict
class's) are protected, they are read-only and you cannot therefore set attributes with there names, like args.get = 2
. If you do need to have a get
argument, you can access it through items: args["get"] = 2
.
Try with examples/protected.py
:
python examples/protected.py server.conf.port=8000 get=3
โญโโโโโโโโโโโโโโโโโโโโโฎ
โ get : 3 โ
โ server โ
โ โconf โ
โ โ โport : 8000 โ
โฐโโโโโโโโโโโโโโโโโโโโโฏ
<built-in method get of MinyDict object at 0x100ccd4a0>
3
dict_items([('get', 3), ('server', {'conf': {'port': 8000}})])
{'conf': {'port': 8000}}
Tests
Run tests and pre-commit checks (isort
, black
, flake8
) with
$ pip install -r requirements-test.txt
$ pre-commit run --all-files
$ pytest -vv --cov=minydra tests/