First of all, great project. I would like to understand if it's poss

Alternative ClassParsers about declarative-parser HOT 6 CLOSED

krassowski commented on July 18, 2024

Alternative ClassParsers

from declarative-parser.

Comments (6)

krassowski commented on July 18, 2024

It should be already possible or somehow possible with small tweaks to the parser. I will look into that today and let you know how to achieve such a result.

from declarative-parser.

krassowski commented on July 18, 2024

Let's start by understanding better what does your code do right now. Let's tweak it a bit so we don't have conflicting symbols (lowercase a and b in two places):

from declarative_parser import Parser
from declarative_parser.constructor_parser import ClassParser


class A:
    def __init__(self, a='a'):
        self.a = a


class B:
    def __init__(self, b='b'):
        self.b = b


class MainParser(Parser):
    a_parser = ClassParser(A)
    b_parser = ClassParser(B)


if __name__ == '__main__':
    parser = MainParser()
    namespace = parser.parse_args()
    print(namespace)

Now, when calling help you will see that the program accepts two sub-parsers (a_parser and b_parser), each of them taking a single, optional argument (a and b respectively):

$ python3 run.py -h
usage: run.py [-h] {a_parser,b_parser} ...

positional arguments:
  {a_parser,b_parser}
    a_parser           Accepts: a
    b_parser           Accepts: b

optional arguments:
  -h, --help           show this help message and exit

So you can already populate your namespace using:

$ python3 run.py a_parser --a 1 b_parser --b 2
Namespace(a_parser=Namespace(a='1'), b_parser=Namespace(b='2'))
# you can change the order of parsers,
# or omit one of them as all the declared arguments are optional

What you may want to do is to let the user jump right ahead, without the need to write a_parser and b_parser each time. This will work perfectly as long as you don't have any colliding arguments.
This could look like this:

$ python3 run.py --a 1 --b 3
Namespace(a='1', b='3')

To make it work, you need to define __pull_to_namespace_above__ = True on your parser. With the normal parser, you just specify this in the body of the sub-parser class definition:

class C(Parser):

    __pull_to_namespace_above__ = True
    c = Argument()

# and later on:
class MainParser(Parser):
    c_parser = C()

In the case of ClassParser, a little tweak of sublassing is needed instead:

from declarative_parser import Parser, Argument
from declarative_parser.constructor_parser import ClassParser


class A:
    def __init__(self, a='a'):
        self.a = a


class B:

    def __init__(self, b='b'):
        self.b = b


class TranslucentClassParser(ClassParser):
    __pull_to_namespace_above__ = True


class MainParser(Parser):
    a_parser = TranslucentClassParser(A)
    b_parser = TranslucentClassParser(B)


if __name__ == '__main__':
    parser = MainParser()
    namespace = parser.parse_args()
    print(namespace)

However, as you might have noticed, if you were to leave it as is, you would lose the nice separation of arguments in the final namespace - and it might prove difficult to separate them out again to construct the desired objects. Here is a way out:

To address this problem there is produce() method (see also: production pattern). It belongs to the Parser, so once again we need to modify sub-classed ClassParser:

class TranslucentClassParser(ClassParser):
    __pull_to_namespace_above__ = True

    def produce(self, unknown_args=None):
        shared_opts = self.namespace
        my_arguments = {
            key: shared_opts.__dict__.pop(key)
            for key in self.all_arguments
        }
        name = self.constructor.__name__
        setattr(shared_opts, name, my_arguments)
        return shared_opts

Here is how it would work:

$ python3 run.py --a 1 --b 3
Namespace(A={'a': '1'}, B={'b': '3'})

And finally there is nothing stopping you to call the constructor inside of the produce() method (just showing the relevant fragment):

        name = self.constructor.__name__
        instance = self.constructor(my_arguments)
        setattr(shared_opts, name, instance)
        return shared_opts

And the result is easy to forsee:

$ python3 alternatives.py --a 1 --b 3
Namespace(A=<__main__.A object at 0x7f54991149b0>, B=<__main__.B object at 0x7f5498e9b1d0>)

As for the exclusion... well it's still possible but with a bit more tweaking. Please see how I implemented parsing of methods in this CLI class (in the terminology of the project I am linking to, a method was like a standalone class implementing a particular method of analyzing data; you may think of it as a sub-program with shared interface; technically this was always "just a class". See: this abstract method definition, this method and this one for examples, and this two lines to see how it was used).

Does it answer your question or even helps a bit?
If you have any doubts, don't hesitate to ask, I'm happy to help.

from declarative-parser.

dcferreira commented on July 18, 2024

Wow thanks for that very complete explanation :)
While I definitely understand a lot more after reading it, I now think I was asking the wrong question.

Here's another attempt (warning long text ahead).
This is a simplified version of my final usecase (commented part is what I was planning on doing to make the arguments mandatory):

from declarative_parser import Parser
from declarative_parser.constructor_parser import ClassParser


class Dataset1:
    def __init__(self, d1_mandatory, d1_optional='d1_optional_val'):
        self.d1_mandatory, self.d1_optional = d1_mandatory, d1_optional


class Dataset2:
    def __init__(self, d2_optional='d2_optional_val'):
        self.d2_optional = d2_optional


class Method1:
    def __init__(self, m1_mandatory, m1_optional='m1_optional_val'):
        self.m1_mandatory, self.m1_optional = m1_mandatory, m1_optional


class Method2:
    def __init__(self, m2_optional='m2_optional_val'):
        self.m2_optional = m2_optional


class DatasetParser(Parser):
    d1 = ClassParser(Dataset1)
    d2 = ClassParser(Dataset2)


class MethodParser(Parser):
    m1 = ClassParser(Method1)
    m2 = ClassParser(Method2)


class MainParser(Parser):
    dataset = DatasetParser()
    method = MethodParser()

    # def produce(self, unknown_args):
    #     opts = self.namespace
    #
    #     assert 'dataset' in vars(opts)
    #     assert opts.dataset is not None
    #     assert opts.dataset.d1 is not None or opts.dataset.d2 is not None
    #
    #     assert 'method' in vars(opts)
    #     assert opts.method is not None
    #     assert opts.method.m1 is not None or opts.method.m2 is not None
    #
    #     return opts


if __name__ == '__main__':
    parser = MainParser()
    args = parser.parse_args()
    print(args)

In relation to the question in the first post, the old MainParser corresponds to DatasetParser/MethodParser here.

So, I have some datasets, and some methods, and each dataset/method has different configuration arguments, defined in classes.
I'd like that the user specify one dataset and one method, together with respective parameters.
Correct help messages would also be nice, but maybe not essential.

Example commands I expect, with expected output:

$ # Works as expected
$ python run.py dataset d1 d1_mandatory_val method m1 m1_mandatory_val
Namespace(dataset=Namespace(d1=Namespace(d1_mandatory='d1_mandatory_val', d1_optional='d1_optional_val'), d2=None), method=Namespace(m1=Namespace(m1_mandatory='m1_mandatory_val', m1_optional='m1_optional_val'), m2=None))
$ # Doesn't work as expected
$ python run.py dataset d2 method m2
Namespace(dataset=Namespace(d1=None, d2=Namespace(d2_optional='d2_optional_val')), method=Namespace(m1=None, m2=Namespace(m2_optional='m2_optional_val')))
# actually returns:
Namespace(dataset=Namespace(d1=None, d2=None), method=Namespace(m1=None, m2=None))

The help message of the main parser seems to hint that something is not as I expected right away:

$ python run.py -h
usage: run.py [-h] {dataset,method} ...

positional arguments:
  {dataset,method}
    dataset         Accepts:
    method          Accepts:

optional arguments:
  -h, --help        show this help message and exit

I would expect that the "Accepts" line includes "d1, d2"/"m1, m2" for the dataset/method, but this doesn't happen because the list is taken from the list of arguments of the parser, which is distinct from the list of parsers.
I would want the usage to be more like:

usage: run.py [-h] dataset {d1,d2} [d1_args] [d2_args] method {m1,m2} [m1_args] [m2_args]

and then I'd manually parse d1_args/d2_args, depending on whether the user chose d1/d2.

Additionally, when asking for the help for dataset, it doesn't give me the arguments for some reason:

$ python run.py dataset -h
usage: run.py dataset [-h]

optional arguments:
  -h, --help  show this help message and exit

The same happens if I replace ClassParser by TranslucentClassParser.

I'm running all this in python 3.7, in case the version matters.
Do you think it's possible to get the behavior described in the example commands above?
Thanks for your patience, if you've read this far :)

from declarative-parser.

krassowski commented on July 18, 2024

If you want to have an additional parser in between, you can easily address the problem of skipped parsers setting __skip_if_absent__ = False in a sub-class. By default, the sub-parser is being skipped if there are no arguments. We want to disable this behaviour:

class AlwaysPresentClassParser(ClassParser):
    __skip_if_absent__ = False

class DatasetParser(Parser):
    d1 = AlwaysPresentClassParser(Dataset1)
    d2 = AlwaysPresentClassParser(Dataset2)

class MethodParser(Parser):
    m1 = AlwaysPresentClassParser(Method1)
    m2 = AlwaysPresentClassParser(Method2)

$ python3 run2.py dataset d1 d1_mandatory_val method m1 m1_mandatory_val
Namespace(dataset=Namespace(d1=Namespace(d1_mandatory='d1_mandatory_val', d1_optional='d1_optional_val'), d2=Namespace(d2_optional='d2_optional_val')), method=Namespace(m1=Namespace(m1_mandatory='m1_mandatory_val', m1_optional='m1_optional_val'), m2=Namespace(m2_optional='m2_optional_val')))

Though this will cause problems when running:

$ python3 run2.py dataset d2 method m2
usage: run2.py [-h] [--d1_optional D1_OPTIONAL] d1_mandatory
run2.py: error: the following arguments are required: d1_mandatory

Obviously we want to skip only these parsers which are not referenced by the user:

from declarative_parser.parser import group_arguments

class ConditionalClassParser(ClassParser):
    __skip_if_absent__ = None     # will be assigned dynamically

class ChoiceParser(Parser):

    def parse_known_args(self, args):
        grouped_args, ungrouped_args = group_arguments(args, self.all_subparsers)

        for name, parser in self.subparsers.items():
            parser.__skip_if_absent__ = [name] != args[:1]

        return super().parse_known_args(args)

class DatasetParser(ChoiceParser):
    d1 = ConditionalClassParser(Dataset1)
    d2 = ConditionalClassParser(Dataset2)


class MethodParser(ChoiceParser):
    m1 = ConditionalClassParser(Method1)
    m2 = ConditionalClassParser(Method2)

$ python3 run3.py dataset d1 d1_mandatory_val method m1 m1_mandatory_val
Namespace(dataset=Namespace(d1=Namespace(d1_mandatory='d1_mandatory_val', d1_optional='d1_optional_val'), d2=None), method=Namespace(m1=Namespace(m1_mandatory='m1_mandatory_val', m1_optional='m1_optional_val'), m2=None))

$ python3 run3.py dataset d2 method m2
Namespace(dataset=Namespace(d1=None, d2=Namespace(d2_optional='d2_optional_val')), method=Namespace(m1=None, m2=Namespace(m2_optional='m2_optional_val')))

$ python3 run3.py dataset d1 method m2
usage: run3.py [-h] [--d1_optional D1_OPTIONAL] d1_mandatory
run3.py: error: the following arguments are required: d1_mandatory

I hope that this works for you.

from declarative-parser.

krassowski commented on July 18, 2024

Another way of doing that would be using simple arguments and dynamically creating parsers:

from declarative_parser import Parser, Argument
from declarative_parser.constructor_parser import ClassParser

class Dataset1:
    def __init__(self, d1_mandatory, d1_optional='d1_optional_val'):
        self.d1_mandatory, self.d1_optional = d1_mandatory, d1_optional

class Dataset2:
    def __init__(self, d2_optional='d2_optional_val'):
        self.d2_optional = d2_optional

class Method1:
    def __init__(self, m1_mandatory, m1_optional='m1_optional_val'):
        self.m1_mandatory, self.m1_optional = m1_mandatory, m1_optional

class Method2:
    def __init__(self, m2_optional='m2_optional_val'):
        self.m2_optional = m2_optional

datasets = dict(
    d1=Dataset1,
    d2=Dataset2
)

methods = dict(
    m1=Method1,
    m2=Method2
)

def sub_parser(constructor, unknown_args):

    parser = ClassParser(constructor)

    # parse arguments
    options, remaining_unknown_args = parser.parse_known_args(unknown_args)

    for argument in unknown_args[:]:
        if argument not in remaining_unknown_args:
            unknown_args.remove(argument)

    return options
    # or with as a ready-to-go class
    # return parser.constructor(**vars(options))

class MainParser(Parser):
    dataset = Argument(choices=datasets)
    method = Argument(choices=methods)

    def produce(self, unknown_args):
        options = self.namespace

        options.method = sub_parser(methods[options.method], unknown_args)
        options.dataset = sub_parser(datasets[options.dataset], unknown_args)

        return options

if __name__ == '__main__':
    parser = MainParser()
    args = parser.parse_args()
    print(args)

python3 run.py --dataset d1 d1_mandatory_val --method m1 m1_mandatory_val
Namespace(dataset=Namespace(d1_mandatory='m1_mandatory_val', d1_optional='d1_optional_val'), method=Namespace(m1_mandatory='d1_mandatory_val', m1_optional='m1_optional_val'))

$ python3 run.py --dataset d2 --method m2
Namespace(dataset=Namespace(d2_optional='d2_optional_val'), method=Namespace(m2_optional='m2_optional_val'))

This is the same method as used in the project I linked to previously. There might be some adjustments needed to make the help pages work nicely (feel free to reuse the code from the CLI class).

from declarative-parser.

dcferreira commented on July 18, 2024

Thank you very much! :D
Both solutions work, I like the cli syntax on the second better.

I don't understand what it's doing yet, but I'll look at your example and try to fix the help pages.
I'll get back to you if needed, but I guess this topic is closed :)

from declarative-parser.

Alternative ClassParsers about declarative-parser HOT 6 CLOSED

Comments (6)

Related Issues (5)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent