fidelity / spock Goto Github PK

Describe the bug
The Link to "Spock As a Drop In Replacement For Argparser" is broken in docs

To Reproduce
Steps to reproduce the behavior:

1. Go to https://fidelity.github.io/spock/Quick-Start
2. Scroll down to bottom of page
3. Click on "here" in the sentence See the docs/example here.

Expected behavior
Docs

Actual Behavior

Page Not Found
We could not find what you were looking for.

Please contact the owner of the site that linked you to the original URL and let them know their link is broken.

@spock
class BaseKernelConfig:
    morph_kernels_thickness: int = 1


@spock
class LineKernelConfig(BaseKernelConfig):
    horizontal_scale_factor: float = 0.95
    vertical_scale_factor: float = 0.95


@spock
class RectangleKernelConfig(BaseKernelConfig):
    morph_tolerance: float = 0.1  # 0.2


class MorphKernelConfig(Enum):
    line = LineKernelConfig
    rectangle = RectangleKernelConfig


@spock
class ImageConfig:
    kernel_config: MorphKernelConfig = RectangleKernelConfig()



@spock
class ExtractionConfig:
    image_config: ImageConfig = ImageConfig()

attrs_class = ConfigArgBuilder(
        CheckboxExtractionConfig, ImageConfig, LineKernelConfig, RectangleKernelConfig,
        desc='I am a description',
    ).generate()

This will generate the following error:

Traceback (most recent call last):
  File ".../spock/spock/builder.py", line 115, in __init__
    raise ValueError(e)
ValueError: kernel_config must be in [<class 'spock.backend.config.LineKernelConfig'>, <class 'spock.backend.config.RectangleKernelConfig'>]

Seemingly the double nested class reference doesn't get mapped to the class Enum correctly... might be an ordering issue or an incorrect recursive mapping call.

from enum import Enum
from spock.config import spock
from typing import List

class MyEnum(Enum):
    value_1 = 'value_1'
    value_2 = 'value_2'

@spock
class ExampleConfig:
    example: List[MyEnum]

This will fail as the recursive check for List type currently doesn't support Enum

Both spock and spock-config require one to import spock. But this would lead to a name clash if both were installed.

Shifting away from dataclasses since the current implementation is kludge consisting of the current Python 3.7 dataclass implementation overridden to only allow kwarg defs and prevent an ordering requirement (which affects inheritance) of default values.

Possible Solutions:

• attrs - https://github.com/python-attrs/attrs

• pydantic - https://github.com/samuelcolvin/pydantic/

Leaning towards attrs as they provide a bit more functionality.

Hi :) Spock is cool and I like what I see so far. I appreciate the work you put into Spock and after using Hydra for over a year I want to check out Spock for configuration. However, I am having some troubles with it.

Describe the bug
Cannot load in a SavePath for the spock configuration used during a run from the config.yaml provided. Right now my ugly workaround is to hardcode the path with
save_path = f"{environ.get('PWD')}/runs"
and
SpockBuilder.save(user_specified_path=save_path)

To Reproduce
Steps to reproduce the behavior:

1. instantiate spock namespace with SpockBuilder(somecfg, desc="somecfg", config=['src/config.yaml']).save(file_extension=".toml").generate()`
2. ValueError Exception was raised: ValueError: Save did not receive a valid path from: (1) markup file(s) or (2) the keyword arg user_specified_path

Expected behavior
I want to specify a mutable path for outputs/saving in the external config.yaml file. Having to do this hardcoded in user_specified_path inside the main file is an inferior alternative.

Desktop (please complete the following information):

• OS: macOS Monterey 12.3.1
• Browser: chrome
• Version: 100.0

Additional context
I hope I am missing it in the docs, but I cannot find it here: SavePath
Here it says We simply specify a SavePath in a spock config, which is a special argument type that is used to set the save path from a configuration file., which does not work for me. I also do not know where to import this SavePath type from.

Thanks in advance for your help,
Sven

Allow for nested configurations (with the full suite of type-checking, etc). The syntax should look something like this:

@spock_config 
ObjectConfig:
    param_1a: int 
    param_1b: float

@spock_config
RepeatedObjectConfig:
    param_2: float

@spock_config
ParentConfig:
    param_3: int
    object: ObjectConfig
    repeated_objects: List[RepeatedObjects]

The corresponding YAML will look like:

param_3: 5
object:
    param_2: 3.14
repeated_objects:
    - param_1a: 2
      param_1b: 2.71
    - param_1a: 4
      param_1b: 1.51

The corresponding command line args (#5 ) will look like:

./executable.py --object.param_2 3.14 

I'm not sure about the syntax for lists of configs in the command line, but we could say those have to be defined in YAMLs

Currently the argparser that handles this falls back on the class attribute type and not the iterable type (List or Tuple). This means you can't override repeated class attributes at the command line currently.

Composition currently doesn't check for circular dependencies which makes things catch 🔥

hi,
i still can't figure out how SavePath is working. The keyword SavePath is not recognised ...
i don't have to import something, right?

Describe the bug
When trying to reproduce an example from https://fidelity.github.io/spock/advanced_features/Advanced-Types#nested-list-tuple-and-dict-types, I get an error saying that Subscripted generics cannot be used with class and instance checks.

To Reproduce
Steps to reproduce the behavior:

1. create file as follow:

from typing import List, Tuple

from spock import SpockBuilder
from spock.config import spock


@spock
class TypeConfig:
    param: Tuple[List[str], List[str]] = ([],[])

config = SpockBuilder(TypeConfig).generate()

2. on the command line: python <file>
3. See error

Expected behavior
No error. but I get

usage: spocktest.py -c [--config] config1 [config2, config3, ...]



configuration(s):

  TypeConfig
    param    Tuple[List[str], List[str]]     (default: ([], []))

Spock class `TypeConfig` could not be instantiated -- attrs message: Subscripted generics cannot be used with class and instance checks
Traceback (most recent call last):
  File "[...]/lib/python3.10/site-packages/spock/backend/field_handlers.py", line 878, in recurse_generate
    spock_instance = spock_cls(**fields)
  File "<attrs generated init spock.backend.config.TypeConfig>", line 6, in __init__
  File "[...]/lib/python3.10/site-packages/attr/validators.py", line 388, in __call__
    self.member_validator(inst, attr, member)
  File "[...]/lib/python3.10/site-packages/attr/validators.py", line 102, in __call__
    if not isinstance(value, self.type):
  File "[...]/lib/python3.10/typing.py", line 994, in __instancecheck__
    return self.__subclasscheck__(type(obj))
  File "[...]/lib/python3.10/typing.py", line 997, in __subclasscheck__
    raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[...]/spocktest.py", line 11, in <module>
    config = SpockBuilder(TypeConfig).generate()
  File "[...]/lib/python3.10/site-packages/spock/builder.py", line 150, in __init__
    raise e
  File "[...]/lib/python3.10/site-packages/spock/builder.py", line 134, in __init__
    self._arg_namespace = self._builder_obj.generate(self._dict_args)
  File "[...]/lib/python3.10/site-packages/spock/backend/builder.py", line 138, in generate
    spock_space_kwargs = self.resolve_spock_space_kwargs(self._graph, dict_args)
  File "[...]/lib/python3.10/site-packages/spock/backend/builder.py", line 160, in resolve_spock_space_kwargs
    spock_instance, special_keys = RegisterSpockCls.recurse_generate(
  File "[...]/lib/python3.10/site-packages/spock/backend/field_handlers.py", line 880, in recurse_generate
    raise _SpockInstantiationError(
spock.exceptions._SpockInstantiationError: Spock class `TypeConfig` could not be instantiated -- attrs message: Subscripted generics cannot be used with class and instance checks

Environment (please complete the following information):

• OS: Ubuntu 20.04
• Python version: 3.10.4
• Other installed packages: torch

Right now the SpockBuilder class has a method spockspace_2_dict to convert an entire Spockspace to a dict but it would be helpful to have another utility function that converts a single spock object namespace into a dictionary

Describe the bug
Default values of nested list with items is a spock class.

I think the tested default values are not of the good format in the tests. Thus, the resulting code is not good.

Should they be respectively?
nested_list_def: List[NestedListStuff] = [NestedListStuff]

and

nested_list_opt_def: Optional[List[NestedListStuff]] = [NestedListStuff]

I could also want to specify something like that in both cases:

nested_list_def: List[NestedListStuff] = [NestedListStuff, NestedListStuff(one=1, two="two"), NestedListStuff(one=3, two="four")]

or

nested_list_def: List[NestedListStuff] = [NestedListStuff(one=1, two="two"), NestedListStuff(one=3, two="four")]

I am wondering why NestedListStuff works without instantiation because there is no default value for it.

Currently the output objects are not able to be pickled since the classes are dynamically created from the class definitions.

See below...

  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'spock.backend.attr.config.XXXX'>: attribute lookup XXXX on spock.backend.attr.config failed

Simple solution is to allow conversion of the attrs class to a dictionary or namespace after the classes are instantiated since then pickle should be able to deal with the more basic type.

When spock is used with another python lib that calls command line args (e.g. FastAPI) the -c flag will fail as the other library can be making implicit calls to cmd line args.

Suggested solution is to add a keyword override to prevent the arg_parser call and force the config kwarg specification.

Is there a reason why dependencies are pinned to a specific version?
Could it be relaxed to major versions? Assuming almost every project uses semver.

Describe the bug
The value provided in a config file is not propagated to the nested instances of the config. However, when provided directly via command line, it does work.

To Reproduce
This is a simplified config and code:

from spock.builder import ConfigArgBuilder
from spock.config import spock

@spock
class TrainProcess:
    epochs: int = 100

@spock
class Train:
    train_process: TrainProcess = TrainProcess()



def main():
    attrs_class = ConfigArgBuilder(
        Train, TrainProcess,
        desc='',
    ).generate()
    print(attrs_class)


if __name__ == '__main__':
    main()

The provided command is:
train --config best_params.yaml

with best_params.yaml

epochs: 1

Expected behavior
--config values should be propagated to nested config instances like values from command line

Screenshots
Snipping of debug pane for attrs_class object

Additional context
spock-config==2.2.6

Test sampling of Ax now throws an exception:

ax.exceptions.core.DataRequiredError: All trials for current model have been generated, but not enough data has been observed to fit next model. Try again when more data are available.

Should subvert this as this test is just making sure draws stay within defined ranges

ValueError raised by attrs validator since the class type is mismatched.

Describe the bug
I would like to use generic types such as list[int] instead of from typing import List; List[int].

I do know that the example in the documentation explicitly show that I need to import List from typing, but ever since python 3.9, it is preferred to directly use list as type annotation.

To Reproduce
Create file simple.py

from spock import SpockBuilder, spock

@spock
class BasicConfig:
    most_fancy_parameter: list[int]

def main():
    config = SpockBuilder(BasicConfig, desc="Quick start example").generate()
    print(config)

if __name__ == "__main__":
    main()

Run the script,

python simple.py --BasicConfig.most_fancy_parameter "[128, 128]"    

I got:

spock.exceptions._SpockInstantiationError: Spock class `BasicConfig` could not be instantiated -- attrs message: isinstance() argument 2 cannot be a parameterized generic

Environment (please complete the following information):

• OS: Mac 10.13
• Python version: 3.10
• Spock version: 3.0.2

Describe the bug

When loading a trained model together with a Spock config an error is thrown:

usage: ipykernel_launcher.py [-c CONFIG [CONFIG ...]] [-h]
[--ModelConfig.save_path MODELCONFIG.SAVE_PATH]
[--ModelConfig.lr MODELCONFIG.LR]
pykernel_launcher.py: error: unrecognized arguments: --ip=127.0.0.1 --stdin=9028 ...

To Reproduce
Steps to reproduce the behavior:
Load config inside of a jupyter notebook:

config = SpockBuilder(ModelConfig, desc='Quick start example').save(file_name='config', file_extension='.yaml', create_save_path=True).generate()
args = config.ModelConfig

--> Results in error.

Expected behavior
The command line should probably be ignored inside a jupyter notebook.

Environment (please complete the following information):

• OS:Ubuntu 20.04

Is this expected behavior? How could I circumvent this problem?

Is your feature request related to a problem? Please describe.
None

Describe the solution you'd like
I want to be able to check whether a Python object is a spock config or not. This will help me distinguish between dict, NamedTuple, and spock.config objects, all of which may contain hyperparameters for different models in a legacy codebase.

Describe alternatives you've considered
None

Additional context
None

Is your feature request related to a problem? Please describe.

In dev mode you often try code before a commit. Spock already saves dirtyness, but if dirty, it would be nice to save dirty files. Either directly or perhaps via a custom ref.

Describe the solution you'd like
Something that captures dirty files somehow, either in git or directly

Describe alternatives you've considered
One could always commit...

Some sort of Adapter pattern to interface with some of the more common hyper-parameter tuning libraries.

NNI: https://github.com/microsoft/nni
Optuna: https://github.com/optuna/optuna
Talos: https://github.com/autonomio/talos
HyperOpt: https://github.com/hyperopt/hyperopt

Katib (supports k8s Job and MPIJob CRD): https://github.com/kubeflow/katib

spock-config is a nice alternative to argparse. However, we have to add namespace name in argument (e.g. --BaseConfig.lr=value). That is sometimes cumbersome. If there is a way to set default namespace, the command-line arguments can work more like argparse.

Of course, to avoid conflict, some arguments (h,help,c,config) cannot be in such default namespace.

Allow command line arguments to define configuration parameters. These arguments should override anything defined in YAMLs.

Describe the bug
Cannot specify the value of a nested parameter via CLI. Unknow argument

To Reproduce

@spock
class TrainProcessConfig:
    """Common options for ax and train commands

    Attributes:
        cuda: Cuda device. If None, uses `CPU`.
        epochs: Maximum number of training epochs
        batch_size: Training batch size for gradient descent
    """

    cuda: Optional[int] = None
    epochs: int = 100
    batch_size: int = 32

@spock
class TrainConfig:
    """Options for train command

    Attributes:
        pw_scalar: Scalar that will multiply the pos_weights
        train_process: Training process configuration

    """

    pw_scalar: float = 1.0
    train_process: TrainProcessConfig = TrainProcessConfig()

Expected behavior

I am trying to execute the following configuration and overwrite the epochs argument with the cli. I am trying:

train --TrainConfig.train_process.TrainProcessConfig.epochs 1
or
train --TrainConfig.train_process.epochs 1

without any success what am I doing wrong?

I always get:

usage: train [-c CONFIG [CONFIG ...]] [-h]
[--TrainConfig.pw_scalar TRAINCONFIG.PW_SCALAR] 
[--TrainConfig.train_process TRAINCONFIG.TRAIN_PROCESS] 
train: error: unrecognized arguments: --TrainConfig.train_process.TrainProcessConfig.epochs

Is your feature request related to a problem? Please describe.
Cannot access e.g., environment variables in config (can I access and compose from other config variables and append to them?). This would be the oc.env resolver.

Maybe a separate feature request but Spock already offers some (or all, haven't checked?) of the oc.decode functionality.

Describe the solution you'd like
save_path: ${oc.env:PWD}/runs

and

some_value: path_to_a_base_dir
composed_value: ${path_to_a_base_dir}/some_other_dir
composed_value_building_on_that: ${composed_value}/another_sub_dir

More examples found in OmegaConf.

Describe alternatives you've considered
Nick mentioned this might be done with the underlying library attrs (?)

Additional context
Started discussing the idea in this issue

Thanks for considering to work on this :)
As I already mentioned in the other thread/issue: happy to help, but no idea where to begin and how to move into a worthwhile direction without guidance. Would love to work on this together though.

hi, the argument type "SavePath" is not recognised somehow? do i have to import something other then spock?
thanks in advance!

Pytest fails on the import from tests.configs_test

spock % pytest tests

tests/test_all.py:10: in <module>
    from tests.configs_test import *
E   ModuleNotFoundError: No module named 'tests'

Solution:

spock % touch tests/__init__.py
spock % pytest tests
================================================= test session starts ==================================================
platform darwin -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/mhancoh/Workspaces/python/spock
plugins: cov-2.10.1
collected 20 items                                                                                                     

tests/test_all.py ....................                                                                           [100%]

================================================== 20 passed in 0.68s ==================================================

Support for SM tuner within hyperparameter tuning support.

See HyperparameterTuner within the SM docs here

Is your feature request related to a problem? Please describe.
I want to be able to use the docstring of a class annotated with @spock to automatically show the relevant description of the variable when using the --help option from command line. This is similar to using the help parameter with argparse.

Describe the solution you'd like
Currently, an example information printed using the --help option looks like this:

USAGE:
  spock_test.py -c [--config] config1 [config2, config3, ...]
CONFIG:
  ModelConfig:
    n_features: int
    dropout: Optional[List[float]]
    hidden_sizes: Optional[Tuple[int]]

The new version could look like this, based on the docstring of the class:

USAGE:
  spock_test.py -c [--config] config1 [config2, config3, ...]
CONFIG:
  ModelConfig: (Configures a DNN model)
    n_features: int, the number of features
    dropout: Optional[List[float]], the amount of dropout
    hidden_sizes: Optional[Tuple[int]], the sizes of the hidden layers

As long as the user can see a description of the variables when using --help, the format doesn't matter.

Describe alternatives you've considered
None

Additional context
None

Portray seems to have limited dev support. Issues with code syntax highlighting haven't been addressed and the autodocs are not picking up @Property decorated attributes on classes.

Suggestion is to shift the docs to Docusaurus to remedy these issues

I have started a simple tutorial for my use case and exploring what spock offers.

from typing import List

from spock import spock, SpockBuilder, SavePath


@spock
class ConfigPath:
    """
    Configuration definition for handling of all paths in project.

    Attributes:
        save_path: where the loaded configuation file is saved
        data_dir: the main data directory
        output_dir: the main output directory
        config_dir: the main configs directory
    """
    save_path: SavePath
    data_dir: str
    output_dir: str
    config_dir: str


if __name__ == "__main__":
    # config = SpockBuilder(ConfigPath, desc="Example configuration for paths").generate()
    # print(config)

    config = SpockBuilder(ConfigPath, desc="Example configuration for paths").save(file_extension='.json').generate()
    print(config)

The corresponding configuration file I have create is the following

{
  "save_path": "C:\\example_project\\configs",
  "data_dir": "C:\\",
  "output_dir": "C:\\",
  "config_dir": "C:\\"
}

When ran with the command python <...>\config_path.py --config paths.json I receive the following exception:

Exception ignored in: <function Popen.__del__ at 0x00000258854A5550>
Traceback (most recent call last):
  File "C:\Users\kdecker\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 949, in __del__
    self._internal_poll(_deadstate=_maxsize)
  File "C:\Users\kdecker\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1348, in _internal_poll
    if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
OSError: [WinError 6] The handle is invalid

Though this does result in, from what I can tell, a correctly saved .toml file. It contains a header with various info about the machine, python, and git state, the [ConfigPath] parameters themselves, and a footer with the various python libraries in my environment.

This issue does not happen when not calling .save, I load the configuration file into a namespace and print it. This is all under Windows 10 system. Using spock-config 3.0.2.

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
I do not always want to specify a value for the --config argument and in some cases would like to use spock similar to argparse, relying on the default values and specifying argument values only when needed.

Describe alternatives you've considered
I am overriding the values in the config file by adding them to explicitly as arguments, but would prefer to avoid this usage.

Additional context

Is your feature request related to a problem? Please describe.
None

Describe the solution you'd like
I'd like to be able to specify a filename during the save() function to make it easier to read the config back in. Currently spock produces a guid by default, with no way of overriding it.

Describe alternatives you've considered
None

Additional context
None

Instead of continued legacy support for the sunsetting dataclass backend we should just simply build a connector that maps the old syntax to new so as to support older versions of code but with the new backend.

Edit: This will also allow for Python 3.6 support as it will only rely on Typing which was released in 3.6 instead of dataclasses which were 3.7

When passed an S3 path spock currently doesn't know how to deal with it (as it is not relative or absolute to the os via a mount).

Solution: Regex for 's3://' and then use S3 Utilities or boto3 to download the file to a local path that is readable. Same for saving as writing to S3 will fail in the same way as reading.