nicholasjng / shelf Goto Github PK

Currently, we unconditionally prepend self.prefix before the rpath in Shelf.(get|put).

This will of course break if self.prefix is not the empty string and the rpath is already fully qualified (i.e. has leading protocol and separator).

Hence, we need to omit the prefix if the input path is already fully qualified.

Currently, types that require multiple distinct files to load (think neural networks with a parameter file and a hyperparameter file) are not supported, and if they are, it's probably by accident.

def load_from_disk(param_file: str, hparam_file: str) -> MyModel:  # needs both files to load successfully
    params = load_params(param_file)
    hparams = load_hparams(hparam_file)
    return MyModel(params, hparams)

Loading artifacts from multiple files should look something like this:

m = shelf.get(["s3://my-bucket/my-file1.txt", "s3://my-bucket/my-file2.txt"], MyModel)

BUT, in case the files are saved in a directory prefix, shelf could allow the following shortcut:

# in S3:
# my-bucket/my-file1.txt
#          /my-file2.txt

m = shelf.get("s3://my-bucket/", MyModel)  # assumes that there are no other files in "my-bucket".

The latter case is not trivial at all, though. What happens if the files are not passed to the type deserializer in the correct order? I'm not sure that they are currently sorted in any order, actually. Might require a custom type order field to set on the IO struct.