yukinarit / pyserde Goto Github PK

View Code? Open in Web Editor NEW

698.0 8.0 40.0 8.5 MB

Yet another serialization library on top of dataclasses, inspired by serde-rs.

Home Page: https://yukinarit.github.io/pyserde/guide/en

License: MIT License

Python 100.00%

dataclasses serialization python msgpack json yaml toml typing serde

pyserde's Introduction

pyserde's People

Contributors

Stargazers

Watchers

Forkers

jfuechsl pranavvp10 adsharma katomono trendingtechnology mtnakayama alexmisk rnestler mauvealerts tardyp ptrbld kngwyu chagui kmsquire gitpushdashf kigawas doeringchristian soaxelbrooke kobzol cyberflamego fredrik-reinholdsen robertodr nicoddemus kykosic gpetrovic-meltin timsavage m472 acolley-gel mdbernard maallaire lkubb jefuller sh1nku rtbs-dev rucas jwsong barsa-net aman-clement k-dash

pyserde's Issues

Add tests for base container type e.g. Dict, List

@dataclass
class Foo:
    items: Dict
    list: List

Migrate to pdoc + mdBook

The documentation generated by pdoc (not pdoc3) looks good.
https://pdoc.dev/docs/pdoc.html

Support custom serializer and deserializer

For example, we want to serialize/deserialize datetime.time which is not supported by pyserde as of v0.3.1.

from datetime import time, datetime

def serialize(data):
    if isinstance(data, time):
        return data.strftime("%H:%M:%S")
    else:
        raise SerdeError(f"Unsupported type: {type(data)}")


def deserialize(typ, data):
    if issubclass(typ, time):
        return datetime.strptime(data, "%H:%M:%S").time()
    else:
        raise SerdeError(f"Unsupported type: {type(data)}")


@deserialize(custom=deserialize)
@serialize(custom=serialize)
@dataclass
class Foo:
    t: time

pyserde doesn't convert case for optional type when "rename_all" attributes specified

I expect to get
From Json: Foo(name='Pikachu', no=25) from the below code, but it actually produces
From Json: Foo(name='Pikachu', no=None).

@deserialize(rename_all='pascalcase')
@serialize(rename_all='pascalcase')
@dataclass()
class Foo:
    name: str
    no: Optional[int] = None


s = '{"Name": "Pikachu", "No": 25}'
print(f"From Json: {from_json(Foo, s)}")

The generated code:

$ python -m serde.inspect examples/rename_all.py Foo

...
def from_dict(data, reuse_instances = True):
  if reuse_instances is Ellipsis:
    reuse_instances = True
  Foo = serde_scope.types['Foo']
  if data is None:
    return None
  return cls(
  data["Name"],
  (data["no"]) if data.get("no") is not None else serde_scope.defaults["no"],
  )

Apparently pyserde doesn't convert the "no" to pascalcase "No" for this case.

Raise unsupported error when the class is loaded

Hi, @ydylla
I have a question. I was working to support typing.Any but found the unsupported error is raised at runtime. Any idea? 🤔

@deserialize
@serialize
@dataclass
class Foo:
    a: Any
    b: Any
    c: Any

Generated code

def to_dict(obj, reuse_instances=True, convert_sets=False):
    if reuse_instances is Ellipsis:
        reuse_instances = True
    if convert_sets is Ellipsis:
        convert_sets = False

    if not is_dataclass(obj):
        return copy.deepcopy(obj)

    res = {}
    res["a"] = raise_unsupported_type(obj.a)
    res["b"] = raise_unsupported_type(obj.b)
    res["c"] = raise_unsupported_type(obj.c)
    return res

PytestConfigWarning: Unknown config option: codestyle_ignore

When I run make test, I get the following warning.

=============================== warnings summary ===============================
../../../.local/share/virtualenvs/pyserde-bgcHHru8/lib/python3.6/site-packages/_pytest/config/__init__.py:1230
  /home/runner/.local/share/virtualenvs/pyserde-bgcHHru8/lib/python3.6/site-packages/_pytest/config/__init__.py:1230: PytestConfigWarning: Unknown config option: codestyle_ignore
  
    self._warn_or_fail_if_strict("Unknown config option: {}\n".format(key))

Feature request: Custom (De)serialization functions for dates

Some objects have multiple ways they can be serialized. When reading JSON files generated by other programs, dates are not always compatible with datetime.fromisoformat. Users should be able to specify how to serialize and deserialize this data on a per-field basis.

This could either be implemented specifically for dates or more generically for other data. For dates, user could specify a string to be used in strftime/strptime.

@deserialize
@serialize
@dataclass
class Foo:
    some_date: datetime.datetime = field(metadata={'serde_date': '%d/%m/%y'})

Or in a more generic case where users can supply custom serialization and deserialization functions.

DATE_FMT = '%d/%m/%y'

@deserialize
@serialize
@dataclass
class Foo:
    some_date: datetime.datetime = field(metadata={
        'serde_serialize': lambda x: x.strftime(DATE_FMT),
        'serde_deserialize': lambda x: datetime.strptime(x, DATE_FMT)
    })

Add example of rename_all class attribute

Create examples/rename_all.py and write an example code of using rename_all class attribute.

Also see tests/test_basics.py.

Make orjson as the default json serializer

orjson is a rust based json serializer and getting more popularity. I would like to change the default json serializer to orjson

If orjson discovered, use `orjson
if not discovered, use json

Here is the benchmark result. Smaller is better 👍

`json`

--- se-small ---
pyserde                                         0.088839, 0.087945, 0.089258, 0.089191, 0.088485
--- se-medium ---
pyserde                                         0.075264, 0.079170, 0.086694, 0.075400, 0.074962
--- de-small ---
pyserde                                         0.098418, 0.097855, 0.096926, 0.096757, 0.097403
--- de-medium ---
pyserde                                         0.094796, 0.098535, 0.094846, 0.094659, 0.094789

`orjson`

--- se-small ---
pyserde                                         0.034388, 0.034202, 0.034044, 0.034112, 0.034198
--- se-medium ---
pyserde                                         0.039117, 0.038936, 0.038580, 0.038555, 0.038569
--- de-small ---
pyserde                                         0.055549, 0.055119, 0.054846, 0.055648, 0.054613
--- de-medium ---
pyserde                                         0.094554, 0.094573, 0.094484, 0.094366, 0.094432

Add python 3.10.0-beta-1 beta to CI pipeline

python 3.10.0-beta-1 is available 🚀
https://github.com/actions/python-versions/releases

class with rename_all attribute gets "Cannot instantiate type Type[<nothing>]" mypy error

@serialize(rename_all='pascalcase')
@dataclass
class Foo:
    v: int

$ mypy f.py
error: Cannot instantiate type "Type[<nothing>]"

macOS
python3.9
pyserde 0.3.1

Compatibility with attrs

This library looks really interesting to me. But I need to support "attrs" in my project.

Is "pyserde" compatible with "attrs" and if not is this on the road map and how hard would it be to support it?

Support XML serialization/deserialization

It would be nice to support XML as well

Optional dependencies?

I'm not planning on using msgpack/toml/yaml in some small project, but they are always installed because pyserde requires them. As far as I understand the code, their use is isolated in separate submodules, so these dependencies can be safely uninstalled while related submodules are not used. So I suggest making these dependencies optional.

It might look something like this:

requires = [...]
msgpack_requires = ['msgpack']
toml_requires = ['toml']
yaml_requires = ['pyyaml']
# ...
setup(
    # ...
    extras_require={
        'msgpack': msgpack_requires,
        'toml': toml_requires,
        'yaml': yaml_requires,
        'full': msgpack_requires + toml_requires + yaml_requires,
        'test': tests_require,
    },
    # ...
)

Then all dependencies can be installed with something like pip install pyserde[full]

Migrate from pipenv to poetry

Poetry seems be faster and easier to use.

Clearer errors when using unsupported types

Minimal example:

from dataclasses import dataclass
from typing import Set

from serde import deserialize, serialize


@deserialize
@serialize
@dataclass
class Foo:
    a: Set

Gives error:

Traceback (most recent call last):
  File "/home/nik/PycharmProjects/pythonProject/main.py", line 10, in <module>
    class Foo:
  File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 104, in deserialize
    return wrap(_cls)
  File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 85, in wrap
    getattr(cls, '__serde_scope__')[typ.__name__] = typ
  File "/usr/lib/python3.8/typing.py", line 760, in __getattr__
    raise AttributeError(attr)
AttributeError: __name__

Suggestion in de.py:

for typ in iter_types(cls):
    try:
        if is_dataclass(typ) or is_enum(typ) or not is_primitive(typ):
            getattr(cls, '__serde_scope__')[typ.__name__] = typ
    except AttributeError as e:
        raise AttributeError(f"the type {typ} in used on a variable in"
                             f" class {cls.__name__} is not supported. "
                             f"See https://github.com/yukinarit/pyserde#supported-types")

would raise

Traceback (most recent call last):
  File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 86, in wrap
    getattr(cls, '__serde_scope__')[typ.__name__] = typ
  File "/usr/lib/python3.8/typing.py", line 760, in __getattr__
    raise AttributeError(attr)
AttributeError: __name__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nik/PycharmProjects/pythonProject/main.py", line 10, in <module>
    class Foo:
  File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 109, in deserialize
    return wrap(_cls)
  File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 88, in wrap
    raise AttributeError(f"the type {typ} in used on a variable in"
AttributeError: the type typing.Set in used on a variable in class Foo is not supported. See https://github.com/yukinarit/pyserde#supported-types

Add example for serializing Numpy data

After playing for some time, it would be nice to have an example for serialization with a Numpy type. I have given a starting example below. The essence is:

Specify type annotation with numpy.typing
Use __post_init__ to convert a List[float] to the annotated type
When (de)serializing, convert to a serializable object type

examply.py

from dataclasses import dataclass, field
from serde import serialize, deserialize
from serde.json import from_json, to_json
import numpy as np
import numpy.typing as npt


@serialize
@deserialize
@dataclass
class Foo:
    a: npt.NDArray[np.float64] = field(
        metadata={
            'serde_serializer': lambda x: x.tolist(),
            'serde_deserializer': lambda x: np.array(x, dtype=np.float64),
        }
    )

    def __post_init__(self):
        self.a = np.array(self.a, dtype=np.float64)

# Create using a list
f = Foo(a=[1., 0.])

# Check type of "f.a"
print(type(f.a))

# Serialize
print(to_json(f))

# Deserialize
s = '{"a": [1.0, 2.0]}'
print(from_json(Foo, s))

python example.py

<class 'numpy.ndarray'>
{"a": [1.0, 0.0]}
Foo(a=array([1., 2.]))

Pit falls
Ideally, numpy.ndarray could serialize itself. If one removes the serde_* metadata, the following error is found:

serde.compat.SerdeError: Unsupported type: ndarray

It is very nice to include the numpy type in the dataclass instead of List[float] because then all the methods can be called directly. (e.g. f.a.max()). Maybe there is a better solution to avoid all the extra field methods?

Any feedback would be appreciated and whether this is a worthwhile contribution.

Reject unformatted code by CI

Host multiple versions of documentation

https://yukinarit.github.io/pyserde: latest release
https://yukinarit.github.io/pyserde/v0.3.1: v0.3.1

Unclear how to define nesting properly

I want to have nested dataclasses for managing configuration parameters for our dev, test, prod environments. This code works:

from dataclasses import dataclass
from serde import serialize, deserialize
from serde.toml import from_toml, to_toml

@deserialize
@serialize
@dataclass(frozen=True)
class GcpProject:
    project_id: str
    bucket: str
    dataset: str

@deserialize
@serialize
@dataclass(frozen=True)
class Gcp:
    dev: GcpProject
    test: GcpProject
    prod: GcpProject

doc_nested = """
[gcp]
    [prod]
    project_id = "dataverbinders"
    bucket = "dataverbinders"
    dataset = "some_dataset"

    [test]
    project_id = "dataverbinders-test"
    bucket = "dataverbinders-test"
    dataset = "some_dataset_test"

    [dev]
    project_id = "dataverbinders-dev"
    bucket = "dataverbinders-dev"
    dataset = "some_dataset_dev"
"""

gcp = from_toml(Gcp, doc_nested)

Look at the toml spec, however, I thought the way to do nesting is as follows:

...
doc_nested = """
[gcp]
    [gcp.prod]
    project_id = "dataverbinders"
    bucket = "dataverbinders"
    dataset = "some_dataset"

    [gcp.test]
    project_id = "dataverbinders-test"
    bucket = "dataverbinders-test"
    dataset = "some_dataset_test"

    [gcp.dev]
    project_id = "dataverbinders-dev"
    bucket = "dataverbinders-dev"
    dataset = "some_dataset_dev"
"""

That returns a KeyError. Could someone clarify why this is?

Canonical serialization

This was discussed here back in 2016 msgpack/msgpack#215

Does pyserde have a repeatable serialization that could be fed to hashlib?

import hashlib

from datetime import datetime
from typing import Optional
from dataclasses import dataclass
from serde import serialize
from serde.msgpack import to_msgpack

@serialize
@dataclass
class Entry:
    feed_url: str
    id: str
    updated: Optional[datetime] = None

e = []
for i in range(3):
    entry = Entry('feed one', 'id', datetime(i+1, i+1, i+1))
    e.append(entry)

for i in range(3):
    for j in range(3):
        print(hashlib.sha256(to_msgpack(e[j])).hexdigest())

seems repeatable. But I haven't tested it across platforms.

Also see:

https://death.andgravity.com/stable-hashing where json was used
https://github.com/yuan-xy/canoser-python where dataclasses could be used, but are not.

Support more standard collections

Incorrect deserialization of dataclasses with similar fields

Here is an example where I have two very similar classes, Alpha and Bravo. Alpha has an additional optional field. I serialize Bravo, and when I deserialize I get back Alpha.

from dataclasses import dataclass
from typing import Union, Optional

from serde import deserialize, serialize
from serde.json import from_json, to_json


@deserialize
@serialize
@dataclass
class Alpha:
    a: str
    b: Optional[str]

@deserialize
@serialize
@dataclass
class Bravo:
    a: str

@deserialize
@serialize
@dataclass
class Foo:
    d: Union[Alpha, Bravo]


def main():
    f = Foo(Bravo(a=1))

    # convert into json
    Bravo_json = to_json(f)
    print(f"Into Json: {Bravo_json}")

    # convert back
    print(f"From Json: {from_json(Foo, Bravo_json)}")


if __name__ == '__main__':
    main()

Output:

Into Json: {"d": {"a": 1}}
From Json: Foo(d=Alpha(a=1, b=None))

Deserialization of extended types inside a container type fails

The following fails to deserialize. It seems that in the "more types" deserializer (for Path) the Optional[Path] field is passed. The deserializer in more_types.py then complains that Optional is an unsupported type. Same thing happens with other container types (e.g. List).

@deserialize
@dataclass
class Foo:
    bar: Optional[Path]

from_json(Foo, '{"bar": "/asdf"}')

Unable to deserialize nested types of union and tuple

Python 3.9 pyserde[toml]==0.3.1

from dataclasses import dataclass
from typing import Union
from serde import toml, deserialize


@deserialize
@dataclass
class TestClass:
    a: Union[None, Union[int, float], tuple[Union[int, float], Union[int, float]]]


if __name__ == '__main__':
    test_class = toml.from_toml(TestClass, 'a = [4, 1]')
    print(test_class)

Running the above code will throw an error:

serde.core.SerdeError: Can not deserialize [4, 1] of type List into Union[NoneType, int, float, Tuple[Union[int, float], Union[int, float]]]. Reason: 'union_de_int_float'

Installed python might not be used in CI

As you can see in the console, it's running in python 3.8 although it's during the Python 3.7 stage.

It appears to me

For Ubuntu:
For macOS: python3.8 is used during 3.6 / 3.7 / pypy3 stages
For Windows:
- python 3.9 is used during 3.6 / 3.7 / 3.8 stages
- python 3.7 is used during pypy stage

https://github.com/yukinarit/pyserde/runs/1404632126?check_suite_focus=true

@alexmisk Any idea? 🤔

msgpack and nested serialization

Trying to figure out how to resolve this problem:

>>> from serde.msgpack import from_msgpack, to_msgpack
>>> from simpleRaft.messages.append_entries import AppendEntriesMessage, LogEntry
>>> a = LogEntry(term=0)

# Can serialize LogEntry below

>>> to_msgpack(a)
b'\x85\xa4term\x00\xa5index\x00\xa7command\x00\xa3key\xc0\xa5value\xc0'

# Just not when it's nested in another serializable object

>>> b = AppendEntriesMessage(sender=1, receiver=2, term=3, entries=[a])
>>> to_msgpack(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 34, in to_msgpack
    return se.serialize(obj, **opts)
  File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 23, in serialize
    return msgpack.packb(maybe_ext, use_bin_type=use_bin_type, **opts)
  File "/home/arun/.local/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'LogEntry' object

Support for NewTypes

The attached example fails to serialize:

from dataclasses import dataclass
from serde import deserialize, serialize
from typing import NewType
from serde.msgpack import from_msgpack, to_msgpack

UserId = NewType("UserId", int)

@deserialize
@serialize
@dataclass
class Test:
    id: UserId


t = Test(id=UserId(10))
print(to_msgpack(t))

Improve the readability of generated code

@ydylla
Running the following command

python -m serde.inspect examples/union.py Foo

I got the output like

...

def from_dict(data, reuse_instances=True):
    if reuse_instances is Ellipsis:
        reuse_instances = True

    Foo = serde_scope.types["Foo"]

    if data is None:
        return None

    return cls(
        serde_scope.funcs["union_de_int_str"](data["v"], reuse_instances),
        serde_scope.funcs["union_de_List_int__Dict_str_int_"](data["c"], reuse_instances),
    )

----------------------------------
serde_scope: SerdeScope(cls=<class 'union.Foo'>, funcs={'union_se_int_str': <function union_se_int_str at 0x10f6ae160>, 'union_se_List_int__Dict_str_int_': <function union_se_List_int__Dict_str_int_ at 0x10fb46b80>, 'to_iter': <function to_iter at 0x10fb46af0>, 'to_dict': <function to_dict at 0x10fb46c10>, 'union_de_int_str': <function union_de_int_str at 0x10fb46dc0>, 'union_de_List_int__Dict_str_int_': <function union_de_List_int__Dict_str_int_ at 0x10fb46940>, 'from_iter': <function from_iter at 0x10fb46e50>, 'from_dict': <function from_dict at 0x10fb67040>}, defaults={}, types={'Foo': <class 'union.Foo'>}, code={'union_se_int_str': 'def union_se_int_str(obj, reuse_instances):\n\n    Foo = serde_scope.types["Foo"]\n\n    union_args = serde_scope.union_se_args["union_se_int_str"]\n\n    if is_instance(obj, union_args[0]):\n        return obj\n\n    if is_instance(obj, union_args[1]):\n        return obj\n\n    raise SerdeError(\n        "Can not serialize "\n        + repr(obj)\n        + " of type "\n        + typename(type(obj))\n        + " for Union[int, str]"\n    )\n', 'union_se_List_int__Dict_str_int_': 'def union_se_List_int__Dict_str_int_(obj, reuse_instances):\n\n    Foo = serde_scope.types["Foo"]\n\n    union_args = serde_scope.union_se_args["union_se_List_int__Dict_str_int_"]\n\n    if is_instance(obj, union_args[0]):\n        return [v for v in obj]\n\n    if is_instance(obj, union_args[1]):\n        return {k: v for k, v in obj.items()}\n\n    raise SerdeError(\n        "Can not serialize "\n        + repr(obj)\n        + " of type "\n        + typename(type(obj))\n        + " for Union[List[int], Dict[str, int]]"\n    )\n', 'to_iter': 'def to_iter(obj, reuse_instances=True, convert_sets=False):\n    if reuse_instances is Ellipsis:\n        reuse_instances = True\n    if convert_sets is Ellipsis:\n        convert_sets = False\n\n    if not is_dataclass(obj):\n        return copy.deepcopy(obj)\n\n    Foo = serde_scope.types["Foo"]\n\n    res = []\n\n    res.append(serde_scope.funcs["union_se_int_str"](obj.v, reuse_instances))\n\n    res.append(serde_scope.funcs["union_se_List_int__Dict_str_int_"](obj.c, reuse_instances))\n    return tuple(res)\n', 'to_dict': 'def to_dict(obj, reuse_instances=True, convert_sets=False):\n    if reuse_instances is Ellipsis:\n        reuse_instances = True\n    if convert_sets is Ellipsis:\n        convert_sets = False\n\n    if not is_dataclass(obj):\n        return copy.deepcopy(obj)\n\n    Foo = serde_scope.types["Foo"]\n    res = {}\n    res["v"] = serde_scope.funcs["union_se_int_str"](obj.v, reuse_instances)\n\n    res["c"] = serde_scope.funcs["union_se_List_int__Dict_str_int_"](obj.c, reuse_instances)\n\n    return res\n', 'union_de_int_str': 'def union_de_int_str(data, reuse_instances):\n\n    Foo = serde_scope.types["Foo"]\n\n    # create fake dict so we can reuse the normal render function\n    fake_dict = {"fake_key": data}\n\n    error = "Exhausted all types"\n\n    if isinstance(data, int):\n        return fake_dict["fake_key"]\n    else:\n        error = "input is not of type int"\n\n    if isinstance(data, str):\n        return fake_dict["fake_key"]\n    else:\n        error = "input is not of type str"\n\n    raise SerdeError(\n        "Can not deserialize "\n        + repr(data)\n        + " of type "\n        + typename(type(data))\n        + " into Union[int, str]. Reason: "\n        + error\n    )\n', 'union_de_List_int__Dict_str_int_': 'def union_de_List_int__Dict_str_int_(data, reuse_instances):\n\n    Foo = serde_scope.types["Foo"]\n\n    # create fake dict so we can reuse the normal render function\n    fake_dict = {"fake_key": data}\n\n    error = "Exhausted all types"\n\n    try:\n        return [v for v in fake_dict["fake_key"]]\n    except Exception as e:\n        error = str(e)\n\n    try:\n        return {k: v for k, v in fake_dict["fake_key"].items()}\n    except Exception as e:\n        error = str(e)\n\n    raise SerdeError(\n        "Can not deserialize "\n        + repr(data)\n        + " of type "\n        + typename(type(data))\n        + " into Union[List[int], Dict[str, int]]. Reason: "\n        + error\n    )\n', 'from_iter': 'def from_iter(data, reuse_instances=True):\n    if reuse_instances is Ellipsis:\n        reuse_instances = True\n\n    Foo = serde_scope.types["Foo"]\n    if data is None:\n        return None\n\n    return cls(\n        serde_scope.funcs["union_de_int_str"](data[0], reuse_instances),\n        serde_scope.funcs["union_de_List_int__Dict_str_int_"](data[1], reuse_instances),\n    )\n', 'from_dict': 'def from_dict(data, reuse_instances=True):\n    if reuse_instances is Ellipsis:\n        reuse_instances = True\n\n    Foo = serde_scope.types["Foo"]\n\n    if data is None:\n        return None\n\n    return cls(\n        serde_scope.funcs["union_de_int_str"](data["v"], reuse_instances),\n        serde_scope.funcs["union_de_List_int__Dict_str_int_"](data["c"], reuse_instances),\n    )\n'}, union_se_args={'union_se_int_str': (<class 'int'>, <class 'str'>), 'union_se_List_int__Dict_str_int_': (typing.List[int], typing.Dict[str, int])}, reuse_instances_default=True, convert_sets_default=False)

It's nice if serde.inspect could print more readable code. 🙂

An example of deserialising an Enum

Does pyserde support serialising/deserialising an Enum, other than integers?
I could not find an example.

dacite is mentioned in readme but not shown in graphs

just browsing I came across this project - I noticed dacite is mentioned but not shown

'DeField' has no attribute 'mangle' error when using skip field on optional[Dict]

I have a basic dataclass, where upon serializing i want to omit, empty/optional tags

eg:

### /workspaces/projectX/src/python/stuff/special/__init__.py
@deserialize
@serialize
@dataclass(eq=True)
class SpecialObject:
    id: str
    name: str
    description: Optional[str]
    parent: Optional[str]
    comment: Optional[str]
    attributes: Optional[Dict[str, str]] = field(default_factory=dict, metadata={'serde_skip_if': lambda v: v is None or len(v) == 0})

but with this field setting it errors...

  File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 160, in from_dataclass
    skip_if.mangled = cls.mangle(f, 'skip_if')
AttributeError: type object 'DeField' has no attribute 'mangle'

bigger stack trace

  File "/workspaces/projectX/src/python/stuff/special/__init__.py", line 17, in <module>
    class SpecialObject:
  File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 84, in deserialize
    return wrap(_cls)
  File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 77, in wrap
    cls = de_func(cls, FROM_ITER, render_from_iter(cls, custom), g)
  File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 431, in render_from_iter
    return env.get_template('iter').render(func=FROM_ITER, cls=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 6, in top-level template code
  File "/usr/local/lib/python3.8/site-packages/jinja2/runtime.py", line 545, in __next__
    rv = next(self._iterator)
  File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 173, in <genexpr>
    return iter(FieldCls.from_dataclass(f) for f in dataclass_fields(cls))
  File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 160, in from_dataclass
    skip_if.mangled = cls.mangle(f, 'skip_if')
AttributeError: type object 'DeField' has no attribute 'mangle'

Can't get code completion on pyserde class

Using vim 8 + vim-lsp + python-language-server, I can get pretty good code completion on this class.

@dataclass
class Foo
    v: int

however, if I put pyserde's decorator, the code completion will disappear.

@deserialize
@serialize
@dataclass
class Foo
    v: int

dataclasses on PyPI is installed for Python > 3.6

I tried to add pyserde to a project with poetry. the dataclasses is included in the list of dependencies.

$ poetry add pyserde
Creating virtualenv nomad-api-6YYlnFLU-py3.9 in /Users/yukinari/Library/Caches/pypoetry/virtualenvs
Using version ^0.2.2 for pyserde

Updating dependencies
Resolving dependencies... (6.7s)

Writing lock file

Package operations: 8 installs, 0 updates, 0 removals

  • Installing markupsafe (1.1.1)
  • Installing mypy-extensions (0.4.3)
  • Installing typing-extensions (3.7.4.3)
  • Installing dataclasses (0.6)
  • Installing jinja2 (2.11.3)
  • Installing stringcase (1.2.0)
  • Installing typing-inspect (0.6.0)
  • Installing pyserde (0.2.2)

Maybe install_requires has to be

requires = [
    ...
    "dataclasses;python_version=='3.6'",
]

Investigate why coverage is not updated in coveralls

coverage report is not updated since Nov. 29, 2019? 🤔
https://coveralls.io/github/yukinarit/pyserde?branch=master

Switch from flake8 to pycodestyle?

I don't know how to fix E741 ambiguous variable name 'I'...

$ pipenv run flake8
./tests/data.py:220:5: E741 ambiguous variable name 'I'
./build/lib/serde/inspect.py:19:1: F401 '.core.logger' imported but unused
make: *** [pep8] Error 1

Support for non-base types?

I use decimal.Decimal frequently, and I want to serialize to yaml. Is there an extensibility feature that will enable this to serialize? Currently I get messages like:

yaml.representer.RepresenterError: ('cannot represent an object', Decimal('75000'))

Allow Enum compatible literal value for a Enum field

Given this Enum and dataclass.

class E(enum.Enum):
    V = 1

@serialize
@dataclass
class Foo:
    v: E

This works

f = Foo(E.V)
asdict(f)

But if a literal 1 is passed, it raises an error

f = Foo(1)
asdict(f)

I want pyserde to allow enum compatible literal value (in this case 1).

Pyserde does not use default_factory during deserialization

I want to define a dict field that is optional but non-nullable, and this can be done by using default_factory. But I found that pyserde cannot deserialize data if a value for this field is not present.

from dataclasses import dataclass, field
from typing import Dict
from serde import deserialize, from_dict

@deserialize
@dataclass
class Foo:
    foo: str
    items: Dict[str, int] = field(default_factory=dict)

# check that default_factory really works
assert Foo(foo="bar").items == {}

# but it fails
foo = from_dict(Foo, {"foo": "bar"})

Traceback (most recent call last):
  File "serde_test.py", line 15, in <module>
    foo = from_dict(Foo, {"foo": "bar"})
  File "/serde/de.py", line 209, in from_dict
    return cls.__serde_from_dict__(o)
  File "<string>", line 9, in __serde_from_dict__
KeyError: 'items'

It seems the renderer completely ignores the existence of default_factory. My experiments with pyserde code have shown that this can in principle be fixed by adding serde.core.Field.default_factory and modifying the renderer (warning, ugly pseudocode):

@dataclass
class Renderer:
    # ...
    def render(self, arg: DeField) -> str:
        if is_dataclass(arg.type):
            result = self.dataclass(arg)
        # ...
        if arg.default_factory is None or isinstance(arg.default_factory, dataclasses._MISSING_TYPE):
            return result
        return f'{result} if "{arg.name}" in {arg.datavar} else cls.__dataclass_fields__["{arg.name}"].default_factory()'

But I'm not smart enough to test all cases and make a proper pull request, so I just leave this issue here.

(Not sure if this is a bug report or feature request...)

dict type in Union might be deserialized incorrectly

@ydylla
I think you are already aware of this issue but let me create this issue.

@deserialize
@serialize
@dataclass
class Foo:
    c: Union[List[int], Dict[str, int]]

s = '{"c": {"bar": 1, "baz": 2}}'
print(f"From Json: {from_json(Foo, s)}")

I expect this will print

From Json: Foo(c={'bar': 1, 'baz': 2})

but it is actually

From Json: Foo(c=['bar', 'baz'])

This is the generated deserialize code:

def union_de_List_int__Dict_str_int_(data, reuse_instances):
  Foo = serde_scope.types['Foo']

  # create fake dict so we can reuse the normal render function
  fake_dict = {"fake_key":data}

  error = "Exhausted all types"
  try:
    return [v for v in fake_dict["fake_key"]]
  except Exception as e:
    error = str(e)

  try:
    return {k: v for k, v in fake_dict["fake_key"].items()}
  except Exception as e:
    error = str(e)

  raise SerdeError("Can not deserialize " + repr(data) + " of type " + typename(type(data)) + " into Union[List[int], Dict[str, int]]. Reason: " + error)

Document how to check the generated code

Support PEP585 type hint notation

https://www.python.org/dev/peps/pep-0585/

@deserialize
@serialize
@dataclass
class C:
    d: dict[int, str]
    l: list[bool]

Add example of a field with default_factory property

Add in examples/default.py

Write more tests around Union

Support PEP563 lazy type evaluation

Adding from __future__ import annotations to examples/simple.py raises an exception.

from __future__ import annotations
from dataclasses import dataclass

from serde import deserialize, serialize
from serde.json import from_json, to_json


@deserialize
@serialize
@dataclass
class Foo:
    i: int
    s: str
    f: float
    b: bool


def main():
    f = Foo(i=10, s='foo', f=100.0, b=True)
    print(f"Into Json: {to_json(f)}")

    s = '{"i": 10, "s": "foo", "f": 100.0, "b": true}'
    print(f"From Json: {from_json(Foo, s)}")


if __name__ == '__main__':
    main()

Into Json: {"i": null, "s": null, "f": null, "b": null}
Traceback (most recent call last):
  File "/Users/yukinari/repos/pyserde/examples/simple.py", line 27, in <module>
    main()
  File "/Users/yukinari/repos/pyserde/examples/simple.py", line 23, in main
    print(f"From Json: {from_json(Foo, s)}")
  File "/Users/yukinari/repos/pyserde/serde/json.py", line 29, in from_json
    return from_dict(c, de.deserialize(s, **opts), reuse_instances=False)
  File "/Users/yukinari/repos/pyserde/serde/de.py", line 274, in from_dict
    return from_obj(cls, o, named=True, reuse_instances=reuse_instances)
  File "/Users/yukinari/repos/pyserde/serde/de.py", line 211, in from_obj
    return serde_scope.funcs[FROM_DICT](o, reuse_instances=reuse_instances)
  File "<string>", line 16, in from_dict
  File "/Users/yukinari/repos/pyserde/serde/core.py", line 82, in raise_unsupported_type
    raise SerdeError(f"Unsupported type: {typename(type(obj))}")
serde.core.SerdeError: Unsupported type: int

Support Union

To read this toml with pyserde,

[packages]
pyserde = { path = "../", editable = true }
toml = { version = "*" }
requests = "*"

I define the pyserde classes like this.
It's nice packages can be Union which is str or Package object.

@deserialize
@dataclass
class Package:
    path: Optional[str] = None
    version: Optional[str] = None
    editable: bool = False

@deserialize
@dataclass
class Pipfile:
    source: List[Source]
    requires: Requires
    packages: Dict[str, Union[str, Package]]

Such a thing could be modeled as:

@dataclass
class LargeObject:
   _field1: type1
   _field2: type2
   ...
   _fieldN: typeN
   
   @lazy_property
   def field1():
      # Code to deserialize
      
   ...

I'm not sure if pyserde can provide an additional @flatbuffer decorator at some point with logic provided by the upstream projects.

No change requested at this time. Putting this on your radar for future discussion.

Document how @serialize and @deserialize works

How does dataclasses work
What is exec all about?
What does @serialize and @deserialize do?

yukinarit / pyserde Goto Github PK

pyserde's Introduction

pyserde's People

Contributors

Stargazers

Watchers

Forkers

pyserde's Issues

json

orjson

Recommend Projects

Recommend Topics

Recommend Org

`json`

`orjson`