yukinarit / pyserde Goto Github PK
View Code? Open in Web Editor NEWYet another serialization library on top of dataclasses, inspired by serde-rs.
Home Page: https://yukinarit.github.io/pyserde/guide/en
License: MIT License
Yet another serialization library on top of dataclasses, inspired by serde-rs.
Home Page: https://yukinarit.github.io/pyserde/guide/en
License: MIT License
@dataclass
class Foo:
items: Dict
list: List
The documentation generated by pdoc (not pdoc3) looks good.
https://pdoc.dev/docs/pdoc.html
For example, we want to serialize/deserialize datetime.time
which is not supported by pyserde as of v0.3.1.
from datetime import time, datetime
def serialize(data):
if isinstance(data, time):
return data.strftime("%H:%M:%S")
else:
raise SerdeError(f"Unsupported type: {type(data)}")
def deserialize(typ, data):
if issubclass(typ, time):
return datetime.strptime(data, "%H:%M:%S").time()
else:
raise SerdeError(f"Unsupported type: {type(data)}")
@deserialize(custom=deserialize)
@serialize(custom=serialize)
@dataclass
class Foo:
t: time
I expect to get
From Json: Foo(name='Pikachu', no=25)
from the below code, but it actually produces
From Json: Foo(name='Pikachu', no=None)
.
@deserialize(rename_all='pascalcase')
@serialize(rename_all='pascalcase')
@dataclass()
class Foo:
name: str
no: Optional[int] = None
s = '{"Name": "Pikachu", "No": 25}'
print(f"From Json: {from_json(Foo, s)}")
The generated code:
$ python -m serde.inspect examples/rename_all.py Foo
...
def from_dict(data, reuse_instances = True):
if reuse_instances is Ellipsis:
reuse_instances = True
Foo = serde_scope.types['Foo']
if data is None:
return None
return cls(
data["Name"],
(data["no"]) if data.get("no") is not None else serde_scope.defaults["no"],
)
Apparently pyserde doesn't convert the "no"
to pascalcase "No"
for this case.
Hi, @ydylla
I have a question. I was working to support typing.Any
but found the unsupported error is raised at runtime. Any idea? 🤔
@deserialize
@serialize
@dataclass
class Foo:
a: Any
b: Any
c: Any
Generated code
def to_dict(obj, reuse_instances=True, convert_sets=False):
if reuse_instances is Ellipsis:
reuse_instances = True
if convert_sets is Ellipsis:
convert_sets = False
if not is_dataclass(obj):
return copy.deepcopy(obj)
res = {}
res["a"] = raise_unsupported_type(obj.a)
res["b"] = raise_unsupported_type(obj.b)
res["c"] = raise_unsupported_type(obj.c)
return res
When I run make test
, I get the following warning.
=============================== warnings summary ===============================
../../../.local/share/virtualenvs/pyserde-bgcHHru8/lib/python3.6/site-packages/_pytest/config/__init__.py:1230
/home/runner/.local/share/virtualenvs/pyserde-bgcHHru8/lib/python3.6/site-packages/_pytest/config/__init__.py:1230: PytestConfigWarning: Unknown config option: codestyle_ignore
self._warn_or_fail_if_strict("Unknown config option: {}\n".format(key))
Some objects have multiple ways they can be serialized. When reading JSON files generated by other programs, dates are not always compatible with datetime.fromisoformat
. Users should be able to specify how to serialize and deserialize this data on a per-field basis.
This could either be implemented specifically for dates or more generically for other data. For dates, user could specify a string to be used in strftime/strptime.
@deserialize
@serialize
@dataclass
class Foo:
some_date: datetime.datetime = field(metadata={'serde_date': '%d/%m/%y'})
Or in a more generic case where users can supply custom serialization and deserialization functions.
DATE_FMT = '%d/%m/%y'
@deserialize
@serialize
@dataclass
class Foo:
some_date: datetime.datetime = field(metadata={
'serde_serialize': lambda x: x.strftime(DATE_FMT),
'serde_deserialize': lambda x: datetime.strptime(x, DATE_FMT)
})
Create examples/rename_all.py
and write an example code of using rename_all
class attribute.
Also see tests/test_basics.py.
orjson is a rust based json serializer and getting more popularity. I would like to change the default json serializer to orjson
orjson
discovered, use `orjsonjson
Here is the benchmark result. Smaller is better 👍
json
--- se-small ---
pyserde 0.088839, 0.087945, 0.089258, 0.089191, 0.088485
--- se-medium ---
pyserde 0.075264, 0.079170, 0.086694, 0.075400, 0.074962
--- de-small ---
pyserde 0.098418, 0.097855, 0.096926, 0.096757, 0.097403
--- de-medium ---
pyserde 0.094796, 0.098535, 0.094846, 0.094659, 0.094789
orjson
--- se-small ---
pyserde 0.034388, 0.034202, 0.034044, 0.034112, 0.034198
--- se-medium ---
pyserde 0.039117, 0.038936, 0.038580, 0.038555, 0.038569
--- de-small ---
pyserde 0.055549, 0.055119, 0.054846, 0.055648, 0.054613
--- de-medium ---
pyserde 0.094554, 0.094573, 0.094484, 0.094366, 0.094432
python 3.10.0-beta-1
is available 🚀
https://github.com/actions/python-versions/releases
@serialize(rename_all='pascalcase')
@dataclass
class Foo:
v: int
$ mypy f.py
error: Cannot instantiate type "Type[<nothing>]"
This library looks really interesting to me. But I need to support "attrs" in my project.
Is "pyserde" compatible with "attrs" and if not is this on the road map and how hard would it be to support it?
It would be nice to support XML as well
I'm not planning on using msgpack/toml/yaml in some small project, but they are always installed because pyserde requires them. As far as I understand the code, their use is isolated in separate submodules, so these dependencies can be safely uninstalled while related submodules are not used. So I suggest making these dependencies optional.
It might look something like this:
requires = [...]
msgpack_requires = ['msgpack']
toml_requires = ['toml']
yaml_requires = ['pyyaml']
# ...
setup(
# ...
extras_require={
'msgpack': msgpack_requires,
'toml': toml_requires,
'yaml': yaml_requires,
'full': msgpack_requires + toml_requires + yaml_requires,
'test': tests_require,
},
# ...
)
Then all dependencies can be installed with something like pip install pyserde[full]
Poetry seems be faster and easier to use.
Minimal example:
from dataclasses import dataclass
from typing import Set
from serde import deserialize, serialize
@deserialize
@serialize
@dataclass
class Foo:
a: Set
Gives error:
Traceback (most recent call last):
File "/home/nik/PycharmProjects/pythonProject/main.py", line 10, in <module>
class Foo:
File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 104, in deserialize
return wrap(_cls)
File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 85, in wrap
getattr(cls, '__serde_scope__')[typ.__name__] = typ
File "/usr/lib/python3.8/typing.py", line 760, in __getattr__
raise AttributeError(attr)
AttributeError: __name__
Suggestion in de.py
:
for typ in iter_types(cls):
try:
if is_dataclass(typ) or is_enum(typ) or not is_primitive(typ):
getattr(cls, '__serde_scope__')[typ.__name__] = typ
except AttributeError as e:
raise AttributeError(f"the type {typ} in used on a variable in"
f" class {cls.__name__} is not supported. "
f"See https://github.com/yukinarit/pyserde#supported-types")
would raise
Traceback (most recent call last):
File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 86, in wrap
getattr(cls, '__serde_scope__')[typ.__name__] = typ
File "/usr/lib/python3.8/typing.py", line 760, in __getattr__
raise AttributeError(attr)
AttributeError: __name__
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nik/PycharmProjects/pythonProject/main.py", line 10, in <module>
class Foo:
File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 109, in deserialize
return wrap(_cls)
File "/home/nik/.cache/pypoetry/virtualenvs/pythonproject-iTa_5C8b-py3.8/lib/python3.8/site-packages/serde/de.py", line 88, in wrap
raise AttributeError(f"the type {typ} in used on a variable in"
AttributeError: the type typing.Set in used on a variable in class Foo is not supported. See https://github.com/yukinarit/pyserde#supported-types
After playing for some time, it would be nice to have an example for serialization with a Numpy type. I have given a starting example below. The essence is:
numpy.typing
__post_init__
to convert a List[float]
to the annotated typeexamply.py
from dataclasses import dataclass, field
from serde import serialize, deserialize
from serde.json import from_json, to_json
import numpy as np
import numpy.typing as npt
@serialize
@deserialize
@dataclass
class Foo:
a: npt.NDArray[np.float64] = field(
metadata={
'serde_serializer': lambda x: x.tolist(),
'serde_deserializer': lambda x: np.array(x, dtype=np.float64),
}
)
def __post_init__(self):
self.a = np.array(self.a, dtype=np.float64)
# Create using a list
f = Foo(a=[1., 0.])
# Check type of "f.a"
print(type(f.a))
# Serialize
print(to_json(f))
# Deserialize
s = '{"a": [1.0, 2.0]}'
print(from_json(Foo, s))
python example.py
<class 'numpy.ndarray'>
{"a": [1.0, 0.0]}
Foo(a=array([1., 2.]))
Pit falls
Ideally, numpy.ndarray
could serialize itself. If one removes the serde_*
metadata, the following error is found:
serde.compat.SerdeError: Unsupported type: ndarray
It is very nice to include the numpy type in the dataclass instead of List[float]
because then all the methods can be called directly. (e.g. f.a.max()
). Maybe there is a better solution to avoid all the extra field methods?
Any feedback would be appreciated and whether this is a worthwhile contribution.
https://yukinarit.github.io/pyserde
: latest releasehttps://yukinarit.github.io/pyserde/v0.3.1
: v0.3.1I want to have nested dataclasses for managing configuration parameters for our dev, test, prod environments. This code works:
from dataclasses import dataclass
from serde import serialize, deserialize
from serde.toml import from_toml, to_toml
@deserialize
@serialize
@dataclass(frozen=True)
class GcpProject:
project_id: str
bucket: str
dataset: str
@deserialize
@serialize
@dataclass(frozen=True)
class Gcp:
dev: GcpProject
test: GcpProject
prod: GcpProject
doc_nested = """
[gcp]
[prod]
project_id = "dataverbinders"
bucket = "dataverbinders"
dataset = "some_dataset"
[test]
project_id = "dataverbinders-test"
bucket = "dataverbinders-test"
dataset = "some_dataset_test"
[dev]
project_id = "dataverbinders-dev"
bucket = "dataverbinders-dev"
dataset = "some_dataset_dev"
"""
gcp = from_toml(Gcp, doc_nested)
Look at the toml spec, however, I thought the way to do nesting is as follows:
...
doc_nested = """
[gcp]
[gcp.prod]
project_id = "dataverbinders"
bucket = "dataverbinders"
dataset = "some_dataset"
[gcp.test]
project_id = "dataverbinders-test"
bucket = "dataverbinders-test"
dataset = "some_dataset_test"
[gcp.dev]
project_id = "dataverbinders-dev"
bucket = "dataverbinders-dev"
dataset = "some_dataset_dev"
"""
That returns a KeyError
. Could someone clarify why this is?
This was discussed here back in 2016 msgpack/msgpack#215
Does pyserde have a repeatable serialization that could be fed to hashlib?
import hashlib
from datetime import datetime
from typing import Optional
from dataclasses import dataclass
from serde import serialize
from serde.msgpack import to_msgpack
@serialize
@dataclass
class Entry:
feed_url: str
id: str
updated: Optional[datetime] = None
e = []
for i in range(3):
entry = Entry('feed one', 'id', datetime(i+1, i+1, i+1))
e.append(entry)
for i in range(3):
for j in range(3):
print(hashlib.sha256(to_msgpack(e[j])).hexdigest())
seems repeatable. But I haven't tested it across platforms.
Also see:
https://death.andgravity.com/stable-hashing where json was used
https://github.com/yuan-xy/canoser-python where dataclasses could be used, but are not.
Here is an example where I have two very similar classes, Alpha and Bravo. Alpha has an additional optional field. I serialize Bravo, and when I deserialize I get back Alpha.
from dataclasses import dataclass
from typing import Union, Optional
from serde import deserialize, serialize
from serde.json import from_json, to_json
@deserialize
@serialize
@dataclass
class Alpha:
a: str
b: Optional[str]
@deserialize
@serialize
@dataclass
class Bravo:
a: str
@deserialize
@serialize
@dataclass
class Foo:
d: Union[Alpha, Bravo]
def main():
f = Foo(Bravo(a=1))
# convert into json
Bravo_json = to_json(f)
print(f"Into Json: {Bravo_json}")
# convert back
print(f"From Json: {from_json(Foo, Bravo_json)}")
if __name__ == '__main__':
main()
Output:
Into Json: {"d": {"a": 1}}
From Json: Foo(d=Alpha(a=1, b=None))
The following fails to deserialize. It seems that in the "more types" deserializer (for Path
) the Optional[Path]
field is passed. The deserializer in more_types.py
then complains that Optional
is an unsupported type. Same thing happens with other container types (e.g. List
).
@deserialize
@dataclass
class Foo:
bar: Optional[Path]
from_json(Foo, '{"bar": "/asdf"}')
Python 3.9 pyserde[toml]==0.3.1
from dataclasses import dataclass
from typing import Union
from serde import toml, deserialize
@deserialize
@dataclass
class TestClass:
a: Union[None, Union[int, float], tuple[Union[int, float], Union[int, float]]]
if __name__ == '__main__':
test_class = toml.from_toml(TestClass, 'a = [4, 1]')
print(test_class)
Running the above code will throw an error:
serde.core.SerdeError: Can not deserialize [4, 1] of type List into Union[NoneType, int, float, Tuple[Union[int, float], Union[int, float]]]. Reason: 'union_de_int_float'
As you can see in the console, it's running in python 3.8 although it's during the Python 3.7
stage.
It appears to me
Ubuntu
:macOS
: python3.8 is used during 3.6 / 3.7 / pypy3 stagesWindows
:
https://github.com/yukinarit/pyserde/runs/1404632126?check_suite_focus=true
@alexmisk Any idea? 🤔
Trying to figure out how to resolve this problem:
>>> from serde.msgpack import from_msgpack, to_msgpack
>>> from simpleRaft.messages.append_entries import AppendEntriesMessage, LogEntry
>>> a = LogEntry(term=0)
# Can serialize LogEntry below
>>> to_msgpack(a)
b'\x85\xa4term\x00\xa5index\x00\xa7command\x00\xa3key\xc0\xa5value\xc0'
# Just not when it's nested in another serializable object
>>> b = AppendEntriesMessage(sender=1, receiver=2, term=3, entries=[a])
>>> to_msgpack(b)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 34, in to_msgpack
return se.serialize(obj, **opts)
File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 23, in serialize
return msgpack.packb(maybe_ext, use_bin_type=use_bin_type, **opts)
File "/home/arun/.local/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'LogEntry' object
The attached example fails to serialize:
from dataclasses import dataclass
from serde import deserialize, serialize
from typing import NewType
from serde.msgpack import from_msgpack, to_msgpack
UserId = NewType("UserId", int)
@deserialize
@serialize
@dataclass
class Test:
id: UserId
t = Test(id=UserId(10))
print(to_msgpack(t))
@ydylla
Running the following command
python -m serde.inspect examples/union.py Foo
I got the output like
...
def from_dict(data, reuse_instances=True):
if reuse_instances is Ellipsis:
reuse_instances = True
Foo = serde_scope.types["Foo"]
if data is None:
return None
return cls(
serde_scope.funcs["union_de_int_str"](data["v"], reuse_instances),
serde_scope.funcs["union_de_List_int__Dict_str_int_"](data["c"], reuse_instances),
)
----------------------------------
serde_scope: SerdeScope(cls=<class 'union.Foo'>, funcs={'union_se_int_str': <function union_se_int_str at 0x10f6ae160>, 'union_se_List_int__Dict_str_int_': <function union_se_List_int__Dict_str_int_ at 0x10fb46b80>, 'to_iter': <function to_iter at 0x10fb46af0>, 'to_dict': <function to_dict at 0x10fb46c10>, 'union_de_int_str': <function union_de_int_str at 0x10fb46dc0>, 'union_de_List_int__Dict_str_int_': <function union_de_List_int__Dict_str_int_ at 0x10fb46940>, 'from_iter': <function from_iter at 0x10fb46e50>, 'from_dict': <function from_dict at 0x10fb67040>}, defaults={}, types={'Foo': <class 'union.Foo'>}, code={'union_se_int_str': 'def union_se_int_str(obj, reuse_instances):\n\n Foo = serde_scope.types["Foo"]\n\n union_args = serde_scope.union_se_args["union_se_int_str"]\n\n if is_instance(obj, union_args[0]):\n return obj\n\n if is_instance(obj, union_args[1]):\n return obj\n\n raise SerdeError(\n "Can not serialize "\n + repr(obj)\n + " of type "\n + typename(type(obj))\n + " for Union[int, str]"\n )\n', 'union_se_List_int__Dict_str_int_': 'def union_se_List_int__Dict_str_int_(obj, reuse_instances):\n\n Foo = serde_scope.types["Foo"]\n\n union_args = serde_scope.union_se_args["union_se_List_int__Dict_str_int_"]\n\n if is_instance(obj, union_args[0]):\n return [v for v in obj]\n\n if is_instance(obj, union_args[1]):\n return {k: v for k, v in obj.items()}\n\n raise SerdeError(\n "Can not serialize "\n + repr(obj)\n + " of type "\n + typename(type(obj))\n + " for Union[List[int], Dict[str, int]]"\n )\n', 'to_iter': 'def to_iter(obj, reuse_instances=True, convert_sets=False):\n if reuse_instances is Ellipsis:\n reuse_instances = True\n if convert_sets is Ellipsis:\n convert_sets = False\n\n if not is_dataclass(obj):\n return copy.deepcopy(obj)\n\n Foo = serde_scope.types["Foo"]\n\n res = []\n\n res.append(serde_scope.funcs["union_se_int_str"](obj.v, reuse_instances))\n\n res.append(serde_scope.funcs["union_se_List_int__Dict_str_int_"](obj.c, reuse_instances))\n return tuple(res)\n', 'to_dict': 'def to_dict(obj, reuse_instances=True, convert_sets=False):\n if reuse_instances is Ellipsis:\n reuse_instances = True\n if convert_sets is Ellipsis:\n convert_sets = False\n\n if not is_dataclass(obj):\n return copy.deepcopy(obj)\n\n Foo = serde_scope.types["Foo"]\n res = {}\n res["v"] = serde_scope.funcs["union_se_int_str"](obj.v, reuse_instances)\n\n res["c"] = serde_scope.funcs["union_se_List_int__Dict_str_int_"](obj.c, reuse_instances)\n\n return res\n', 'union_de_int_str': 'def union_de_int_str(data, reuse_instances):\n\n Foo = serde_scope.types["Foo"]\n\n # create fake dict so we can reuse the normal render function\n fake_dict = {"fake_key": data}\n\n error = "Exhausted all types"\n\n if isinstance(data, int):\n return fake_dict["fake_key"]\n else:\n error = "input is not of type int"\n\n if isinstance(data, str):\n return fake_dict["fake_key"]\n else:\n error = "input is not of type str"\n\n raise SerdeError(\n "Can not deserialize "\n + repr(data)\n + " of type "\n + typename(type(data))\n + " into Union[int, str]. Reason: "\n + error\n )\n', 'union_de_List_int__Dict_str_int_': 'def union_de_List_int__Dict_str_int_(data, reuse_instances):\n\n Foo = serde_scope.types["Foo"]\n\n # create fake dict so we can reuse the normal render function\n fake_dict = {"fake_key": data}\n\n error = "Exhausted all types"\n\n try:\n return [v for v in fake_dict["fake_key"]]\n except Exception as e:\n error = str(e)\n\n try:\n return {k: v for k, v in fake_dict["fake_key"].items()}\n except Exception as e:\n error = str(e)\n\n raise SerdeError(\n "Can not deserialize "\n + repr(data)\n + " of type "\n + typename(type(data))\n + " into Union[List[int], Dict[str, int]]. Reason: "\n + error\n )\n', 'from_iter': 'def from_iter(data, reuse_instances=True):\n if reuse_instances is Ellipsis:\n reuse_instances = True\n\n Foo = serde_scope.types["Foo"]\n if data is None:\n return None\n\n return cls(\n serde_scope.funcs["union_de_int_str"](data[0], reuse_instances),\n serde_scope.funcs["union_de_List_int__Dict_str_int_"](data[1], reuse_instances),\n )\n', 'from_dict': 'def from_dict(data, reuse_instances=True):\n if reuse_instances is Ellipsis:\n reuse_instances = True\n\n Foo = serde_scope.types["Foo"]\n\n if data is None:\n return None\n\n return cls(\n serde_scope.funcs["union_de_int_str"](data["v"], reuse_instances),\n serde_scope.funcs["union_de_List_int__Dict_str_int_"](data["c"], reuse_instances),\n )\n'}, union_se_args={'union_se_int_str': (<class 'int'>, <class 'str'>), 'union_se_List_int__Dict_str_int_': (typing.List[int], typing.Dict[str, int])}, reuse_instances_default=True, convert_sets_default=False)
It's nice if serde.inspect
could print more readable code. 🙂
Does pyserde support serialising/deserialising an Enum, other than integers?
I could not find an example.
just browsing I came across this project - I noticed dacite is mentioned but not shown
I have a basic dataclass, where upon serializing
i want to omit, empty/optional tags
eg:
### /workspaces/projectX/src/python/stuff/special/__init__.py
@deserialize
@serialize
@dataclass(eq=True)
class SpecialObject:
id: str
name: str
description: Optional[str]
parent: Optional[str]
comment: Optional[str]
attributes: Optional[Dict[str, str]] = field(default_factory=dict, metadata={'serde_skip_if': lambda v: v is None or len(v) == 0})
but with this field setting it errors...
File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 160, in from_dataclass
skip_if.mangled = cls.mangle(f, 'skip_if')
AttributeError: type object 'DeField' has no attribute 'mangle'
bigger stack trace
File "/workspaces/projectX/src/python/stuff/special/__init__.py", line 17, in <module>
class SpecialObject:
File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 84, in deserialize
return wrap(_cls)
File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 77, in wrap
cls = de_func(cls, FROM_ITER, render_from_iter(cls, custom), g)
File "/usr/local/lib/python3.8/site-packages/serde/de.py", line 431, in render_from_iter
return env.get_template('iter').render(func=FROM_ITER, cls=cls)
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 6, in top-level template code
File "/usr/local/lib/python3.8/site-packages/jinja2/runtime.py", line 545, in __next__
rv = next(self._iterator)
File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 173, in <genexpr>
return iter(FieldCls.from_dataclass(f) for f in dataclass_fields(cls))
File "/usr/local/lib/python3.8/site-packages/serde/core.py", line 160, in from_dataclass
skip_if.mangled = cls.mangle(f, 'skip_if')
AttributeError: type object 'DeField' has no attribute 'mangle'
Using vim 8
+ vim-lsp
+ python-language-server
, I can get pretty good code completion on this class.
@dataclass
class Foo
v: int
however, if I put pyserde's decorator, the code completion will disappear.
@deserialize
@serialize
@dataclass
class Foo
v: int
I tried to add pyserde to a project with poetry. the dataclasses is included in the list of dependencies.
$ poetry add pyserde
Creating virtualenv nomad-api-6YYlnFLU-py3.9 in /Users/yukinari/Library/Caches/pypoetry/virtualenvs
Using version ^0.2.2 for pyserde
Updating dependencies
Resolving dependencies... (6.7s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing markupsafe (1.1.1)
• Installing mypy-extensions (0.4.3)
• Installing typing-extensions (3.7.4.3)
• Installing dataclasses (0.6)
• Installing jinja2 (2.11.3)
• Installing stringcase (1.2.0)
• Installing typing-inspect (0.6.0)
• Installing pyserde (0.2.2)
Maybe install_requires has to be
requires = [
...
"dataclasses;python_version=='3.6'",
]
coverage report is not updated since Nov. 29, 2019? 🤔
https://coveralls.io/github/yukinarit/pyserde?branch=master
I don't know how to fix E741 ambiguous variable name 'I'
...
$ pipenv run flake8
./tests/data.py:220:5: E741 ambiguous variable name 'I'
./build/lib/serde/inspect.py:19:1: F401 '.core.logger' imported but unused
make: *** [pep8] Error 1
I use decimal.Decimal
frequently, and I want to serialize to yaml. Is there an extensibility feature that will enable this to serialize? Currently I get messages like:
yaml.representer.RepresenterError: ('cannot represent an object', Decimal('75000'))
Given this Enum and dataclass.
class E(enum.Enum):
V = 1
@serialize
@dataclass
class Foo:
v: E
This works
f = Foo(E.V)
asdict(f)
But if a literal 1
is passed, it raises an error
f = Foo(1)
asdict(f)
I want pyserde to allow enum compatible literal value (in this case 1
).
I want to define a dict field that is optional but non-nullable, and this can be done by using default_factory. But I found that pyserde cannot deserialize data if a value for this field is not present.
from dataclasses import dataclass, field
from typing import Dict
from serde import deserialize, from_dict
@deserialize
@dataclass
class Foo:
foo: str
items: Dict[str, int] = field(default_factory=dict)
# check that default_factory really works
assert Foo(foo="bar").items == {}
# but it fails
foo = from_dict(Foo, {"foo": "bar"})
Traceback (most recent call last):
File "serde_test.py", line 15, in <module>
foo = from_dict(Foo, {"foo": "bar"})
File "/serde/de.py", line 209, in from_dict
return cls.__serde_from_dict__(o)
File "<string>", line 9, in __serde_from_dict__
KeyError: 'items'
It seems the renderer completely ignores the existence of default_factory
. My experiments with pyserde code have shown that this can in principle be fixed by adding serde.core.Field.default_factory
and modifying the renderer (warning, ugly pseudocode):
@dataclass
class Renderer:
# ...
def render(self, arg: DeField) -> str:
if is_dataclass(arg.type):
result = self.dataclass(arg)
# ...
if arg.default_factory is None or isinstance(arg.default_factory, dataclasses._MISSING_TYPE):
return result
return f'{result} if "{arg.name}" in {arg.datavar} else cls.__dataclass_fields__["{arg.name}"].default_factory()'
But I'm not smart enough to test all cases and make a proper pull request, so I just leave this issue here.
(Not sure if this is a bug report or feature request...)
@ydylla
I think you are already aware of this issue but let me create this issue.
@deserialize
@serialize
@dataclass
class Foo:
c: Union[List[int], Dict[str, int]]
s = '{"c": {"bar": 1, "baz": 2}}'
print(f"From Json: {from_json(Foo, s)}")
I expect this will print
From Json: Foo(c={'bar': 1, 'baz': 2})
but it is actually
From Json: Foo(c=['bar', 'baz'])
This is the generated deserialize code:
def union_de_List_int__Dict_str_int_(data, reuse_instances):
Foo = serde_scope.types['Foo']
# create fake dict so we can reuse the normal render function
fake_dict = {"fake_key":data}
error = "Exhausted all types"
try:
return [v for v in fake_dict["fake_key"]]
except Exception as e:
error = str(e)
try:
return {k: v for k, v in fake_dict["fake_key"].items()}
except Exception as e:
error = str(e)
raise SerdeError("Can not deserialize " + repr(data) + " of type " + typename(type(data)) + " into Union[List[int], Dict[str, int]]. Reason: " + error)
https://www.python.org/dev/peps/pep-0585/
@deserialize
@serialize
@dataclass
class C:
d: dict[int, str]
l: list[bool]
Add in examples/default.py
Adding from __future__ import annotations
to examples/simple.py raises an exception.
from __future__ import annotations
from dataclasses import dataclass
from serde import deserialize, serialize
from serde.json import from_json, to_json
@deserialize
@serialize
@dataclass
class Foo:
i: int
s: str
f: float
b: bool
def main():
f = Foo(i=10, s='foo', f=100.0, b=True)
print(f"Into Json: {to_json(f)}")
s = '{"i": 10, "s": "foo", "f": 100.0, "b": true}'
print(f"From Json: {from_json(Foo, s)}")
if __name__ == '__main__':
main()
Into Json: {"i": null, "s": null, "f": null, "b": null}
Traceback (most recent call last):
File "/Users/yukinari/repos/pyserde/examples/simple.py", line 27, in <module>
main()
File "/Users/yukinari/repos/pyserde/examples/simple.py", line 23, in main
print(f"From Json: {from_json(Foo, s)}")
File "/Users/yukinari/repos/pyserde/serde/json.py", line 29, in from_json
return from_dict(c, de.deserialize(s, **opts), reuse_instances=False)
File "/Users/yukinari/repos/pyserde/serde/de.py", line 274, in from_dict
return from_obj(cls, o, named=True, reuse_instances=reuse_instances)
File "/Users/yukinari/repos/pyserde/serde/de.py", line 211, in from_obj
return serde_scope.funcs[FROM_DICT](o, reuse_instances=reuse_instances)
File "<string>", line 16, in from_dict
File "/Users/yukinari/repos/pyserde/serde/core.py", line 82, in raise_unsupported_type
raise SerdeError(f"Unsupported type: {typename(type(obj))}")
serde.core.SerdeError: Unsupported type: int
To read this toml with pyserde,
[packages]
pyserde = { path = "../", editable = true }
toml = { version = "*" }
requests = "*"
I define the pyserde classes like this.
It's nice packages
can be Union
which is str
or Package
object.
@deserialize
@dataclass
class Package:
path: Optional[str] = None
version: Optional[str] = None
editable: bool = False
@deserialize
@dataclass
class Pipfile:
source: List[Source]
requires: Requires
packages: Dict[str, Union[str, Package]]
Should be updated only when master
has changed
More context in: google/flatbuffers#6428
Unlike msgpack, flatbuffers has the additional requirement that in a large type with many fields, if the caller is interested in deserializing only a couple of fields, they don't pay the cost of deserializing all fields.
Such a thing could be modeled as:
@dataclass
class LargeObject:
_field1: type1
_field2: type2
...
_fieldN: typeN
@lazy_property
def field1():
# Code to deserialize
...
I'm not sure if pyserde can provide an additional @flatbuffer
decorator at some point with logic provided by the upstream projects.
No change requested at this time. Putting this on your radar for future discussion.
dataclasses
workexec
all about?@serialize
and @deserialize
do?A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.