Use third party class as property in pydantic schema about fastapi HOT 21 CLOSED

I'm using the solution proposed by @tiangolo up above, just I preferred doing this:

class ObjectIdStr(str):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        try:
            ObjectId(str(v))
        except InvalidId:
            raise ValuerError("Not a valid ObjectId")
        return str(v)

this way I can either pass a valid ObjectId string or an ObjectId instance.

This works pretty nicely also with mongoengine, as you'll be able to pass that ObjectIdStr directly to the db_model, and it will convert the ObjectIdStrings to actual ObjectIds in Mongo.

What I'm striving to understand now, though, is why can't I get an ObjectId back from the jsonable_encoder by setting this in Config's json_encoders property:

from bson import ObjectId
from pydantic import BaseModel
from upabove import ObjectIdStr

class SomeItem(BaseModel):
    some_id: ObjectIdStr
   
    class Config:
        json_encoders = {ObjectIdStr: lambda x: ObjectId(x)}

Why wouldn't some_id be converted to an ObjectId when calling jsonable_encoder on SomeItem instance? Is it maybe because being some_id a str it won't be passed further down to the custom json_encoders? This even if ObjectIds, are not json serializable.

from fastapi.

About pydantic/pydantic#520, it was superseded by pydantic/pydantic#562.

While reviewing it I tested with bson, and I realized that it doesn't necessarily fix the problem, but that you can fix it like this:

from bson import ObjectId
from pydantic import BaseModel


class ObjectIdStr(str):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        if not isinstance(v, ObjectId):
            raise ValueError("Not a valid ObjectId")
        return str(v)


class UserId(BaseModel):
    object_id: ObjectIdStr = None


class User(UserId):
    email: str
    salt: str
    hashed_password: str

# Just for testing
user = User(object_id = ObjectId(), email="[email protected]", salt="12345678", hashed_password="letmein")
print(user.json())
# Outputs:
# {"object_id": "5c7e424225e2971c8c548a86", "email": "[email protected]", "salt": "12345678", "hashed_password": "letmein"}

The trick is, there's no way to declare a JSON Schema for a BSON ObjectId, but you can create a custom type that inherits from a str, so it will be declarable in JSON Schema, and it can take an ObjectId as input.

Then, if you need the ObjectId itself (instead of the str version), you can create another model that has the ObjectId as you declared it before, and copy the values from the input/to the output.

from fastapi.

@stefanondisponibile I'm currently working on this PR in Pydantic: pydantic/pydantic#520

It will allow you to declare object_id: str and then FastAPI will take your ObjectId("5cdc01a6d8893f59a36d9957") and convert it to a string automatically.

from fastapi.

Solved reading the implementation of hbusul/kucukdev_api#18

import typing as T
from datetime import datetime

from bson.errors import InvalidId
from bson.objectid import ObjectId as BsonObjectId
from pydantic import (
    BaseModel as _BaseModel,
    Field,
)


class ObjectId(BsonObjectId):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        try:
            return cls(v)
        except InvalidId:
            raise ValueError(f'{v} is not a valid ObjectId')
+
+   @classmethod
+   def __modify_schema__(cls, field_schema):
+       field_schema.update(type='string')


class BaseModel(_BaseModel):
    class Config:
        json_encoders = {ObjectId: str}


class BaseMongoCreateModel(BaseModel):
    date_created: T.Optional[datetime]
    date_updated: T.Optional[datetime]
    
    class Config:
        allow_population_by_field_name=True
        orm_mode=True


class BaseMongoRetrieveModel(BaseMongoCreateModel):
    id: T.Union[str, ObjectId] = Field(...)

    class Config:
        orm_mode = True
        allow_population_by_field_name=True

from fastapi.

@Charlie-iProov not yet, but it's on the backlog.

from fastapi.

Actually, I found the answer here: pydantic/pydantic#1671

So it should be:

class ObjectId(bson.ObjectId):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, value):
        try:
            return cls(value)
        except bson.errors.InvalidId:
            raise ValueError("Not a valid ObjectId")


class BaseModel(p.BaseModel):

    class Config:
        json_encoders = {ObjectId: str}

from fastapi.

@gustavorps thanks for that code it is useful! Here is the solution I did to make it work for my arbitary type: tiangolo/sqlmodel#235 (comment)

from fastapi.

Thanks for the report, sorry for the delay.

I see/assume you are using MongoDB, right?

I hope to check and debug it soon, but it might take a bit as I have to set up a stack with mongo (and I don't have a project generator with Mongo just yet).

from fastapi.

Thanks for your response.

You are right, I'm using MongoDB.
But the problem isn't related to MongoDB, so I don't think you need it to debug the error.

You just need to install the following pip packages:

• FastAPI
• bson

And then put the first code snippet in a module called user.py and the second one into some arbitrary module (e.g. server.py). When you then start the server and open the automatic documentation in your browser, you should be greeted by the error in the Log accordion above.

Note: I had to slightly modify my code snippets to make it possible to just copy-paste them.

from fastapi.

Excellent, I'll use that to debug/develop it.

from fastapi.

Was there a conclusion to this? Trying to parse mongos _id field is proving to be quite tricky unless i just delete it before returning the response

from fastapi.

Not sure this being a bug or a feature.
I'm still digging into fastAPI, but when you're saying:

@app.post("/user", tags=["user"], response_model=UserId)

you're basically declaring that your response will be a UserId, that is:

class UserId(BaseModel):
    object_id: ObjectId = None

if I was on the other side, receiving this response, I would then have to expect this kind of json:

{
  "object_id": ObjectId("5cdc01a6d8893f59a36d9957")
}

which would be pretty strange, since I couldn't have that ObjectId there.

Moreover, having to POST to that endpoint I would have a similar problem, since User inherits from UserId:

@app.post("/user", tags=["user"], response_model=UserId)
def create_user(user: User)

class User(UserId):
    email: str
    salt: str
    hashed_password: str

again, my problem would be what to send as an object_id:

{
  "object_id": ObjectId("5cdc01a6d8893f59a36d9957"),
  "email" : "[email protected]",
  "password": "letmein",
  "salt":"12345678"
}

That's why defining custom json_encoders wouldn't help here.

from fastapi.

That's great!

from fastapi.

That will do for now, thanks for you effort!

I will also test my example with the changes to Pydantic you referenced when I get around to it.

from fastapi.

Okay, so pardon me if I don't make much sense. I face this 'ObjectId' object is not iterable whenever I run the collections.find() functions. Going through the answers here, I'm not sure where to start. I'm new to programming, please bear with me.

Every time I hit the route which is supposed to fetch me data from Mongodb, I getValueError: [TypeError("'ObjectId' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')].

Help

from fastapi.

Hi @senjenathaniel ! Are you sure your problem fits this issue? If you could give some more details, and an example of the code you're using, I think someone could give the proper advice :)

from fastapi.

from bson import ObjectId
from pydantic import BaseModel
from upabove import ObjectIdStr

class SomeItem(BaseModel):
    some_id: ObjectIdStr
   
    class Config:
        json_encoders = {ObjectIdStr: lambda x: ObjectId(x)}

Why wouldn't some_id be converted to an ObjectId when calling jsonable_encoder on SomeItem instance? Is it maybe because being some_id a str it won't be passed further down to the custom json_encoders? This even if ObjectIds, are not json serializable.

@stefanondisponibile Have you solved this issue? I am also getting TypeError: Object of type 'ObjectId' is not JSON serializable though I have defined json_encoders.

from fastapi.

+1

INFO:     127.0.0.1:42104 - "GET /docs HTTP/1.1" 200 OK
INFO:     127.0.0.1:42104 - "GET /openapi.json HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 368, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/fastapi/applications.py", line 164, in openapi
    return JSONResponse(self.openapi())
  File "/home/gustavorps/code/issue/http/server.py", line 48, in custom_openapi
    openapi_schema = get_openapi(
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/fastapi/openapi/utils.py", line 389, in get_openapi
    definitions = get_model_definitions(
  File "/home/gustavorps/.miniconda3/envs/python3/lib/python3.8/site-packages/fastapi/utils.py", line 24, in get_model_definitions
    m_schema, m_definitions, m_nested_models = model_process_schema(
  File "pydantic/schema.py", line 617, in pydantic.schema.model_process_schema
  File "pydantic/schema.py", line 658, in pydantic.schema.model_type_schema
  File "pydantic/schema.py", line 258, in pydantic.schema.field_schema
  File "pydantic/schema.py", line 563, in pydantic.schema.field_type_schema
  File "pydantic/schema.py", line 848, in pydantic.schema.field_singleton_schema
  File "pydantic/schema.py", line 748, in pydantic.schema.field_singleton_sub_fields_schema
  File "pydantic/schema.py", line 563, in pydantic.schema.field_type_schema
  File "pydantic/schema.py", line 947, in pydantic.schema.field_singleton_schema
ValueError: Value not declarable with JSON Schema, field: name='id_ObjectId' type=ObjectId required=True

import typing as T
from datetime import datetime

from bson.errors import InvalidId
from bson.objectid import ObjectId as BsonObjectId
from pydantic import (
    BaseModel as _BaseModel,
    Field,
)


class ObjectId(BsonObjectId):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        try:
            return cls(v)
        except InvalidId:
            raise ValueError(f'{v} is not a valid ObjectId')


class BaseModel(_BaseModel):
    class Config:
        json_encoders = {ObjectId: str}


class BaseMongoCreateModel(BaseModel):
    date_created: T.Optional[datetime]
    date_updated: T.Optional[datetime]
    
    class Config:
        allow_population_by_field_name=True
        orm_mode=True


class BaseMongoRetrieveModel(BaseMongoCreateModel):
    id: T.Union[str, ObjectId] = Field(...)

    class Config:
        orm_mode = True
        allow_population_by_field_name=True

$ python
Python 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:53:36) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydantic; pydantic.__version__
'1.9.0'
>>> import fastapi; fastapi.__version__
'0.73.0'

from fastapi.

I'm getting the same issues with trying to access geoalchemy2 data through shapely geometry types.

@gustavorps' comment above works, but doesn't seem very "safe". Why are we having to monkey patch the type of the field?

from fastapi.

I'm getting the same issues with trying to access geoalchemy2 data through shapely geometry types.

@gustavorps' comment above works, but doesn't seem very "safe". Why are we having to monkey patch the type of the field?

are you patching the geometry type to include more classmethods? I am interested to see how you have made it work as I am also facing this issue using sqlmodel + shapely

from geoalchemy2 import Geometry
from pydantic import validator, BaseConfig
from shapely.geometry import Point, asShape
from sqlalchemy import Column
from sqlmodel import Field, SQLModel

class LocBase(SQLModel):
    class Config:
        allow_population_by_field_name = True
        arbitrary_types_allowed = True

    name: str
    [...]
    longitude: Optional[float] = None
    latitude: Optional[float] = None
    geom: Optional[Point] = Field(
        nullable=True,
        index=True,
        sa_column=Column(Geometry("POINT", srid=4326)),
        alias="point",
    )

    @validator("geom", pre=True)
    def geom_to_shape(cls, v):
        return asShape(v)

I had to add

BaseConfig.arbitrary_types_allowed = True

as arbitrary_types_allowed=True does not seem to be doing the job.
The final error that I'm getting is ValueError: Value not declarable with JSON Schema, field: name='geom' type=Optional[Point] required=False default=None alias='point'

from fastapi.

Assuming the original need was handled, this will be automatically closed now. But feel free to add more comments or create new issues or PRs.

from fastapi.