Bug: OpenAPI schema invalid when using Pydantic v2 (latest) about powertools-lambda-python HOT 10 CLOSED

Hi all, apologies for the late update. Here are my findings:

• @kbakk is obviously correct that Pydantic v1 and v2 generates two different types of JSON Schemas (v1 compatible with OpenAPI v3.0.x, and v2 compatible with OpenAPI v3.1.x)
• There's no internal Pydantic flag to change that behavior. Pydantic v1 will always be OpenAPI 3.0, Pydantic v2 will always be OpenAPI 3.1.
• We actually have control over the majority of the generated OpenAPI schemas (so we could fix the summary part above). However, the problem will always be when we ask Pydantic to create a JSON schema from a Pydantic model (which will generate two different things depending on v1 / v2)

In my opinion we need to:

• Document that different versions of pydantic generate different types of OpenAPI schemas
• Follow @kbakk's advice and automatically set the correct version depending on the Pydantic version used (right now it always says 3.0.0, which is wrong)
• Add a big fat warning in docs (and a warning in code) when using BedrockAgentResolver + Pydantic v2: the generated schema will not be supported by Bedrock's API at this moment

What do you think of this?

from powertools-lambda-python.

it's very hard to support multiple versions of OpenAPI spec with the same API.

That was not my intention. The point is that the version of Pydantic you have installed (1.x or 2.x) will determine which API spec will be produced.

In case Pydantic 1.x is installed, the default OpenAPI spec produced with get_openapi_json_schema() will be valid (since it's defaulting to 3.0.0). But if Pydantic 2.x is installed, the user should (now) specify version 3.1.0: get_openapi_json_schema(openapi_version="3.1.0").

I was thinking that get_openapi_json_schema could determine to use version 3.1.0 in case Pydantic 2.x is installed, so that get_openapi_json_schema() produces a valid output in both cases. I would consider this a bugfix rather than a breaking change (bug being that an invalid OpenAPI spec is produced).

from powertools-lambda-python.

thanks a lot for the feedback @kbakk :) We'll sync internally for a way out on this -- Ruben briefly mentioned it'd impact the upcoming Agent for Amazon Bedrock Event Handler feature at a first glance.

Luckily all of this will go away on v3 too when we drop Pydantic v1. We'll come back to you as soon as we have a meaningful update to share - appreciate your patience!!

from powertools-lambda-python.

According to Pydantic docs: JSON Schema, Pydantic is compaible with OpenAPI 3.1.0. Running get_openapi_json_schema(openapi_version="3.1.0") will then produce a valid schema.

I will leave this issue open, since I believe this could be addressed in an automatic fashion: if Pydantic <2.0 use 3.0.0, else use 3.1.0. Also the summary should only be added if using OpenAPI version >=3.1.0.

from powertools-lambda-python.

Hi @kbakk, thanks for filing the issue – we'll look into it to see if this is something we should add!

from powertools-lambda-python.

cc @rubenfonseca who's oncall this week and might have missed.

from powertools-lambda-python.

Looking at this now

from powertools-lambda-python.

@kabkk Thank you for the very detailed explanation! I need to run some extra checks on this, but the major problem I have right now is that it's very hard to support multiple versions of OpenAPI spec with the same API. And on the other hand I'm not 100% comfortable in bumping the default to 3.1.0. Do you have any feedback on what would be the best for you?

from powertools-lambda-python.

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

from powertools-lambda-python.

This is now released under 2.35.0 version!

from powertools-lambda-python.