mousazeidbaker / aws-lambda-typing Goto Github PK

Not sure how much more broadly this issue extends, but I figured I'd share types relating to new "HTTP" API Lambda Authorizers seem to be missing.

Docs links:

• For "REST" APIs (what your types seem to model): https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-lambda-authorizer-input.html
• For "HTTP" APIs (seems to be missing): https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html#http-api-lambda-authorizer.payload-format

https://docs.aws.amazon.com/controltower/latest/userguide/lifecycle-events.html

Support for custom SNS events done via sns:Publish.

After testing API proxy V1 lambdas, all of the response keys seem optional and have default values (meaning we can add total=False)

Also, it appears that the body can also be bool, float and None and the response will still be valid.

In events/s3_batch.py, S3BatchResponseResult.resultCode is typed too loosely as str, but should be the same as S3BatchResponse.treatMissingKeysAs.

Consider adding another type for the literals:

S3BatchResponseResultCode = Literal[
    "Succeeded",
    "TemporaryFailure",
    "PermanentFailure",
]

Then update the types of S3BatchResponseResult.resultCode and S3BatchResponse.treatMissingKeysAs to S3BatchResponseResultCode.

Hi @MousaZeidBaker,

Thanks for the great project. We've been using it for a while.

I wonder if Powertools for AWS Lambda (Python) supersedes this project?

Recently typing-extensions switched to semantic versioning and bumped the version to 4.0.0

The library dependencies right now specify ^3.10.0 which means 4.X is deemed incompatible and as such the library isn't compatible with the new versioning scheme.

Also the only feature this library needs is TypedDict so you can forego the dependency on typing-extensions for Python versions starting from 3.8
I don't know if poetry allows the same controls but in vanilla setup.cfg you can declare it like this:

install_requires =
    typing-extensions; python_version < "3.8"

First of all, thanks a lot for the typing! Really helpful.

Sorry for the (maybe) dumb question but could the TypeDict have optional keys instead of a total=False?

As an S3Object, could always have some key but the value could be null ?

So

class SomeType(TypedDict, total=False): 
    myAlwaysPresentKey: str

would become

class SomeType(TypedDict): 
    myAlwaysPresentKey: str | None # Or Optional[str] for python < 3.10

else, it could also be NotRequired which indicates to python that the key can be not here (without total=False which makes all the keys NotRequired)

class SomeType(TypedDict): 
    myAlwaysPresentKey: NotRequired[str] # If key can be optional but when it's here it's sure it has a value
# Or 
class SomeType(TypedDict): 
    myAlwaysPresentKey: NotRequired[str | None] # If key and value can be None

That would make code like below compliant with type checkers :

# Could not access the item in TypedDict
# "s3" is not a required key in "S3EventRecord", so access may result in a runtime exception
s3_object: S3 = event["Records"][0]["s3"]

Thanks in advance and sorry if indeed all keys are optional then my question doesn't make sense.

When running mypy src, I see the following errors:

src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "RequestType" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "RequestId" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "ResponseURL" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "ResourceType" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "LogicalResourceId" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "StackId" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "ResourceProperties" while merging
src/aws_lambda_typing/events/cloud_formation_custom_resource.py:83: error: Overwriting TypedDict field "PhysicalResourceId" while merging
Found 8 errors in 1 file (checked 34 source files)

This occurs because of the following in the indicated file (doc comments removed):

class CloudFormationCustomResourceEvent(
    CloudFormationCustomResourceUpdate,
    CloudFormationCustomResourceDelete,
):
    pass

and both CloudFormationCustomResourceUpdate and CloudFormationCustomResourceDelete extend CloudFormationCustomResourceCommon.

I don't know the reasoning behind structuring the types this way, but perhaps a slight refactoring along the following lines would be an acceptable way to eliminate these errors, and also provide additional type safety.

First, remove RequestType from CloudFormationCustomResourceCommon so that each extension is required to specify the corresponding Literal type.

Then, define CloudFormationCustomResourceCreate, which is missing:

class CloudFormationCustomResourceCreate(
    CloudFormationCustomResourceCommon,
    total=False,
):
    RequestType: Literal["Create"]

Similarly, for CloudFormationCustomResourceUpdate and CloudFormationCustomResourceDelete, set RequestType to Literal["Update"] and Literal["Delete"], respectively.

Finally, define CloudFormationCustomResourceEvent as Union[CloudFormationCustomResourceCreate, CloudFormationCustomResourceUpdate, CloudFormationCustomResourceDelete] to eliminate the mypy errors above.

We are using EventBridge, which as you no doubt know is the successor to CloudWatch Events -

Amazon EventBridge is the preferred way to manage your events. CloudWatch Events and EventBridge are the same underlying service and API, but EventBridge provides more features.

The CloudWatchEventsMessageEvent provides the correct typing, but the name adds a bit of confusion since it adds some inconsistency with naming. It's easy for us to alias it internally but I think it would be good if this library had Event Bridge natively, e.g. EventBridgeEvent.

Thanks for providing this library since it allows for a much nicer developer experience.

Thank you for the wonderful product.

I'm considering contributing in the future, but the tests do not pass for the master branch. Could you provide some additional information on the Getting started section of CONTRIBUTING.md?

Here's what I've tried:

To use the pure environment, boot Python 3.10 official docker image with the following command:

docker run -it --rm python:3.10 bash

Upon starting the bash prompt inside the container, I executed the following:

# install poetry
curl -sSL https://install.python-poetry.org | python3 -
alias poetry=/root/.local/bin/poetry
# git clone
git clone https://github.com/MousaZeidBaker/aws-lambda-typing.git
cd aws-lambda-typing/

# commands from CONTRIBUTING.md
poetry install --sync && poetry shell
pre-commit install --install-hooks --overwrite
pre-commit run --all-files
pytest tests

As a result of the final pytest, I received the following:

bash: pytest: command not found

It appears that pytest cannot be used. I suspected that the reason was the absence of pytest in pyproject.toml, so I added pytest to dev-dependencies section.

  [tool.poetry.dependencies]
  python = "^3.6"
+ pytest = "^7.0"
  typing-extensions = { version = "^4.1.1", python = "<3.8" }

After making the changes to pyproject.toml as mentioned above, I executed the following command.

alias poetry=/root/.local/bin/poetry
poetry lock && poetry install --sync
pytest tests

The test results were 1 failed, 51 passed, 1 error. As this is the master branch, I was expecting all tests to pass, but that wasn't the case.

I suspect there might be some issues with my environment setup. If there are any differences between my environment and those of the contributors, I would greatly appreciate it if you could let me know.

For your reference, I have attached the results of the final pytest: pytest_result.txt

I'm writing a lambda function receiving DynamoDB stream events.
It replies a StreamsEventResponse to report failed items in partial error case, so that the AWS re-delivers only the failed event items.
I would appreciate if that response type is provided.

Document for the StreamsEventResponse is below:
https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html#streams-batchfailurereporting-syntax

At least, the following keys are required to work reporting the item failures.

{ 
  "batchItemFailures": [ 
        {
            "itemIdentifier": "<id str>"
        }
    ]
}

Ref: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_StreamRecord.html#:~:text=before%20it%20was%20modified.-,Type%3A%20String%20to%20AttributeValue%20object%20map,-Key%20Length%20Constraints%3A%20Maximum
as well as
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodbstreams.html#:~:text=%22BOOL%22%3A%20true-,OldImage,-(dict)%20%2D%2D

Looks like OldImage isn't optional, just like NewImage isn't.

Also, it looks like many fields are annotated as Optional but the docs don't state they are (say userIdentity and ApproximateCreationDateTime in the same file). Is this the desired behaviour?

We should be able to easily import deeper types other than what is defined in aws_lambda_typing/__init__.py. Editors have a hard time navigating the types when I need to explicitly define types for classes like SQSMessage. Currently I have to manually import them with

from aws_lambda_typing.events.s3 import S3EventRecord
# or

from aws_lambda_typing.events import s3

for example. But vscode and pycharm isn't able to find the paths easily. Though pycharm is a bit better at it.
What seems to work well for me is to update the imports like so

# aws_lambda_typing/__init__.py
from aws_lambda_typing import context, events, responses

then in each of the 3 types, in their init files, just import the files. so for events, it would be

# aws_lambda_typing/events/__init__.py

from . import (api_gateway_proxy, cloud_watch_events, cloud_watch_logs,
               code_pipeline, config, dynamodb_stream, kinesis_firehose,
               kinesis_stream, mq, s3, s3_batch, ses, sns, sqs

This should make importing easier. It will likely break old imports but in the top level init, you can keep those in as python wont import a file again once it's been loaded. This allows me to do this, and have the editors easily know what is where.

from aws_lambda_typing import context, events

def handler(event: events.sqs.SQSEvents, context: context.Context):
    pass

It would be nice for the README to specify which python versions are supported.

Also, 3.7 is now not supported. Is that because of missing Literal, TypedDIct or something? Could it be made supported?

The RequestContextV1 structure is missing the optional field operationName.

The operationName field is associated with the SDK settings. I know two ways to configure this.

First, While in the AWS API Gateway console, when you are viewing a method for a resource. Click into the "Method Request" section. Expand the "SDK Setting" section. By default this is blank. If you were to set to to something like "SomeOperation", and save. Then go back and test this the method. You will see in the event object passed to the lambda function that it will contain operationName and its value will be SomeOperation.

Second, this operationName is automatically configured if you deployed a restapi from a openapi document where you use the field operationId.

I'm getting this error when I try to install the package with pip

(.venv) PS D:\Programming\Python\lmbdFnct> pip install aws-lambda-typing
ERROR: Could not find a version that satisfies the requirement aws-lambda-typing (from versions: none)
ERROR: No matching distribution found for aws-lambda-typing

Minimal version

Is possible downgrade to 3.7? I use 3.7 to test the lambda locally.

Certain fields are able to be null values. For example, here are the AWS docs for an API Gateway Event:

https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html#apigateway-example-event

This body field, for example, is currently set to str

https://github.com/MousaZeidBaker/aws-lambda-typing/blob/master/src/aws_lambda_typing/events/api_gateway_proxy.py#L181

I think it should be body: Optional[str]

Let me know if I should do a PR

As stated in PEP-589 - partial totality can be achieved with inheritance.
Using Optional means that a field can have None as a value, which is not the case in most AWS types.

aws_lambda_typing surprises me, in that I'm accustomed to an emphasis in lambda coding on the lightest, fastest computations possible. As fond as I am of typing rigor, how can my lambda definitions afford to import aws_lambda_typing, rather than just crudely and "locally" approximating an SQSEvent as a dict, and a Context as an object?

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html

Similar to APIGateway, lambdas can be triggered by websocket events.

First want to say, love the project, thanks for maintaining for community.

Just wanted to give a heads up that it appears prior versions of the project when specified in a requirements.txt ex 1.0.3 result in actually pulling 2.0.0.