Comments (11)
okay managed to reproduce - validation at nested fields aren't working, that's why tests worked but @dreamorosi payload didn't. Docs also use a single key/value, compound would also work, but nested fields didn't.
NOTE. This is not related to ReturnValueOnConditionCheckFailure
. Our tests caught the change before, and that's why @rubenfonseca wasn't able to validate.
I've created a functional test along w/ some basic tooling (we sorely need testing tooling) to make it easier to identify the root cause. I suspect it might be down to ordering as hashing it's occurring.
def test_idempotent_lambda_already_completed_with_validation_bad_nested_payload(
persistence_store: DynamoDBPersistenceLayer,
timestamp_future,
lambda_context,
request: FixtureRequest,
):
# GIVEN an idempotency config with a compound idempotency key (refund, customer_id)
# AND with payload validation key to prevent tampering
validation_key = "details"
idempotency_config = IdempotencyConfig(
event_key_jmespath='["refund_id", "customer_id"]', payload_validation_jmespath=validation_key
)
# AND a previous transaction already processed in the persistent store
transaction = {
"refund_id": "ffd11882-d476-4598-bbf1-643f2be5addf",
"customer_id": "9e9fc440-9e65-49b5-9e71-1382ea1b1658",
"details": {"company_name": "Parker, Johnson and Rath", "currency": "Turkish Lira"},
}
stubber = stub.Stubber(persistence_store.client)
ddb_response = build_idempotency_put_item_response_stub(
data=transaction,
expiration=timestamp_future,
status="COMPLETED",
request=request,
validation_data=transaction[validation_key],
)
stubber.add_client_error("put_item", "ConditionalCheckFailedException", modeled_fields=ddb_response)
stubber.activate()
# AND an upcoming tampered transaction
tampered_transaction = copy.deepcopy(transaction)
tampered_transaction["details"]["currency"] = "Euro"
@idempotent(config=idempotency_config, persistence_store=persistence_store)
def lambda_handler(event, context):
return event
# WHEN the tampered request is made
# THEN we should raise
with pytest.raises(IdempotencyValidationError):
lambda_handler(tampered_transaction, lambda_context)
stubber.assert_no_pending_responses()
stubber.deactivate()
from powertools-lambda-python.
Thank you for opening this detailed issue Andrea! I'll try to finish the PR
from powertools-lambda-python.
not at all. First, better safe than sorry. Second, this was an useful exercise to improve our test coverage on payload validation as it was overly simplistic :)
If anything, keep it coming @dreamorosi. It was also useful exercise to discover the ordering issue in TS nevertheless.
In case I haven't said it this month... it's a pleasure working with you :)
from powertools-lambda-python.
I have tried implementing the fix I was suggesting in the OP but I wasn't able to make the tests pass :/
from powertools-lambda-python.
So acording to
from powertools-lambda-python.
couldn't find the test for this enhancement (that potentially triggered this regression) -- looking at the history now
from powertools-lambda-python.
found the test that was modified with this feature but seems to be working as expected, will look at E2E and create one if there isn't one to be absolutely sure.
Tasks
- Reproduce
- Create E2E for payload validation (in addition to functional to be extra safe)
- Create a test for nested field validation error
- Create functional test
E2E Handler for payload validation
import os
import uuid
from aws_lambda_powertools.utilities.idempotency import (
DynamoDBPersistenceLayer,
IdempotencyConfig,
idempotent,
)
TABLE_NAME = os.getenv("IdempotencyTable", "")
persistence_layer = DynamoDBPersistenceLayer(table_name=TABLE_NAME)
config = IdempotencyConfig(event_key_jmespath='["refund_id", "customer_id"]', payload_validation_jmespath="amount")
@idempotent(config=config, persistence_store=persistence_layer)
def lambda_handler(event, context):
return {"request": str(uuid.uuid4())}
E2E test for payload validation tampering
@pytest.mark.xdist_group(name="idempotency")
def test_payload_tampering_validation(payload_tampering_validation_fn_arn: str):
# GIVEN a transaction with the idempotency key on refund and customer IDs
transaction = {
"refund_id": "ffd11882-d476-4598-bbf1-643f2be5addf",
"customer_id": "9e9fc440-9e65-49b5-9e71-1382ea1b1658",
"amount": 100,
}
# AND a second transaction with the exact idempotency key but different amount
second_transaction = {**transaction, "amount": 1000}
# WHEN we make both requests to a Lambda Function that enabled payload validation
data_fetcher.get_lambda_response(
lambda_arn=payload_tampering_validation_fn_arn, payload=json.dumps(transaction)
)
# THEN we should receive a payload validation error in the second request
with pytest.raises(RuntimeError, match="Payload does not match stored record"):
data_fetcher.get_lambda_response(
lambda_arn=payload_tampering_validation_fn_arn,
payload=json.dumps(second_transaction),
)
from powertools-lambda-python.
Turns out I only managed to reproduce because of fat finger ... I used the wrong validation key after editing the wrong file (same name but local cache for E2E).
I can't reproduce with both functional and E2E tests. I'm gonna have another look at the original snippet see if I can spot anything, and push a PR to improve our tests regardless.
That said, TS has another issue I thought it affected us... ordering. We deep sort payloads both incoming and for validation before generating a hash.
Phew!
from powertools-lambda-python.
Waiting for E2E to run with this new case (functional tests are passing) -- I can't for the love of me reproduce it. Added exact same nesting and change in functional and E2E test above in the PR -- I'll try one last time with a manual Lambda tomorrow.
https://github.com/aws-powertools/powertools-lambda-python/actions/runs/7976558714
from powertools-lambda-python.
I'll try one last time with a manual Lambda tomorrow.
No need.
I have deployed the manual stack from scratch & ran the two payloads again and the second request is throwing as expected.
Given this I can only assume that I must have made a mistake. Apologies for making you run in circles and wasting your time @rubenfonseca & @heitorlessa.
I'll be more thorough in the future.
from powertools-lambda-python.
⚠️ COMMENT VISIBILITY WARNING⚠️
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
from powertools-lambda-python.
Related Issues (20)
- Maintenance: Refactor artifact name strategy for parallel uploads HOT 5
- Maintenance: Roadmap update HOT 3
- Bug: Logs populating into cloudwatch as multiple lines instead of a single object HOT 10
- Maintenance: Update readme to style badge and remove email HOT 3
- Maintenance: Layer ARN release update is partially working HOT 4
- Docs: remove leftover announcement banner HOT 2
- [I Made This]: reinvent session & code with Heitor HOT 3
- Docs: Fix feature flags documentation HOT 4
- Bug: Swagger generated UI has JSON format invalid URL HOT 4
- [I Made This]: Serverless API Documentation with Powertools for AWS HOT 1
- Feature request:Provide event handler for Lexv2 lambda integration HOT 1
- Bug: idempotency logic should first validate the payload and then cache HOT 2
- Bug: Event Handler Data Validation `KeyError: 'multiValueHeaders'` regression when running locally HOT 3
- Docs: Improve install section with minimal dependencies first HOT 2
- Static typing: missing @overload to ensure return type is a str when default_value is set HOT 3
- Bug: RequestValidationError handling behavior should not be affected if we add exception handler for Exception HOT 7
- Feature request: Function to retrieve mutli-value query string parameters HOT 2
- Feature request: Add event object sent from Cloudwatch Alarms HOT 7
- Bug: CORS headers not appending to API Gateway REST API responses HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from powertools-lambda-python.