Hi, not sure if this is the right place or it should rather go to ddtrace or somewhere else:
Reproduction repo with CDK: https://github.com/maxfriedrich/datadog-cloud-payload-tagging-repro
The cloud request/response payload tagging feature causes a big slowdown for us for any DynamoDB query, even when "disabled" with the environment variable value "$.*" (0.xx seconds to > 10 seconds query), with the following setup:
- Docker Lambda based on the Lambda Python 3.11 (or 3.13) base image
- Installing the Datadog Lambda extension into the container
- Installing datadog-lambda and ddtrace with into the container
- Running the function with
datadog_lambda.handler
(this function outgrew the zip size limit, this is why it's built like this -- is this a supported setup?)
We can work around it by setting the environment variable values to "", fully disabling the feature, not just redacting all fields.
The problem only occurs with a Docker based Lambda function, the handler from my reproduction is fine in the Python runtime and the Datadog extension and Lambda layers.
Output from my test script:
Testing slow function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function1904AA941-QLjJ2nwy96AV
Execution time: 11.545 seconds
Execution time: 10.865 seconds
Execution time: 11.279 seconds
Execution time: 11.604 seconds
Execution time: 11.659 seconds
Testing fast function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function28EA0674A-36P7xO9x9CbG
Execution time: 0.235 seconds
Execution time: 0.188 seconds
Execution time: 0.168 seconds
Execution time: 0.194 seconds
Execution time: 0.206 seconds
Do you have any ideas what could cause this? Feel free to move the issue if it belongs somewhere else. Thank you very much!
Hi, not sure if this is the right place or it should rather go to ddtrace or somewhere else:
Reproduction repo with CDK: https://github.com/maxfriedrich/datadog-cloud-payload-tagging-repro
The cloud request/response payload tagging feature causes a big slowdown for us for any DynamoDB query, even when "disabled" with the environment variable value
"$.*"(0.xx seconds to > 10 seconds query), with the following setup:datadog_lambda.handler(this function outgrew the zip size limit, this is why it's built like this -- is this a supported setup?)
We can work around it by setting the environment variable values to
"", fully disabling the feature, not just redacting all fields.The problem only occurs with a Docker based Lambda function, the handler from my reproduction is fine in the Python runtime and the Datadog extension and Lambda layers.
Output from my test script:
Do you have any ideas what could cause this? Feel free to move the issue if it belongs somewhere else. Thank you very much!