feat(idempotency): adding redis as idempotency backend (#2567)

* (redis): initial commit

* feat(redis/idempotency): creating redis connections

* feat(redis/idempotency): creating redis connections

* feat(redis/idempotency): fixing import

* feat(redis/idempotency): adding base class

* feat(redis/idempotency): adding logic to get record

* feat(redis/idempotency): adding expiry timeout

* feat(redis) - refactoring connection and fixing mypy errors

* feat(redis) - removing wrong print

* feat(redis) - removing fields and adding additional logic to validate the idempotency key

* feat(redis) - adding redis as dev dependency

* Update idempotency.md

Signed-off-by: Vandita Patidar <vandita.vk20@gmail.com>

* Update idempotency.md

Signed-off-by: Vandita Patidar <vandita.vk20@gmail.com>

* resolve regarding to ruben's comment

* resolve regarding to ruben's comment

* local test, minor fixes

* add redis to extra dep

* fix git issue

* fix docstring

* fix poetry, address some Leandro's suggestion

* change redis connection

* add mock redis, redis validation, sentinel support

* fix test

* add redis

* add doc, few todos still need to address

* add docs, test. Removed Connection

* fix test on delete

* add types-redis to pyproj

* fix Literal for 3.7

* add comment for delete

* add redis as dev dep

* fix poetry

* Simplifying DX

* remove redis-config in doc,test

* handle race condition

* remove todo

* change to use redis.set

* support decode_response=False

* fix mock redis

* add test for no decode

* fix docstring

* fix coverage

* add a test case to demostrate race condition

* add a race condition test for empty record

* add abs_lambda_path, protocol for redis client

* fix typing

* remove awaitable

* optimize protocol

* Fix Bandit issue

* Refactoring integration tests to use testcontainers

* Removing code smell

* fix makefile, remove sentinel setup

* Adding e2e tests

* Testing pipeline

* Removing things

* Removing things

* Fixing docstring and removing old code

* Improving the documentation

* Highlights + code removal

* Removing unnecessary tests

* Removing unnecessary tests

* Documentation

* Addressing initial Ruben's feedback

* Addressing Ruben's feedback - documentation

* Addressing Ruben's feedback - docstring

* Addressing Ruben's feedback - SSL

* Addressing Ruben's feedback - db_index

* Addressing Ruben's feedback - db_index

* Addressing Ruben's feedback - db_index

* redis comment improvements, minor refactor.

* fix example and docstring import

* fix import in inegration test

* Addressing Ruben's feedback - Redis name

* Addressing Ruben's feedback - Comments

* Minor changes in the documentation

* Minor changes in the documentation

* Removing Redis as builti-in dependency in our layer

* Adding Redis to install when creating a new env

* Adressing Ruben's feedback

* Making sonar happy

* e2e failing due to wrong import

---------

Signed-off-by: Vandita Patidar <vandita.vk20@gmail.com>
Signed-off-by: Leandro Damascena <lcdama@amazon.pt>
Co-authored-by: Leandro Damascena <leandro.damascena@gmail.com>
Co-authored-by: Vandita Patidar <vandita.vk20@gmail.com>
Co-authored-by: Leandro Damascena <lcdama@amazon.pt>
Co-authored-by: Cavalcante Damascena <lcdama@b0be8355743f.ant.amazon.com>

Author: 4 people

.github/workflows/quality_check_pydanticv2.yml

@@ -58,7 +58,9 @@ jobs:
           python-version: ${{ matrix.python-version }}
           cache: "poetry"
       - name: Replacing Pydantic v1 with v2 > 2.0.3
-        run: poetry add "pydantic=^2.0.3"
+        run: |
+          rm -rf poetry.lock
+          poetry add "pydantic=^2.0.3"
       - name: Install dependencies
         run: make dev
       - name: Test with pytest

Makefile

@@ -8,13 +8,13 @@ dev:
 	pip install --upgrade pip pre-commit poetry
 	poetry config --local virtualenvs.in-project true
 	@$(MAKE) dev-version-plugin
-	poetry install --extras "all datamasking-aws-sdk"
+	poetry install --extras "all datamasking-aws-sdk redis"
 	pre-commit install
 
 dev-gitpod:
 	pip install --upgrade pip poetry
 	@$(MAKE) dev-version-plugin
-	poetry install --extras "all datamasking-aws-sdk"
+	poetry install --extras "all datamasking-aws-sdk redis"
 	pre-commit install
 
 format:

aws_lambda_powertools/shared/functions.py

@@ -6,6 +6,7 @@
 import os
 import warnings
 from binascii import Error as BinAsciiError
+from pathlib import Path
 from typing import Any, Dict, Generator, Optional, Union, overload
 
 from aws_lambda_powertools.shared import constants
@@ -250,3 +251,32 @@ def dataclass_to_dict(data) -> dict:
     import dataclasses
 
     return dataclasses.asdict(data)
+
+
+def abs_lambda_path(relative_path: str = "") -> str:
+    """Return the absolute path from the given relative path to lambda handler.
+
+    Parameters
+    ----------
+    relative_path : str, optional
+        The relative path to the lambda handler, by default an empty string.
+
+    Returns
+    -------
+    str
+        The absolute path generated from the given relative path.
+        If the environment variable LAMBDA_TASK_ROOT is set, it will use that value.
+        Otherwise, it will use the current working directory.
+        If the path is empty, it will return the current working directory.
+    """
+    # Retrieve the LAMBDA_TASK_ROOT environment variable or default to an empty string
+    current_working_directory = os.environ.get("LAMBDA_TASK_ROOT", "")
+
+    # If LAMBDA_TASK_ROOT is not set, use the current working directory
+    if not current_working_directory:
+        current_working_directory = str(Path.cwd())
+
+    # Combine the current working directory and the relative path to get the absolute path
+    absolute_path = str(Path(current_working_directory, relative_path))
+
+    return absolute_path

aws_lambda_powertools/utilities/idempotency/__init__.py

@@ -11,4 +11,10 @@
 
 from .idempotency import IdempotencyConfig, idempotent, idempotent_function
 
-__all__ = ("DynamoDBPersistenceLayer", "BasePersistenceLayer", "idempotent", "idempotent_function", "IdempotencyConfig")
+__all__ = (
+    "DynamoDBPersistenceLayer",
+    "BasePersistenceLayer",
+    "idempotent",
+    "idempotent_function",
+    "IdempotencyConfig",
+)

aws_lambda_powertools/utilities/idempotency/exceptions.py

@@ -83,3 +83,21 @@ class IdempotencyNoSerializationModelError(BaseError):
     """
     No model was supplied to the serializer
     """
+
+
+class IdempotencyPersistenceConfigError(BaseError):
+    """
+    The idempotency persistency configuration was unsupported
+    """
+
+
+class IdempotencyPersistenceConnectionError(BaseError):
+    """
+    Idempotency persistence connection error
+    """
+
+
+class IdempotencyPersistenceConsistencyError(BaseError):
+    """
+    Idempotency persistency consistency error, needs to be removed
+    """

aws_lambda_powertools/utilities/idempotency/persistence/base.py

@@ -374,7 +374,6 @@ def save_inprogress(self, data: Dict[str, Any], remaining_time_in_millis: Option
             now = datetime.datetime.now()
             period = datetime.timedelta(milliseconds=remaining_time_in_millis)
             timestamp = (now + period).timestamp()
-
             data_record.in_progress_expiry_timestamp = int(timestamp * 1000)
         else:
             warnings.warn(

aws_lambda_powertools/utilities/idempotency/persistence/dynamodb.py

@@ -66,6 +66,8 @@ def __init__(
             DynamoDB attribute name for status, by default "status"
         data_attr: str, optional
             DynamoDB attribute name for response data, by default "data"
+        validation_key_attr: str, optional
+            DynamoDB attribute name for hashed representation of the parts of the event used for validation
         boto_config: botocore.config.Config, optional
             Botocore configuration to pass during client initialization
         boto3_session : boto3.Session, optional