[core] Add Support for Structured JSON Logging

[sebsto]

AWS Lambda now supports advanced logging controls that enable functions to emit logs in JSON structured format and control log level granularity. The Swift AWS Lambda Runtime should support these capabilities to provide developers with enhanced logging, filtering, and observability features.

Currently, the Swift runtime emits logs in plaintext (unstructured) format. Adding support for JSON structured logging would enable:

• Easier searching, filtering, and analysis of logs
• Automated log analysis and dashboard creation
• Compliance with OpenTelemetry (OTel) Logs Data Model
• Dynamic log level control without code changes

Current Behavior

1. Lambda functions emit logs in plaintext format
2. No native support for JSON structured logging
3. Log level control requires code changes
4. Difficult to query and filter logs programmatically

Expected Behavior

The Swift runtime should support the logging configuration provided by Lambda through environment variables:

1. Log Format: Support both Text (default) and JSON formats via AWS_LAMBDA_LOG_FORMAT
2. Application Log Level: Control granularity of application logs (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) via AWS_LAMBDA_LOG_LEVEL

Note: Custom runtimes are NOT responsible for emitting system logs (START, END, REPORT). The Lambda service handles these automatically.

JSON Log Format Structure

When JSON format is enabled, application logs should follow this structure:

{
  "timestamp": "2024-01-16T10:30:45.586Z",
  "level": "INFO",
  "message": "Processing request",
  "requestId": "79b4f56e-95b1-4643-9700-2807f4e68189"
}

Additional fields can be included based on the logging context:

• logger: The name/source of the logger
• Custom metadata fields from the application

Implementation Considerations

1. Configuration via Environment Variables

Lambda provides logging configuration through environment variables that custom runtimes should read:

• AWS_LAMBDA_LOG_FORMAT: Text or JSON (default: Text)
• AWS_LAMBDA_LOG_LEVEL: Application log level - TRACE, DEBUG, INFO, WARN, ERROR, FATAL (default: INFO)
• LOG_LEVEL: Legacy environment variable for log level (backward compatibility)

Log Level Precedence:

The runtime should support both AWS_LAMBDA_LOG_LEVEL (new) and LOG_LEVEL (existing) with the following precedence:

1. Text format (default):

   • If both LOG_LEVEL and AWS_LAMBDA_LOG_LEVEL are set: Use LOG_LEVEL (backward compatibility)
   • If only AWS_LAMBDA_LOG_LEVEL is set: Use it
   • If only LOG_LEVEL is set: Use it (existing behavior)
   • If neither is set: Use default log level

2. JSON format:

   • If both LOG_LEVEL and AWS_LAMBDA_LOG_LEVEL are set: Use AWS_LAMBDA_LOG_LEVEL and emit a warning
   • If only AWS_LAMBDA_LOG_LEVEL is set: Use it
   • If only LOG_LEVEL is set: Use it but emit a warning recommending AWS_LAMBDA_LOG_LEVEL
   • If neither is set: Use default log level

Important: When AWS_LAMBDA_LOG_FORMAT=Text (or not set), the runtime should continue working exactly as it does today with no changes. The implementation considerations below only apply when AWS_LAMBDA_LOG_FORMAT=JSON.

From the AWS documentation:

"When using a custom runtime, you can integrate Lambda's advanced logging controls by checking the value of the AWS_LAMBDA_LOG_FORMAT and AWS_LAMBDA_LOG_LEVEL environment variables and configuring your runtime's loggers accordingly."

2. Integration with swift-log

The Swift runtime uses the swift-log library for logging. When AWS_LAMBDA_LOG_FORMAT=JSON, we need to:

1. Create a custom LogHandler that supports JSON output
2. Respect the configured log level from AWS_LAMBDA_LOG_LEVEL
3. Include Lambda-specific metadata (requestId, traceId, etc.)
4. Format logs according to the expected structure

When AWS_LAMBDA_LOG_FORMAT=Text (default), continue using the existing logging implementation.

Logger Initialization Strategy:

The logger initialization should follow a two-phase approach:

1. Runtime Initialization (once per runtime instance):

   • Read AWS_LAMBDA_LOG_FORMAT and AWS_LAMBDA_LOG_LEVEL environment variables
   • Create a LoggingConfiguration object
   • Store configuration for use during invocations

2. Per-Request Logger Creation (once per invocation):

   • In the run loop, after receiving invocation metadata
   • Create a new Logger instance with request-specific metadata (requestID, traceID)
   • Use the appropriate LogHandler based on configuration (JSON or Text)
   • Pass this logger to LambdaContext

Why create a new Logger per invocation?

• Logger is a value type (struct) - copying is cheap
• The LogHandler is immutable - cannot be swapped after Logger creation
• For JSON format, request metadata must be in the handler constructor (not added as metadata)
• Clean separation between runtime-level and request-level logging
• No metadata cleanup needed after invocation

This approach ensures:

• Configuration is read once (performance)
• Each invocation gets a logger with correct request metadata
• The logger is automatically propagated through LambdaContext to all components and user handlers
• No changes needed to user code - they access context.logger as they do today

3. Structured Logging API

Provide a developer-friendly API for structured logging:

import AWSLambdaRuntime
import Logging

struct MyLambda: SimpleLambdaHandler {
    func handle(_ event: String, context: LambdaContext) async throws -> String {
        // Automatic JSON formatting when AWS_LAMBDA_LOG_FORMAT=JSON
        context.logger.info("Processing event", metadata: [
            "eventSize": "\(event.count)",
            "userId": "user123"
        ])
        
        context.logger.debug("Detailed debug information")
        context.logger.error("Error occurred", metadata: [
            "errorCode": "E001"
        ])
        
        return "Success"
    }
}

4. Log Level Filtering

When AWS_LAMBDA_LOG_LEVEL is set, implement efficient log level filtering:

// Only emit logs at or above the configured level
if logLevel >= configuredLevel {
    emitLog(message)
}

This allows operators to control log verbosity without code changes, regardless of whether the format is Text or JSON.

5. Supported Runtimes and Logging Methods

According to AWS documentation, for managed runtimes to support JSON application logs, they must use specific logging methods. For custom runtimes like Swift:

• The runtime must check AWS_LAMBDA_LOG_FORMAT and format logs accordingly
• The runtime must respect AWS_LAMBDA_LOG_LEVEL for filtering
• Application logs must be written to stdout/stderr
• JSON logs must include at minimum: timestamp, level, message, requestId

Proposed Changes

Files to Create/Modify

1. Sources/AWSLambdaRuntime/Logging/JSONLogHandler.swift (new)

   • Custom LogHandler implementation for JSON format
   • Include Lambda context metadata (requestId, traceId)
   • Format according to expected structure

2. Sources/AWSLambdaRuntime/Logging/LoggingConfiguration.swift (new)

   • Read AWS_LAMBDA_LOG_FORMAT, AWS_LAMBDA_LOG_LEVEL, and LOG_LEVEL from environment
   • Implement log level precedence rules (see above)
   • Emit warnings when both log level env vars are set
   • Validate and parse log levels
   • Provide configuration to runtime

3. Sources/AWSLambdaRuntime/Runtime/LambdaRuntime.swift (modify)

   • Initialize LoggingConfiguration once during runtime initialization
   • Store configuration as instance variable

4. Sources/AWSLambdaRuntime/Lambda.swift (modify)

   • In runLoop, create per-request logger using LoggingConfiguration.makeLogger()
   • Pass request-specific logger to LambdaContext
   • Remove manual metadata manipulation (requestID) - now handled by LogHandler

5. Sources/AWSLambdaRuntime/Docs.docc/ (modify)

   • Add documentation for structured logging
   • Provide examples of JSON logging usage
   • Document log level configuration

Actual Implementation

Implementation Status: ✅ Complete and tested on both macOS and Linux (Amazon Linux 2)

The implementation has been completed in the feature/structured-json-logging branch with the following files:

1. Sources/AWSLambdaRuntime/Logging/LoggingConfiguration.swift

Complete implementation with log level precedence rules. See the actual file for full details.

Key features:

• Reads AWS_LAMBDA_LOG_FORMAT and AWS_LAMBDA_LOG_LEVEL environment variables
• Supports legacy LOG_LEVEL for backward compatibility
• Implements precedence rules with appropriate warnings
• Provides makeLogger() factory method for per-request logger creation

2. Sources/AWSLambdaRuntime/Logging/JSONLogHandler.swift

Complete LogHandler implementation for JSON format. See the actual file for full details.

Key features:

• Uses JSONEncoder with .iso8601 date encoding strategy
• Includes request metadata (requestID, traceID) in every log entry
• Maps swift-log levels to AWS Lambda levels (TRACE, DEBUG, INFO, WARN, ERROR, FATAL)
• Outputs compact JSON (no pretty printing)
• Uses conditional Foundation imports for cross-platform compatibility:

  #if canImport(FoundationEssentials)
  import FoundationEssentials
  #else
  import Foundation
  #endif

Important Implementation Details:

1. Date Formatting: Uses JSONEncoder.dateEncodingStrategy = .iso8601 which is available and works correctly on both macOS and Linux (Amazon Linux 2). This produces RFC 3339 compliant timestamps like 2024-01-16T10:30:45.586Z.

2. Cross-Platform Compatibility: Uses conditional imports to support both FoundationEssentials (Linux) and Foundation (macOS):

   #if canImport(FoundationEssentials)
   import FoundationEssentials
   #else
   import Foundation
   #endif

3. Per-Request Logger Creation: Each invocation gets a new Logger instance with request-specific metadata. This is efficient because:

   • Logger is a struct (value type) - copying is cheap
   • LogHandler is immutable - cannot be swapped after creation
   • Request metadata (requestID, traceID) must be in the handler constructor for JSON format
   • No metadata cleanup needed after invocation

4. Compilation Verified:

   • ✅ macOS build successful
   • ✅ Linux (Amazon Linux 2) build successful via Docker
   • ✅ All 98 tests pass

Integration Points

In LambdaRuntime.swift:

public final class LambdaRuntime<Handler>: Sendable where Handler: StreamingLambdaHandler {
    let handlerStorage: SendingStorage<Handler>
    let logger: Logger
    let eventLoop: EventLoop
    let loggingConfiguration: LoggingConfiguration  // NEW
    
    public init(
        handler: sending Handler,
        eventLoop: EventLoop = Lambda.defaultEventLoop,
        logger: Logger = Logger(label: "LambdaRuntime")
    ) {
        self.handlerStorage = SendingStorage(handler)
        self.eventLoop = eventLoop
        
        // Initialize logger for runtime-level logging (before reading config)
        var log = logger
        log.logLevel = Lambda.env("LOG_LEVEL").flatMap { .init(rawValue: $0) } ?? logger.logLevel
        self.logger = log
        
        // NEW: Read logging configuration (may emit warnings via logger)
        self.loggingConfiguration = LoggingConfiguration(logger: self.logger)
    }
}

In Lambda.swift run loop:

public static func runLoop<RuntimeClient: LambdaRuntimeClientProtocol, Handler>(
    runtimeClient: RuntimeClient,
    handler: Handler,
    loggingConfiguration: LoggingConfiguration,  // NEW parameter
    logger: Logger  // Keep for runtime-level logging
) async throws where Handler: StreamingLambdaHandler {
    var handler = handler
    
    while !Task.isCancelled {
        logger.trace("Waiting for next invocation")
        let (invocation, writer) = try await runtimeClient.nextInvocation()
        
        // NEW: Create per-request logger with request metadata
        let requestLogger = loggingConfiguration.makeLogger(
            label: "Lambda",
            requestID: invocation.metadata.requestID,
            traceID: invocation.metadata.traceID
        )
        
        // Pass request-specific logger to context
        try await handler.handle(
            invocation.event,
            responseWriter: writer,
            context: LambdaContext(
                requestID: invocation.metadata.requestID,
                traceID: invocation.metadata.traceID,
                tenantID: invocation.metadata.tenantID,
                invokedFunctionARN: invocation.metadata.invokedFunctionARN,
                deadline: LambdaClock.Instant(
                    millisecondsSinceEpoch: invocation.metadata.deadlineInMillisSinceEpoch
                ),
                logger: requestLogger  // NEW: Per-request logger
            )
        )
    }
}

SAM/CloudFormation Configuration Example

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .build/plugins/AWSLambdaPackager/outputs/MyLambda/MyLambda.zip
      Handler: bootstrap
      Runtime: provided.al2
      LoggingConfig:
        LogFormat: JSON
        ApplicationLogLevel: DEBUG

Testing Requirements

Completed Tests

• Compilation verified on macOS (Swift 6.1)
• Compilation verified on Linux (Amazon Linux 2 via Docker)
• All existing tests pass (98 tests)
• Cross-platform Foundation imports working correctly

Remaining Tests

• Unit tests for JSONLogHandler
  • Test JSON output format structure
  • Test log level mapping (trace→TRACE, debug→DEBUG, etc.)
  • Test metadata inclusion
  • Test requestID and traceID inclusion
  • Test ISO8601 timestamp format
• Unit tests for LoggingConfiguration parsing
  • Test AWS_LAMBDA_LOG_FORMAT parsing (Text/JSON)
  • Test AWS_LAMBDA_LOG_LEVEL parsing
  • Test LOG_LEVEL parsing (backward compatibility)
• Test log level precedence rules
  • Text format: LOG_LEVEL preferred over AWS_LAMBDA_LOG_LEVEL
  • JSON format: AWS_LAMBDA_LOG_LEVEL preferred over LOG_LEVEL
  • Test with only AWS_LAMBDA_LOG_LEVEL set
  • Test with only LOG_LEVEL set
  • Test with neither set (default behavior)
• Test warning messages
  • Warning when both log level env vars set with JSON format
  • Warning when only LOG_LEVEL set with JSON format
  • Debug message when both set with Text format
• Integration tests with different log levels (TRACE, DEBUG, INFO, WARN, ERROR, FATAL)
• Verify JSON format structure matches expected format
• Test log level filtering
• Verify backward compatibility with text format (default)
• Verify backward compatibility with LOG_LEVEL environment variable
• Test with CloudWatch Logs Insights queries (manual/integration)
• Performance testing (JSON serialization overhead)
• Test that requestId is included in logs
• Test with Lambda Managed Instances (JSON format is mandatory)

Benefits

1. Improved Observability: Structured logs are easier to query and analyze
2. Better Debugging: Dynamic log level control without redeployment
3. Cost Optimization: Reduce log volume in production with higher log levels
4. CloudWatch Integration: Better integration with CloudWatch Logs Insights
5. Standards Alignment: Follows AWS Lambda logging conventions
6. Lambda Managed Instances Support: Required for LMI functions (JSON only)

Migration Considerations

• Default behavior remains unchanged (text format)
• Opt-in via Lambda configuration (LoggingConfig)
• Existing functions continue to work without changes
• LOG_LEVEL environment variable continues to work for backward compatibility
• When using JSON format, prefer AWS_LAMBDA_LOG_LEVEL over LOG_LEVEL
• Runtime emits warnings when both log level env vars are set to help with migration
• Breaking change warning: Switching to JSON may affect existing log parsing pipelines

References

• AWS Blog: Introducing advanced logging controls for AWS Lambda functions
• AWS Lambda Developer Guide: Configuring JSON and plain text log formats
• AWS Lambda Developer Guide: Log-level filtering
• AWS Lambda Developer Guide: Building a custom runtime
• AWS Lambda Developer Guide: Environment variables
• swift-log GitHub Repository

Related Issues

• [core] Implement X-Ray Trace ID Propagation via Environment Variable #633 - X-Ray Trace ID Propagation (trace ID should be included in JSON logs)

Labels

• enhancement
• logging
• observability
• cloudwatch

Priority

Medium-High: This is a significant enhancement that improves observability and aligns with AWS Lambda best practices. It's also required for Lambda Managed Instances support (which mandates JSON format).