feat: add support for Message-ID handling and idempotency in send API

[max-kuklin]

Add Idempotency Support for Email Sending APIs

Summary

We send a large volume of emails through the Postal API with instances installed on remote ISPs. Network hiccups are inevitable in this setup, causing API requests to timeout or fail intermittently. Without idempotency, retrying these failed requests results in duplicate emails being sent to recipients.

This PR implements idempotency for the API endpoints (/api/v1/send/message and /api/v1/send/raw) using RFC 5322 Message-ID headers. Clients can now safely retry failed requests without sending duplicate emails, using client-defined Message-IDs for precise control over deduplication.

Changes

API Enhancements

/api/v1/send/message endpoint:

• New optional message_ids parameter accepts a hash mapping recipient email addresses to custom Message-IDs
• Enables per-recipient idempotency control
• Returns message_id and existing flag for each recipient in the response

/api/v1/send/raw endpoint:

• Automatically extracts Message-ID from raw email headers
• Performs duplicate detection per recipient
• Returns message_id and existing flag for each recipient in the response
• No API parameter changes required

Response Format

When Message-IDs are provided or detected:

{
  "status": "success",
  "data": {
    "messages": {
      "user@example.com": {
        "id": 123,
        "token": "abc123",
        "message_id": "<unique-id@example.com>",
        "existing": false
      }
    }
  }
}

Validation

• Message-IDs must follow RFC 5322 format: local-part@domain (angle brackets optional in API but stripped for storage)
• Invalid Message-IDs return InvalidMessageID error with clear format requirements
• Validation occurs at model level before database interaction

Implementation Details

• Model: OutgoingMessagePrototype enhanced with message_ids attribute, validation logic, and per-recipient duplicate detection
• Controller: SendController updated to accept message_ids parameter and extract Message-ID from raw email headers
• Database: Uses existing message_id column (varchar 255) with partial index on first 8 characters
• Duplicate Detection: Queries existing messages by Message-ID and recipient before creating new records
• No Migration Required: Leverages existing database schema

Testing

Comprehensive test suite with 46 passing tests (0 failures):

• 13 tests for /message endpoint idempotency (basic flow, per-recipient IDs, duplicates, partial duplicates, angle brackets, non-hash params)
• 10 tests for /raw endpoint idempotency (Message-ID extraction, duplicates, partial duplicates, angle brackets)
• 3 tests for Message-ID format validation (invalid format, missing local part, spaces)
• 20 existing tests for backward compatibility

Usage Examples

/message endpoint with idempotency:

POST /api/v1/send/message
{
  "to": ["user1@example.com", "user2@example.com"],
  "from": "sender@example.com",
  "subject": "Test",
  "plain_body": "Hello",
  "message_ids": {
    "user1@example.com": "unique-id-1@example.com",
    "user2@example.com": "unique-id-2@example.com"
  }
}

/raw endpoint (automatic Message-ID extraction):

POST /api/v1/send/raw
{
  "mail_from": "sender@example.com",
  "rcpt_to": ["user@example.com"],
  "data": "Message-ID: <unique-id@example.com>\r\nFrom: sender@example.com\r\n..."
}

Benefits

1. Prevents Duplicate Sends: Clients can safely retry failed requests without sending duplicate emails
2. Per-Recipient Control: Different Message-IDs for each recipient in multi-recipient messages
3. RFC 5322 Compliant: Uses standard email Message-ID format
4. Backward Compatible: Existing clients work unchanged; idempotency is optional
5. No Database Changes: Uses existing schema with efficient indexed queries

Backward Compatibility

✅ 100% Backward Compatible

• message_ids parameter is optional for /message endpoint
• Existing requests without Message-IDs work exactly as before
• Response format extended but doesn't break existing parsers
• No breaking changes to any endpoint

[willpower232]

sounds exciting, thanks for your efforts!

[max-kuklin]

Hi @adamcooke, do you think this has a chance to be included in the next release? If I can help you in any way please let me know 🙏

[quaaantumdev]

EDIT: i made some incorrect suggestions for code changes here, mixed up some commits. View history for details.

[max-kuklin]

@quaaantumdev it sounds like you are referencing a different idempotency design pattern (likely Stripe-style idempotency keys with database locking) rather than Message-ID based deduplication approach. The fixes you suggest would require implementing an entirely new architecture that doesn't currently exist in this PR (including the files you propose to fix).

[quaaantumdev]

@max-kuklin sorry max, i was just in desprite search for idempotency with postal and must have mixed up this pull request with some other commit, solving the same issue in a different way. I actually prefer your solution, would be great to see this RP merged for a future release.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[max-kuklin]

Hi @adamcooke just following up, do you think we could merge this? I would really like to avoid maintaining a fork and rather help upstream project 🙏