CIVIMM-463: Add instalment charge scheduled job

[erawat]

Overview

Implements Job 2 in the recurring payment processing system - a scheduled job that automatically charges due instalment contributions. The core extension handles contribution selection and PaymentAttempt management, while processor-specific charging is delegated to payment processor extensions (Stripe, GoCardless) via Symfony events.

Before

No automated mechanism to charge due instalments from recurring contributions. Each payment processor extension would need to implement its own selection logic, leading to code duplication and inconsistent behavior.

After

A unified scheduled job that:

• Selects eligible contributions using consistent criteria across all processors
• Creates/manages PaymentAttempt records with race-condition-safe status transitions
• Dispatches batch events for processor extensions to handle the actual API calls

Technical Details

Job Configuration

• Job name: PaymentInstalmentCharge
• Job label: "Charge due instalments for recurring contributions"
• Run frequency: Always
• Default parameters: processor_type=Stripe, batch_size=500, max_retry_count=3

Selection Query Criteria

The service selects contributions matching ALL of these:

• contribution_status_id IN (Pending, Partially Paid)
• (total_amount - paid_amount) > 0 (outstanding balance exists)
• receive_date <= NOW() (due date has passed)
• contribution_recur_id IS NOT NULL (linked to recurring)
• Parent recurring: status != Cancelled, payment_token_id IS NOT NULL
• contribution_recur.failure_count <= max_retry_count (NULL treated as 0)
• No existing PaymentAttempt with status IN ('processing', 'completed', 'cancelled')
• Filtered by payment_processor_type.name for processor-specific batches

Performance Optimizations

1. Sequential processor processing - Each processor type processed separately to avoid OOM
2. Batch PaymentAttempt lookup - Single query fetches all existing attempts (prevents N+1)
3. Single batch event dispatch - One event per processor type instead of per-contribution
4. Oldest-first ordering - ORDER BY receive_date ASC ensures fair processing
5. Explicit API4 joins - Multi-level FK relationships use explicit addJoin() for reliability

Race Condition Prevention

// Atomic status transition using WHERE clause
public static function updateStatusAtomic(int $id, string $expectedStatus, string $newStatus): bool {
  $result = \Civi\Api4\PaymentAttempt::update(FALSE)
    ->addWhere('id', '=', $id)
    ->addWhere('status', '=', $expectedStatus)  // Only update if still pending
    ->setValues(['status' => $newStatus])
    ->execute();
  return $result->count() > 0;
}

Symfony Event Dispatch

Payment processor extensions subscribe to ChargeInstalmentBatchEvent:

// In Stripe extension's event subscriber:
public function onChargeInstalmentBatch(ChargeInstalmentBatchEvent $event): void {
  if ($event->getProcessorType() !== 'Stripe') {
    return;  // Ignore events for other processors
  }
  
  $maxRetryCount = $event->getMaxRetryCount();  // For failure handling
  
  foreach ($event->getItems() as $item) {
    // $item contains: contributionId, paymentAttemptId, recurringContributionId,
    // contactId, amount, currency, paymentTokenId, paymentProcessorId
    
    // Call Stripe API to charge...
    // Update PaymentAttempt with processor_payment_id on success
    // Use maxRetryCount to decide when to mark contribution as Failed
  }
}

API Parameters

// Required
'processor_type' => 'Stripe'           // Single processor
'processor_type' => 'Stripe,GoCardless' // Comma-separated
'batch_size' => 500                     // Max per processor type

// Optional
'max_retry_count' => 3                  // Default: 3

New PaymentAttempt Status

Added 'processing' status for in-flight charges:

• pending → processing (atomic transition when job claims it)
• processing → completed (webhook confirms success)
• processing → failed (webhook confirms failure)

Entity Framework Migration

Upgraded from v1 to v2 via civix:

• Schema definitions moved from xml/schema/*.xml to schema/*.entityType.php
• Removed sql/auto_install.sql and sql/auto_uninstall.sql
• Added civimix-schema mixin

Civix Regeneration

After schema changes, DAO files were regenerated using:

civix generate:entity-boilerplate

This regenerates:

• CRM/Paymentprocessingcore/DAO/*.php - Data Access Objects
• paymentprocessingcore.civix.php - Extension utilities

Note: These files are auto-generated and should not be manually edited.

Core overrides

None

Comments

• Processor extensions (Stripe, GoCardless) will implement ChargeInstalmentBatchEvent subscribers in separate PRs
• Job 3 (Reconcile stuck 'processing' attempts) will be implemented separately
• Docker test environment updated to PHP 8.1 for readonly property support

[gemini-code-assist]

Summary of Changes

Hello @erawat, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust and scalable automated system for charging recurring instalment contributions. It establishes a core service responsible for identifying eligible payments and managing their lifecycle through a new 'processing' status, while leveraging an event-driven architecture to allow payment processor extensions to handle the actual payment execution. This design significantly reduces code duplication, improves consistency across payment processors, and lays the groundwork for future enhancements in recurring payment management.

Highlights

• Automated Instalment Charging Job: Implemented a new scheduled job (InstalmentCharge.Run) that automatically identifies and initiates charges for due instalment contributions, forming Job 2 in the recurring payment processing system.
• Event-Driven Charging Architecture: Introduced a new InstalmentChargeService that selects eligible contributions, manages PaymentAttempt records, and dispatches ChargeInstalmentBatchEvent events. This delegates the actual processor-specific API calls to payment processor extensions (e.g., Stripe, GoCardless) via Symfony events, promoting modularity and reusability.
• Enhanced PaymentAttempt Status Management: Added a new 'processing' status to PaymentAttempt records and a new updateStatusAtomic method to prevent race conditions when multiple workers attempt to process the same payment simultaneously.
• Entity Framework Migration: Migrated schema definitions from XML files (xml/schema/*.xml) to PHP-based entity type files (schema/*.entityType.php) and updated the info.xml to use the civimix-schema mixin for modern schema management.
• Performance Optimizations and Robustness: Incorporated performance optimizations such as sequential processing per processor type, batch fetching of PaymentAttempt records, and single batch event dispatch. Selection criteria include checks for outstanding balance, due dates, recurring status, and retry counts.

Changelog

• CRM/Paymentprocessingcore/BAO/PaymentAttempt.php
  • Added 'processing' status to the list of available payment attempt statuses.
  • Updated updateStatus method's PHPDoc to include the new 'processing' status.
  • Introduced updateStatusAtomic method for optimistic locking during status transitions to prevent race conditions.
• Civi/Paymentprocessingcore/DTO/ChargeInstalmentItem.php
  • Added new Data Transfer Object (DTO) ChargeInstalmentItem to encapsulate data for a single instalment charge.
• Civi/Paymentprocessingcore/Event/ChargeInstalmentBatchEvent.php
  • Added new Symfony event ChargeInstalmentBatchEvent to dispatch batches of instalment items to payment processor extensions.
• Civi/Paymentprocessingcore/Hook/Container/ServiceContainer.php
  • Registered the new InstalmentChargeService in the service container for dependency injection.
• Civi/Paymentprocessingcore/Service/InstalmentChargeService.php
  • Added new service InstalmentChargeService to manage the core logic for selecting, preparing, and dispatching instalment charges.
  • Implemented chargeInstalments method to process instalments for specified processor types in batches.
  • Implemented getEligibleContributions method with detailed SQL query for selecting contributions based on various criteria.
  • Implemented batchFetchPaymentAttempts for efficient retrieval of existing payment attempts.
  • Implemented getOrCreatePaymentAttempt to handle creation or reuse of PaymentAttempt records with atomic status transitions.
• api/v3/InstalmentCharge/Run.php
  • Added new APIv3 endpoint InstalmentCharge.Run to trigger the instalment charging process.
  • Included logic to parse processor_type, batch_size, and max_retry_count parameters.
• docker-compose.phpstan.yml
  • Updated the Docker image for PHPStan to composer:2.
  • Modified the command to install composer dependencies and run PHPStan from vendor/bin.
• docker-compose.test.yml
  • Updated the CiviCRM Docker image to compucorp/php-fpm.civicrm:8.1 to support PHP 8.1.
• info.xml
  • Updated civix format version to 25.10.2.
  • Updated mgd-php mixin to version 2.0.0.
  • Updated entity-types-php mixin to version 2.0.0.
  • Added smarty mixin version 1.0.3 and scan-classes mixin version 1.0.0.
  • Changed the upgrader class to CiviMix\Schema\Paymentprocessingcore\AutomaticUpgrader for PHP-based schema management.
• managed/Job_InstalmentCharge.mgd.php
  • Added a new managed entity definition for the 'Charge due instalments for recurring contributions' scheduled job.
• mixin/lib/civimix-schema@5.93.beta1/pathload.main.php
  • Added new pathload.main.php for civimix-schema library, enabling reloadable schema management.
• mixin/lib/civimix-schema@5.93.beta1/src/AutomaticUpgrader.php
  • Added AutomaticUpgrader class for automated schema creation and destruction using PHP entity definitions.
• mixin/lib/civimix-schema@5.93.beta1/src/CiviMixSchema.php
  • Added CiviMixSchema class to manage loading and retrieval of schema helpers.
• mixin/lib/civimix-schema@5.93.beta1/src/DAO.php
  • Added DAO class for dynamic DAO generation from PHP entity definitions.
• mixin/lib/civimix-schema@5.93.beta1/src/SchemaHelper.php
  • Added SchemaHelper class providing utility methods for schema management (install, uninstall, alter fields) based on PHP entity definitions.
• mixin/lib/civimix-schema@5.93.beta1/src/SchemaHelperInterface.php
  • Added SchemaHelperInterface defining methods for schema management utilities.
• mixin/lib/civimix-schema@5.93.beta1/src/SqlGenerator.php
  • Added SqlGenerator class to generate SQL statements from PHP entity definitions.
• mixin/mgd-php@2.0.0.mixin.php
  • Added new mgd-php mixin version 2.0.0 for auto-registering **.mgd.php files with a narrower search scope.
• schema/PaymentAttempt.entityType.php
  • Added PHP-based entity type definition for PaymentAttempt, replacing the old XML schema.
• schema/PaymentProcessorCustomer.entityType.php
  • Added PHP-based entity type definition for PaymentProcessorCustomer, replacing the old XML schema.
• schema/PaymentWebhook.entityType.php
  • Added PHP-based entity type definition for PaymentWebhook, replacing the old XML schema.
• scripts/setup.sh
  • Updated the script to install the php-bcmath extension using the detected PHP version.
• sql/auto_install.sql
  • Removed the auto-generated SQL installation script, replaced by PHP-based schema management.
• sql/auto_uninstall.sql
  • Removed the auto-generated SQL uninstallation script, replaced by PHP-based schema management.
• tests/phpunit/CRM/Paymentprocessingcore/BAO/PaymentAttemptTest.php
  • Updated testGetStatuses to assert the presence of the new 'processing' status.
  • Added testValidateStatusAcceptsProcessing to verify the new status is valid.
  • Added testValidateStatusRejectsInvalid to ensure invalid statuses are rejected.
  • Added testUpdateStatusAtomicSucceedsOnMatch to test successful atomic status updates.
  • Added testUpdateStatusAtomicFailsOnMismatch to test atomic updates failing on status mismatch.
• tests/phpunit/Civi/Paymentprocessingcore/DTO/ChargeInstalmentItemTest.php
  • Added new unit tests for the ChargeInstalmentItem DTO, verifying constructor and property accessibility.
• tests/phpunit/Civi/Paymentprocessingcore/Event/ChargeInstalmentBatchEventTest.php
  • Added new unit tests for the ChargeInstalmentBatchEvent, verifying constructor, getters, and event name constant.
• tests/phpunit/Civi/Paymentprocessingcore/Service/InstalmentChargeServiceTest.php
  • Added comprehensive unit tests for InstalmentChargeService, covering contribution selection, payment attempt handling, batch event dispatch, and various edge cases.
• xml/schema/CRM/Paymentprocessingcore/PaymentAttempt.xml
  • Removed the XML schema definition for PaymentAttempt, replaced by schema/PaymentAttempt.entityType.php.
• xml/schema/CRM/Paymentprocessingcore/PaymentProcessorCustomer.xml
  • Removed the XML schema definition for PaymentProcessorCustomer, replaced by schema/PaymentProcessorCustomer.entityType.php.
• xml/schema/CRM/Paymentprocessingcore/PaymentWebhook.xml
  • Removed the XML schema definition for PaymentWebhook, replaced by schema/PaymentWebhook.entityType.php.

Ignored Files

• Ignored by pattern: CRM/Paymentprocessingcore/DAO/*.php (3)
  • CRM/Paymentprocessingcore/DAO/PaymentAttempt.php
  • CRM/Paymentprocessingcore/DAO/PaymentProcessorCustomer.php
  • CRM/Paymentprocessingcore/DAO/PaymentWebhook.php
• Ignored by pattern: paymentprocessingcore.civix.php (1)
  • paymentprocessingcore.civix.php
• Ignored by pattern: xml/schema/*.entityType.php (3)
  • xml/schema/CRM/Paymentprocessingcore/PaymentAttempt.entityType.php
  • xml/schema/CRM/Paymentprocessingcore/PaymentProcessorCustomer.entityType.php
  • xml/schema/CRM/Paymentprocessingcore/PaymentWebhook.entityType.php

Activity

• The pull request implements Job 2 of the recurring payment processing system, focusing on charging due instalment contributions.
• It establishes a new InstalmentChargeService and an API endpoint to trigger the charging process.
• A new processing status has been introduced for PaymentAttempt records, along with atomic update logic to handle concurrent operations.
• The project's entity schema definitions have been migrated from XML to PHP files, leveraging the civimix-schema library for modern schema management.
• Docker test environments have been updated to PHP 8.1 to support new language features like readonly properties.
• The PR description notes that processor-specific implementations for charging (e.g., Stripe, GoCardless) will be handled in separate pull requests.
• It also indicates that Job 3, for reconciling stuck 'processing' attempts, will be implemented separately.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new scheduled job for charging due instalments, a core part of the recurring payment processing system. The changes are extensive and well-structured, including a new InstalmentChargeService, a corresponding APIv3 endpoint, and the use of Symfony events to delegate charging to processor-specific extensions. The implementation correctly uses atomic updates to prevent race conditions and includes a comprehensive suite of new unit tests.

My review focuses on a potential issue in the payment retry logic. I've identified that contributions with a failed PaymentAttempt will be permanently skipped, preventing retries. I have provided suggestions across PaymentAttempt.php and InstalmentChargeService.php to address this by allowing failed attempts to be reused for retries.

The migration from XML-based schema definitions to the new civimix-schema with PHP entity types is also a significant and positive change in this PR.