[Performance] High database read volume on `storedWorkflow->logs()` during workflow rehydration

[geangontijo]

Description
While building a Saga pattern implementation using Durable Workflow, I noticed an excessively high number of database reads targeting the workflow logs. Although the queue messages and database writes are behaving as expected for an event-sourced architecture, the read volume grows exponentially as the workflow progresses, creating a potential bottleneck under high concurrency.

In a single execution of a relatively simple travel package workflow (3 sequential activities + 2 compensation activities triggered by an exception), the system executed 44 database reads for storedWorkflow->logs().

The Problem
Because Durable Workflow uses Event Sourcing to rebuild the state of the workflow every time it wakes up, it re-reads the event stream. However, it seems to be querying the database repeatedly instead of fetching the logs once per worker execution cycle.

Here is a summarized timeline of the event frequency for a single failed workflow execution (Saga pattern):

• 44x - read storedWorkflow->logs() 🚩
• 10x - Queue messages (activities, compensations, and exceptions)
• 10x - write storedWorkflow->status ("pending", "running", "waiting", "failed")

At a low volume, this is unnoticeable. But if 1,000 users trigger this workflow concurrently, it results in 44,000 database reads just to rehydrate states, which will quickly exhaust the database connection pool and spike CPU usage.

Steps to Reproduce

1. Create a workflow that yields multiple sequential activities (e.g., HotelReservation, FlightReservation, TourGuideReservation).
2. Throw an exception in the last activity to trigger a try/catch block that queues compensation activities.
3. Monitor the database queries. You will see SELECT * FROM workflow_logs WHERE ... being executed dozens of times for a single workflow lifecycle.

Environment:

• PHP Version: v8.4
• Laravel Version: v11.48.0
• Durable Workflow Version: v1.0.61

[rmcdaniel]

Can you confirm if this is above and beyond the expected volume for a replay-based workflow engine? The fact that performance decreases over time is a known tradeoff and after a certain point you will need to use the contine-as-new pattern to reduce the workflow history. For example if you look at the Temporal docs https://docs.temporal.io/workflow-execution/continue-as-new they say the following.

A Workflow Execution with a long, or large Event History, such as one calling many Activities, may bog down and have performance issues. It could even generate more Events than allowed by the Event History limits.

Do you think that you could gather details on what the expected performance baseline should be? I would love to improve performance if possible but there are also inherent limitations to replay-based workflow engines.

[rmcdaniel]

I think I have identified a way to reduce the number of queries and increase performance but the inherent growth of workflow history is still a seperate issue. We could add some helpers to the StoredWorkflow model and used the currently loaded logs relation if one exists. This would avoid the queries caused by Activity/Timer/Side Effect/etc and other features.