troubleshooting_guide.md

Python Troubleshooting Guide

This section provides a list of common issues and solutions for troubleshooting when using the Temporal Python SDK. Always refer first to the sdk-python documentation and the official documentation for the most up-to-date information.

If you landed here by accident, please feel free to reach out through the community channels or open a support ticket if you are a Temporal Cloud customer.

WorkflowTaskTimeout

• What is a Workflow Task Timeout?

Possible Causes and Solutions

The gRPC payload size exceed the limit.

A common scenario is when there is a large number of activities (or child workflows) scheduled concurrently in a loop. This can cause the gRPC payload size to exceed the limit.

Possible solutions

• Split the loop in batches to reduce the payload size.
• Use data converters to compress the payloads or store the data in an external storage service.

The thread inside an async def Python function is blocked

According to the Python SDK Documentation ⚠️ WARNING: Do not block the thread in async def Python functions. This can stop the processing of the rest of the Temporal.

Blocking the thread can significantly delay execution, potentially leading to a WorkflowTaskTimeout.

Possible solutions

• configure the workers to run activities in a different thread https://github.com/temporalio/sdk-python?tab=readme-ov-file#synchronous-multithreaded-activities
• use asyncio.to_thread(my_blocking_func) or asyncio.get_running_loop().run_in_executor(None, my_blocking_func) to run the blocking code inside the async function.
• if you are unsure if an activity makes blocking calls, convert the async activities into synchronous ones (docs).
• see the "Registering Activities" page of the Python 101 course for guidance on sync and async activities.
• see the API docs for the worker initialization to understand the configuration options.

Code

• This example demonstrate how blocking the thread causes the issue.

Memory Leak

If you are seeing a memory leak, then make sure you are passing through your modules into the workflow sandbox (docs).

1. Separate your activity code and your workflow code into different files.
2. In the workflow file, pass through the third party modules and your personal modules (including activity code) that don't have side effects upon import.