files_upload_v2 should respect the retry handlers

[tjstum]

The WebClient (and async version of the same) have the convenience method files_upload_v2 that does the three step process for uploading a file using the "new" style: call getUploadURLExternal, POST the file to that URL, then call completeUploadExternal. For steps (1) and (3), the call goes through api_call so will respect the retry handlers that are available on the client itself. However, step (2) is not covered by the retry handlers, so transient HTTP errors can escape up to the caller, who could only retry the entire operation (and would need to implement retry support from scratch, not using the retry system available in the SDK).

This started coming up for us recently. We are seeing 2-6 HTTP 504 errors per day in the _upload_file internal call. Here's an example from our logs, showing the timing (all times EDT). It looks like about 30 seconds from the upload starting to the exception being raised from the SDK:

20260406 16:34:46.349 INFO     txn 85 </rpc.slackmessenger.SlackMessenger/SendSlackThread: 75978> processing request from ('<ip>', <port>)
20260406 16:34:46.350 INFO     pylib3.slackmessenger.basic 466 </rpc.slackmessenger.SlackMessenger/SendSlackThread: 75978> SendSlackThread()
20260406 16:34:46.350 INFO     root 281 </rpc.slackmessenger.SlackMessenger/SendSlackThread: 75978> Channel '@<user> resolved to User IDs ['U<uid>'].
20260406 16:34:46.351 INFO     pylib3.slackmessenger.basic 112 </rpc.slackmessenger.SlackMessenger/SendSlackThread: 75978> Uploading a file with title=Test result summary, filetype=text, filename=Test_result_summary.txt, type=text
20260406 16:35:17.740 ERROR    pylib3.rpc.servers.exceptions_and_metadata 274 </rpc.slackmessenger.SlackMessenger/SendSlackThread: 75978> The following exception was propagated to the client:
Traceback (most recent call last):
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/rpc/servers/exceptions_and_metadata.py", line 248, in handle_exceptions_and_metadata_asyncio
    yield
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/rpc/servers/service_base.py", line 347, in method_wrapper_asyncio
    yield
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/rpc/servers/service_base.py", line 355, in wrapper
    await method(stream)
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/rpc/servers/trio_decorators.py", line 122, in aio_rpc_handler
    stream, await trio_as_aio(rpc_func)(self, request, metadata)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio_asyncio/_handles.py", line 148, in _run
    res = await self._callback(*self._args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/slackmessenger/basic.py", line 472, in SendSlackThread
    return await self._send_slack_thread(message_state, request.messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/slackmessenger/basic.py", line 452, in _send_slack_thread
    await message_state.direct_send(first_message, self.slack_client)
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/slackmessenger/basic.py", line 138, in direct_send
    permalink = await self._upload_file(slack_client, message.new_file)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio_asyncio/_adapter.py", line 54, in __call__
    return await self.loop.run_aio_coroutine(f)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio_asyncio/_base.py", line 225, in run_aio_coroutine
    return await run_aio_future(fut)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio_asyncio/_util.py", line 43, in run_aio_future
    res = await trio.lowlevel.wait_task_rescheduled(abort_cb)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio/_core/_traps.py", line 178, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/outcome/_impl.py", line 213, in unwrap
    raise captured_error
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/trio_asyncio/_adapter.py", line 17, in _call_defer
    return await proc(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/macaroon_binaries/obin/obin-bamboo/20260405233709/pylib3/slackmessenger/basic.py", line 119, in _upload_file
    uploaded = await slack_client.files_upload_v2(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/venvs/opsvenv2/lib/python3.11/site-packages/slack_sdk/web/async_client.py", line 3844, in files_upload_v2
    raise e.SlackRequestError(message)
slack_sdk.errors.SlackRequestError: Failed to upload a file (status: 504, body: None, filename: Test_result_summary.txt, title: Test result summary)

Can you consider an enhancement to _upload_file to have it use the same retry system? Do you have other advice for handling or avoiding this situation? I will also open a support request to understand why this has started happening more often.

Category

• slack_sdk.web.WebClient (sync/async) (Web API client)

[WilliamBergamin]

Hi @tjstum thanks for bringing this up 💯

It does seems like there is a bug here where

• the sync implementation does not support the retry handlers on this step of files_upload_v2, this is something we can add in a minor release
• async does support the retry handlers but the AsyncServerErrorRetryHandler does not actually cause a retry on 504 status reponses

For the async implementation you should be able to use your own retry handler for status 504

  class GatewayErrorRetryHandler(RetryHandler):                                                                                                                                                                                
      def _can_retry(self, *, state, request, response=None, error=None):                                                                                                                                                      
          return response is not None and response.status_code == 504                                                                                                                                                          
                                                                                                                                                                                                                               
  client = AsyncWebClient(                                                                                                                                                                                                     
      token="xoxb-...",                                                                                                                                                                                                        
      retry_handlers=[GatewayErrorRetryHandler(max_retry_count=3)],                                                                                                                                                            
  )   

[tjstum]

Hey @WilliamBergamin. Thanks for the quick response.

I do see what you're saying that the AsyncWebClient does use the retry. However, we are already using a custom retry handler of our own! This is to account for other transient errors (#1715):

class AsyncSlackmessengerRetryHandler(AsyncRetryHandler):
    """
    Custom catch-all retry handler for observed transient SlackApiError's not covered by any stock
    slack_sdk handler.
    """

    async def _can_retry_async(
        self,
        *,
        state: object,
        request: object,
        response: HttpResponse | None = None,
        error: object = None,
    ) -> bool:
        if response is None:
            return False

        match response.status_code, response.body:
            case 200, {"ok": False, "error": "fatal_error"}:
                return True
            case 200, {"ok": False, "error": "delete_failed"}:
                return True
            case 504, {}:
                return True

        return False

Re-reading this now, I guess our handler could be failing to match on the 504 case if response.body is None (which it is allowed to be). I'll fix that and also increase the maximum number of attempts and see if it helps.

For full transparency, here's what support said about the underlying issue:

It sounds like this might be a CloudFront issue. Can you try adding a small pause between upload requests a 1-second gap between files to help avoid triggering the CDN rate limiter. Can you check if the upload actually went through (e.g., by calling files.completeUploadExternal and seeing if it succeeds) before retrying. Can you also try increasing the timeout to 60 from the default 30 seconds.