Using Faraday in_parallel without immediately raising Faraday::TimeoutError

[d11wtq]

We're not currently using Async in our project, though we are looking to switch some code that was previously using Faraday with the :typhoeus adapter to instead use the :async_http adapter.

One feature we were relying on was the ability to run a batch of requests in parallel without raising exceptions if one of the requests hits a timeout. Say 20 requests are run in parallel and one of them times out, we still want the 19 successful responses (and we can retry the one timeout separately).

When I simulate this situation with the :async_http adapter, what I'm seeing is that the Async::TimeoutError is handled, then re-raised as Faraday::TimeoutError, even when running in parallel mode. This means that the whole in_parallel block raises an exception and therefore the other responses cannot be reached.

What typhoeus does instead is it puts two values into the response env:

• env[:typhoeus_connection_failed] (can be understood as Faraday::ConnectionFailed with the details found in env[:typhoeus_return_message])
• env[:typhoeus_timed_out] (can be understood as Faraday::TimeoutError)

Is there any way to keep the same capability of firing off 20 requests and letting some of them timeout or hit connection issues without failing all the others?

[ioquatix]

The simplest way would be to use Async to fan out the requests, but still use Faraday's interface for individual requests. Since your level of complexity is higher than normal users, I'd actually suggest you just use Async::HTTP::Internet with Async rather than trying to shoe-horn this usage into Faraday.

Beyond that, perhaps an official extension to Faraday would make more sense.

cc @iMacTia