Are cats.mtl.Handle[F[_], E] and cats.ApplicativeError[F[_], E] the same abstraction?

[benhutchison]

Are cats.mtl.Handle[F[_], E] and cats.ApplicativeError[F[_], E] the same abstraction?

I can't see a meaningful difference between them. Handle includes code to derive itself from ApplicativeError. The reverse derivation also seems possible. So they appear isomorphic.

Which suggests the machinery for submarine error propagation could equally be supported for ApplicativeError, potentially reaching the wider audience of Cats users.

It also seems confusing, to me at least, to have two isomorphic type-classes in the same ecosystem.

There is also cats.MonadError, a subclass of ApplicativeError which is often more useful in practice, because it permits sequencing operations with flatMap - a common requirement of application code.

MonadError[F, E] suffices to express fallible-but-othewise-pure computations and neatly abstracts across concrete effects Either, and the more powerful cats.effect.IO (with submarines to permit non-Throwable error types). So the ability to derive Handle+Monad=>MonadError seems valuable.

given [F[_]: Monad as m, E](using h: Handle[F, E]): MonadError[F, E] = new MonadError:
  def pure[A](x: A): F[A] = h.applicative.pure(x)
  def handleErrorWith[A](fa: F[A])(f: E => F[A]): F[A] = h.handleWith(fa)(f)
  def raiseError[A](e: E): F[A] = h.raise(e)
  def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B] = m.flatMap(fa)(f)
  def tailRecM[A, B](a: A)(f: A => F[Either[A, B]]): F[B] = m.tailRecM(a)(f)

[djspiewak]

This is a superb question and it's something that was heavily debated when Cats MTL was forked off from Cats itself. Answering this (and your other related question that I've lost the link to) requires first taking a step back…

Cats MTL uses a very different typeclass encoding from Cats and Cats Effect. In particular, Cats and Cats Effect chose a subtype implication encoding, while Cats MTL uses a non-auto-deriving compositional implication encoding. That that means is the following sorts of patterns in Cats:

trait Functor[F[_]]:
  // ...

trait Applicative[F[_]] extends Functor[F]:
  // ...

As opposed to the following in Cats MTL:

trait Functor[F[_]]:
  // ...

trait Applicative[F[_]]:
  def functor: Functor[F]
  // ...

The major downside to the latter encoding is that having an Applicative[F] doesn't automatically give you a Functor[F], so such encodings are often paired with implicit materializers which go "up" the hierarchy to auto-derive the implication:

implicit def applicativeToFunctor[F[_]: Applicative]: Functor[F] = Applicative[F].functor

The cracks in this encoding start to show when you think about where this materialization needs to exist: it needs to be on Functor or in some imported implicit scope (so in practice, on Functor). Keep pulling the string on this and you get the Scato encoding (famously used in Scalaz 8).

The advantages to compositional typeclass implication are that you can create implied diamonds without generating ambiguities. In Cats, the only major diamond we need to worry about is between Traverse and Applicative (a problem we very carefully sidestep in practice by prioritizing the syntax resolution). This is because the functor hierarchy is fairly linear in nature. Now, there are other hierarchies which are less linear in nature, but we tend to not reach for them as ubiquitously.

Cats MTL has the opposite problem: there is no meaningful linearization between its capabilities at all! Obviously, Handle <: Raise, but what should the relationship be between Handle and Ask? Clearly none at all, but that brings up an immediate problem: if Handle and Ask were to both extend Applicative, then how could you have them both in scope at once? We would be recreating Traverse/Applicative several times over, and in a way that we can't dodge with creative syntax definitions.

Scalaz 7 had (and has) a variant of these same MTL classes directly ported from Haskell's MTL library, and unlike Cats MTL they do extend the functor hierarchy directly, and they are practically unusable as a result since you get implicit ambiguity everywhere. This was actually one of the early motivating cases for the Scato encoding in the first place!

Unfortunately, the Scato encoding isn't really a panacea for several reasons, but the most notable is that it isn't really an open hierarchy. In order to maintain auto-derivation, you need to explicitly encode a full inverted semi-lattice over your typeclass hierarchy via chained implicits. This in turn means that the whole hierarchy, from top to bottom, must be entirely known in one place. For Scalaz 8 you can see that place if you look in the package object, which contains (among other things) a very long list of implicits which derive up the hierarchy. The construction of these implicits guarantees that theoretical ambiguities always have a single possible resolution, but its existence also guarantees that it's impossible for third parties to attach new typeclasses downstream.

As aside, some of these inevitable complexities are part of why Cats MTL uses a baby version of the Scato encoding, rather than the full thing, which in turn is why you cannot implicitly get an Applicative from a Handle.

This was a known problem when Cats was built. We actually considered using Scato because it really does solve a lot of the subtyping-related problems, and it would have allowed Cats MTL to remain a part of the core Cats library. However, we already knew at the time that something like Cats Effect was coming eventually (Fs2 already had a variant of the early CE typeclasses by then), and we weren't willing to preclude third-party extension of the hierarchy in general. So we chose to keep the hierarchy open.

Unfortunately, Cats Effect inherits the worst version of all of these problems! The Cats Effect hierarchy has multiple diamonds, one of which is quite commonly hit in practice (Temporal / Sync), and it is definitely a third-party extension of the Cats hierarchy. We also considered using Scato for Cats Effect – and I mean really really considered! – but in the end, uniformity with Cats won out. Also, by simplifying some things relative to the earlier CE proposals, we were able to condense the hierarchy into a roughly linear shape, which is the form it takes today.

One of the tradeoffs for condensing Cats Effect into a roughly linear form is that we needed an extension point in Cats which made sense. Monad is sufficient, but if we only had Monad then we would have basically reinvented MonadError downstream, which would certainly be a waste. However, the existence of Handle (which, as previously argued, fundamentally must encode its typeclass in a different way from ApplicativeError and co) means that MonadError is basically redundant, so the whole thing became a huge mess.

At one point, we were really considering removing MonadError and ApplicativeError from Cats, and the anticipated future needs of Cats Effect were what stopped us.

So in summary… we've basically argued that the requirements we set for ourselves for Cats, Cats Effect, and Cats MTL have all jointly conspired to give us two completely isomorphic (but not equivalent!) and differently named encodings of ApplicativeError. Which sucks and is confusing, though it at least has the advantage of avoiding ambiguity for users in essentially all practical cases while still keeping Cats' binary compatibility and versioning independent of Cats Effect (something which saved us an unbelievable amount of headache during the CE3 process).

Coming back to your more tactical asks though, implicitly materializing an ApplicativeError from a Handle is possible, but not without an import (because we can't put that implicit on the ApplicativeError companion), so I'm skeptical as to its utility. As for reimplementing the submarine error handling in Cats for ApplicativeError (and probably also MonadError), that's doable but it runs immediately into the diamond problem! This trick only works because we're introducing a brand new instance, but that in turn only works because an instance of Handle[F] doesn't imply a redundant instance of Applicative[F]. So I think we probably can't do it.

[benhutchison]

Thanks for your detailed, informative response (and history lesson!) @djspiewak :)

I guess my TLDR is that:

1. morally Handle & ApplicativeError are the same abstraction
2. Implementation-level constraints around typeclasses in Scala 2 (specifically, avoiding an ambiguity condition triggered by passing two typeclasses that share a common ancestor trait, the "point of a diamond") resulted in Handle being a version of ApplicativeError with the Applicative supertype removed.

I don't think this is changing in Cats any time soon, given the constraints on such a foundational library.

But for any dreamers, wonderers, or type-astronauts who have escaped the gravity well of Scala 2, I will suggest that times are a-changin' in Scala 3. The recut of the implicit prioritisation rules that landed in Scala 3.7 seems to eliminate some implicit-ambiguities and opens up new design possibilities that are not feasible in Scala 2. If anyone reading this missed it, the heart of the change is:

"Under the old rules, the compiler always tried to select the instance with the most specific subtype of the requested type. Under the new rules, it always selects the instance with the most general subtype that satisfies the context bound."

Here's an example where we can use two instances extending MonadError (thus also extending Monad, Applicative, Functor) in the same scope, and make use of operations defined at multiple layers of the hierarchy. The "price" is the need to pass Monad as an extra constraint, which stimulates the compiler to resolve a common Monad parent for both typeclasses.

//> using scala 3.7.3
//> using dep org.typelevel::cats-effect:3.5.4

import cats.*
import cats.syntax.all.*
import cats.effect.*

type ErrS[F[_]] = MonadError[F, String]

def scala3_WontCompile[F[_]: Sync: ErrS]: F[Unit] = List("error").traverse(_.pure).map(_.mkString).flatMap(_.raiseError)

def scala3_OK[F[_]: Monad: Sync: ErrS]: F[Unit] = List("error").traverse(_.pure).map(_.mkString).flatMap(_.raiseError)

How practical and reliable this technique proves when used "in anger, at scale and/or in the wild" remains an open question, for me at least... 🤔 💭

[djspiewak]

How practical and reliable this technique proves when used "in anger, at scale and/or in the wild" remains an open question, for me at least... 🤔 💭

IMO, it's time well-spent to close that open question! https://github.com/typelevel/spotted-leopards exists for some of this experimentation, though it's been dormant for a while. In general, we're expecting Cats 3 to be Scala 3 exclusive and to fully leverage its language features. The extent to which this unlocks novel and beneficial encodings of practical typeclass hierarchies really remains to be understood, and that would be great knowledge to obtain.

Put in a slightly more grandiose fashion… Typeclass encodings in Scala have really stagnated for the past decade, as we've really only had two viable options this whole time: sub-typing and scato. As illustrated in my earlier comment, neither of these are great options, but no one has been able to come up with anything unequivocally better in all this time (historical sidebar: there are a few encodings which predate these two – most notably the one used in Scalaz 6 and previous, which was most similar to the present sub-type encoding in Cats – but they had even larger tradeoffs). Scala 3 should enable new encodings here, if only because of the major differences in how implicits are handled, but that space remains a frontier.

[benhutchison]

In general, we're expecting Cats 3 to be Scala 3 exclusive and to fully leverage its language features.

Er.. "Cats 3"?.. yay! Is the development of Cats 3 actively happening?

I feel like an undertaking of that scale would need someone(s) willing to dedicate to leading it. Ie it wouldn't happen just by a patchwork of small community contributions.

If there is a Cats 3, there's a few small breaking fixes in the algebra hierarchy I'd love to see addressed.
typelevel/cats#4664
typelevel/cats#4570