Change entry-points to be toml files

This also changes the auto-generation from decorator version
to create the entry-point files to use:
```
python -m spatch update-entrypoints file [file ...]
```
if you add the special:
```
[functions.auto-generation]
backend = "spatch._spatch_example.backend:backend1"
modules = ["spatch._spatch_example.backend"]
```
section to the entry-point file itself.

I really like this, plus the old abuse of black to format things nicely is fun.

TIL: `tomlkit` is really cool about editing toml files!

Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

Author: seberg

docs/source/api/for_backends.rst

@@ -3,14 +3,19 @@ Backend API
 
 Backends have to do the heavy lifting of using spatch.
 At the moment we suggest to check the
-`example <https://github.com/scientific-python/spatch/tree/main/spatch/_spatch_example>`_.
+`example <https://github.com/scientific-python/spatch/tree/main/src/spatch/_spatch_example>`_.
 
 Entry point definition
 ----------------------
-To extend an existing library with a backend, you need to define a `Python entry-point <https://packaging.python.org/en/latest/specifications/entry-points/>`_.
+To extend an existing library with a backend, you need to define a
+`Python entry-point <https://packaging.python.org/en/latest/specifications/entry-points/>`_.
 This entry point includes the necessary information for spatch to find and
 dispatch to your backend.
 
+``spatch`` entry-points are TOML files and *not* Python objects.
+The entry-point value is ``module:path/to/entrypoint.toml`` rather than
+the typical pattern of ``module.submodule:value``.
+
 Before writing a backend, you need to think about a few things:
 
 * Which types do you accept?  This could be NumPy, dask, jax, etc. arrays.
@@ -25,11 +30,11 @@ Before writing a backend, you need to think about a few things:
   In that case, your backend should likely only be used if prioritized
   by the user.
 
-Please check the example linked above.  These example entry-points include
-code that means running them modifies them in-place if the ``@implements``
-decorator is used (see next section).
+Please check the example linked above.  ``spatch`` can automatically update
+the functions entries in these entry-points if the ``@implements`` decorator
+is used (see next section).
 
-Some of the most important things are:
+Some of the most important fields are:
 
 ``name``
 ^^^^^^^^
@@ -55,13 +60,14 @@ However, we do support the following, e.g.:
 - ``"~numpy:ndarray"`` to match any subclass of NumPy arrays
 - ``"@module:qualname"`` to match any subclass of an abstract base class
 
-If you use an abstract base class, note that you must take a lot of care:
+.. warning::
+  If you use an abstract base class, note that you must take additional care:
 
-- The abstract base class must be cheap to import, because we cannot avoid
-  importing it.
-- Since we can't import all classes, ``spatch`` has no ability to order abstract
-  classes correctly (but we order them last if a primary type, which is typically right).
-- ``spatch`` will not guarantee correct behavior if an ABC is mutated at runtime.
+  - The abstract base class must be *cheap to import*, because we cannot avoid
+    importing it.
+  - Since we can't import all classes, ``spatch`` has no ability to order abstract
+    classes correctly (but we order them last if a primary type, which is typically right).
+  - ``spatch`` will not guarantee correct behavior if an ABC is mutated at runtime.
 
 ``requires_opt_in``
 ^^^^^^^^^^^^^^^^^^^
@@ -93,19 +99,36 @@ functions
 ^^^^^^^^^
 
 A mapping of library functions to your implementations.  All fields use
-the ``__module__:__qualname__`` identifiers to avoid immediate import.
+the ``__module__:__qualname__`` identifiers.
 The following fields are supported for each function:
 
 - ``function``: The implementation to dispatch to.
 - ``should_run`` (optional): A function that gets all inputs (and context)
   and can decide to defer.  Unless you know things will error, try to make sure
   that this function is light-weight.
-- ``uses_context``: Whether the implementation needs a ``DispatchContext``.
+- ``uses_context`` (optional): Whether the implementation needs a ``DispatchContext``.
 - ``additional_docs`` (optional): Brief text to add to the documentation
   of the original function. We suggest keeping this short but including a
   link, but the library guidance should be followed.
 
-``spatch`` provides tooling to help create this mapping.
+A typical part of the entry-point TOML will look like this::
+
+  [functions."skimage.filters:gaussian"]
+  function = "cucim.skimage.filters:gaussian"
+  uses_context = true
+  additional_docs = "CUDA enabled version..."
+
+An additional ``[functions.defaults]`` key can be added to set defaults for all
+functions and avoid repeating e.g. ``uses_context``.
+
+``spatch`` provides tooling to help create this mapping.  This tooling uses the
+additional fields::
+
+  [functions.auto-generation]
+  # where to find the BackendImplementation:
+  backend = "module.submodule:backend_name"
+  # Additional modules to be imported (to ensure all functions are found):
+  modules = ["spatch._spatch_example.backend"]
 
 Manual backend prioritization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

pyproject.toml

@@ -66,8 +66,12 @@ content-type = "text/markdown"
 path = "README.md"
 
 [project.entry-points._spatch_example_backends]
-backend1 = 'spatch._spatch_example.entry_point'
-backend2 = 'spatch._spatch_example.entry_point2'
+backend1 = 'spatch:_spatch_example/entry_point.toml'
+backend2 = 'spatch:_spatch_example/entry_point2.toml'
+
+[[tool.spatch.update_functions]]
+backend1 = "spatch._spatch_example.backend"
+backend2 = "spatch._spatch_example.backend"
 
 [tool.pytest.ini_options]
 minversion = "6.0"

src/spatch/__main__.py

@@ -0,0 +1,26 @@
+import argparse
+
+from .backend_utils import update_entrypoint
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="python -m spatch")
+    subparsers = parser.add_subparsers(
+        help='subcommand help', required=True, dest="subcommand")
+
+    update_entrypoint_cmd = subparsers.add_parser(
+        'update-entrypoints',
+        help='update the entrypoint toml file')
+    update_entrypoint_cmd.add_argument(
+        "paths", type=str, nargs="+",
+        help="paths to the entrypoint toml files to update")
+
+    args = parser.parse_args()
+
+    assert args.subcommand == "update-entrypoints"
+    for path in args.paths:
+        update_entrypoint(path)
+
+
+if __name__ == "__main__":
+    main()

src/spatch/_spatch_example/README.md

@@ -14,8 +14,9 @@ The "library" contains only:
   designed for only `int` inputs.
 
 We then have two backends with their corresponding definitions in `backend.py`.
-The entry-points are `entry_point.py` and `entry_point2.py` and these files
-can be run to generate their `functions` context (i.e. if you add more functions).
+The entry-points are `entry_point.toml` and `entry_point2.toml`.  When code changes,
+these can be updated via ``python -m spin update-entrypoints *.toml``
+(the necessary info is in the file itself).
 
 For users we have the following basic capabilities. Starting with normal
 type dispatching.

src/spatch/_spatch_example/entry_point.py

@@ -0,0 +1,21 @@
+name = "backend1"
+primary_types = ["builtins:float", "builtins:int"]
+secondary_types = []
+requires_opt_in = false
+
+# higher_priority_than = ["default"]
+
+[functions.auto-generation]
+# Backend object that to query
+backend = "spatch._spatch_example.backend:backend1"
+# Modules to load (including submodules) to ensure all functions are initialized
+modules = ["spatch._spatch_example.backend"]
+
+[functions.defaults]
+# Options here will be defaults for all functions.
+uses_context = true
+
+[functions."spatch._spatch_example.library:divide"]
+function = "spatch._spatch_example.backend:divide"
+should_run = "spatch._spatch_example.backend:divide._should_run"
+additional_docs = """This implementation works well on floats."""

src/spatch/_spatch_example/entry_point.toml

@@ -0,0 +1,15 @@
+name = "backend2"
+primary_types = ["builtins:float", "builtins:complex"]
+secondary_types = ["builtins:int"]
+requires_opt_in = false
+
+[functions.auto-generation]
+backend = "spatch._spatch_example.backend:backend2"
+# `modules` not really needed (already backend does this) 
+
+[functions."spatch._spatch_example.library:divide"]
+function = "spatch._spatch_example.backend:divide2"
+should_run = "spatch._spatch_example.backend:divide2._should_run"
+additional_docs = """This is a test backend!
+    and it has a multi-line docstring which makes this longer than normal.
+    """

src/spatch/_spatch_example/entry_point2.py

@@ -1,16 +1,18 @@
 import contextvars
 import dataclasses
 import functools
+import importlib.util
 import os
 import sys
 import textwrap
 import warnings
 from collections.abc import Callable
 from dataclasses import dataclass
-from types import MethodType
+from types import MethodType, SimpleNamespace
 from typing import Any
 
 import importlib_metadata
+import tomllib
 
 from spatch import from_identifier, get_identifier
 from spatch.utils import EMPTY_TYPE_IDENTIFIER, TypeIdentifier, valid_backend_name
@@ -588,7 +590,29 @@ def _get_entry_points(group, blocked):
             if ep.name in blocked:
                 continue
             try:
-                namespace = ep.load()
+                mod, _, filename = ep.value.partition(":")
+                if "." in mod:
+                    raise RuntimeError(
+                        f"Entrypoint {ep.name} has a module name with a dot: '{mod}'. "
+                        "It must use a top-level module and include submodules as path."
+                    )
+
+                spec = importlib.util.find_spec(mod)
+                reader = spec.loader.get_resource_reader(spec.name)
+                with reader.open_resource(filename) as f:
+                    backend_info = tomllib.load(f)
+
+                # We allow a `functions.defaults` field, apply them here to all functions
+                # and clean up the special fields (defaults and auto-generation).
+                backend_info["functions"].pop("auto-generation", None)  # no need to propagate
+                defaults = backend_info["functions"].pop("defaults", None)
+                if defaults:
+                    functions = backend_info["functions"]
+                    for fun in functions:
+                        functions[fun] = {**defaults, **functions[fun]}
+
+                # We use a namespace internally right now (convenient for in-code backends/tests)
+                namespace = SimpleNamespace(**backend_info)
                 if ep.name != namespace.name:
                     raise RuntimeError(
                         f"Entrypoint name {ep.name!r} and actual name {namespace.name!r} mismatch."