Meta: linked data in responses

[Sporiff]

I recently posted in the chatroom about HATEOAS (Hypermedia As The Engine Of Application State) as a potential format for our API responses. I didn't go deeply into it since it was just a passing thought, but the more I look into the discussions we've had previously surrounding issues such as pagination, data linking, and ease of discovery, the more I think it would be a good idea.

In particular, a standard called HAL (Hypertext Application Language) provides an excellent template for API responses which can easily be leveraged to give API users more information about what they can do with data they receive.

I would propose that adopting HAL makes the API design more scalable and much easier to follow from both a client and server implementation perspective (although client developers can shed more light on this from their end). It creates a standard response format for every endpoint, and a much easier method of navigating related data. Here are a couple of examples.

Pagination

As an example, let's take a look at our subscription fetch response:

{
   "total": 1,
   "page": 1,
   "per_page": 5,
   "subscriptions": [
      {
         "feed_url": "https://example.com/rss1",
         "guid": "daac3ce5-7b16-4cf0-8294-86ad71944a64",
         "is_subscribed": true,
         "guid_changed": "2022-12-23T10:24:14.670Z",
         "new_guid": "36a47c4c-4aa3-428a-8132-3712a8422002"
      }
   ]
}

This object contains a pagination object, but no information about how to navigate between pages of responses. It exposes how many pages there are and how many items are in each page, but nothing more. It's down to the developer to do the maths to figure out the pagination limit.

How might we represent this in HAL? The first thing we could do is add contextual links for pagination. Instead of revealing how many pages there are and how many objects are on each page, we can instead expose navigational links. For example:

{
  "_links": {
    "self": {
      "href": "http://opa.org/api/v1/subscriptions?page=3&per_page=5"
    },
    "next": {
      "href": "http://opa.org/api/v1/subscriptions?page=4&per_page=5"
    },
    "prev": {
      "href": "http://opa.org/api/v1/subscriptions?page=2&per_page=5"
    },
    "first": {
      "href": "http://opa.org/api/v1/subscriptions?page=1&per_page=5"
    },
    "last": {
      "href": "http://opa.org/api/v1/subscriptions?page=7&per_page=5"
    }
  }
}

This would tell developers exactly how to maneuver through the paginated list. We can expose the same page metadata underneath as well for clarity:

  "page" : {
    "size" : 5,
    "totalElements" : 34,
    "totalPages" : 7,
    "number" : 0
  }

Embedded objects

Similar to pagination, our current response layout doesn't really tell the API consumer what they can even do with the objects in the subscriptions array. They appear as read-only objects. If the developer reads the specs fully, then they know that they can perform other actions such as updating and deleting. However, there's no way to know that from this response.

By following the HAL standard, we could expose more information about the API and its responses right there in the model itself. For example:

• What is the link to get the resource?
• What actions can the current user take on this resource and where are they linked?
• Are there any related endpoints (e.g. episodes) that would be useful?

To give an example, here's a full example of our current approach for the subscriptions endpoint:

{
   "total": 2,
   "page": 1,
   "per_page": 5,
   "subscriptions": [
      {
         "feed_url": "https://example.com/rss1",
         "guid": "31740ac6-e39d-49cd-9179-634bcecf4143",
         "is_subscribed": true,
         "guid_changed": "2022-09-21T10:25:32.411Z",
         "new_guid": "8d1f8f09-4f50-4327-9a63-639bfb1cbd98"
      },
      {
         "feed_url": "https://example.com/rss2",
         "guid": "968cb508-803c-493c-8ff2-9e397dadb83c",
         "is_subscribed": false,
         "subscription_changed": "2022-04-24T17:53:21.573Z"
      }
   ]
}

In HAL, it might look like this:

{
  "_embedded": {
    "subscriptions": [
      {
        "feed_url": "https://example.com/rss1",
        "guid": "31740ac6-e39d-49cd-9179-634bcecf4143",
        "is_subscribed": true,
        "guid_changed": "2022-09-21T10:25:32.411Z",
        "_links": {
          "new_location": {
            "href": "http://opa.org/api/v1/subscriptions/8d1f8f09-4f50-4327-9a63-639bfb1cbd98"
          },
          "self": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          },
          "delete": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          },
          "update": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          }
        }
      },
      {
        "feed_url": "https://example.com/rss2",
        "guid": "968cb508-803c-493c-8ff2-9e397dadb83c",
        "is_subscribed": false,
        "subscription_changed": "2022-04-24T17:53:21.573Z",
        "_links": {
          "self": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          },
          "delete": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          },
          "update": {
            "href": "http://opa.org/api/v1/subscriptions/31740ac6-e39d-49cd-9179-634bcecf4143"
          }
        }
      }
    ]
  },
  "_links": {
    "self": {
      "href": "http://opa.org/api/v1/subscriptions?page=3&per_page=5"
    },
    "next": {
      "href": "http://opa.org/api/v1/subscriptions?page=4&per_page=5"
    },
    "prev": {
      "href": "http://opa.org/api/v1/subscriptions?page=2&per_page=5"
    },
    "first": {
      "href": "http://opa.org/api/v1/subscriptions?page=1&per_page=5"
    },
    "last": {
      "href": "http://opa.org/api/v1/subscriptions?page=7&per_page=5"
    }
  },
  "page" : {
    "size" : 5,
    "totalElements" : 34,
    "totalPages" : 7,
    "number" : 0
  }
}

Each embedded object contains a full map of what actions the current user can actually do with the element, including updates to the record or just fetching the record proper from its current location if it's tombstoned.

Capabilities

The capabilities endpoint is another place where links become useful. In the current endpoint proposal, there's a lot of complexity hidden behind parsing parameters on the server-side, and in parsing response data on the client-side:

{
  "urn:opa:core": {
    "1.0.0": {
      "status": "STABLE",
      "root": "/api/v1"
    }
  },
  "urn:opa:extra:playcount": {
    "0.0.1": {
      "status": "DEPRECATED",
      "root": "/api/v0/playcount"
    },
    "1.0.0": {
      "status": "STABLE",
      "root": "/api/v1/playcount"
    },
    "2.0.0": {
      "status": "UNSTABLE",
      "root": "/api/v2/playcount"
    }
  }
}

However, in a HAL approach this could be cleaned up using links that expose metadata about each capability.

{
  "_links": {
    "core:v1": {
      "href": "http://opa.org/api/v1",
      "version": "1.0.0",
      "status": "STABLE"
    },
    "core:v2": {
      "href": "http://opa.org/api/v2",
      "version": "2.0.0",
      "status": "UNSTABLE"
    },
    "playcount:v1": {
      "href": "http://opa.org/api/v1/playcount",
      "version": "1.0.0",
      "status": "STABLE"
    },
    "playcount:v2": {
      "href": "http://opa.org/api/v2/playcount",
      "version": "2.0.0",
      "status": "UNSTABLE"
    }
  }
}

The root page for the different API versions (e.g. /api/v1 and /api/v2) can also act as a directory for services.

{
  "_links": {
    "self": {
      "href": "http://opa.org/api/v1"
    },
    "subscriptions": {
      "href": "http://opa.org/api/v1/subscriptions"
    },
    "playcount": {
      "href": "http://opa.org/api/v1/playcount"
    }
  },
  "version": "1.0.0",
  "status": "STABLE",
  "description": "Open Podcast API version 1"
}

[Sporiff]

As a general point here: HAL is given as an example, but it's not the only choice we have for enriching API responses. Hydra is another option, which makes use of JSON-LD to link data together.

Hydra/JSON-LD gives us a lot of flexibility in defining schemas that are discoverable to clients. It does require more work on our part, as we would need to provide the @context for each endpoint. However, I don't consider this much of a lift. The bigger question would be how easy it is on the client side and how good the support for Hydra is in different languages.

The benefit of HAL is that it gets some of the way towards creating a linked map of data while also being significantly simpler to implement than Hydra. Adopting one or the other will mean a tradeoff. However, I am of the opinion that using either would make the API far more flexible and would aid server developers in guiding clients towards different features, which will become important as the specification grows and additional features are added.