Subscriptions endpoint draft spec

[Sporiff]

Subscriptions endpoint

The subscriptions endpoint is used to synchronize subscriptions between a server and connected clients. A client can perform the following actions:

1. Retrieve a list of subscriptions
2. Retrieve a single subscription
3. Add new subscriptions
4. Update an existing subscription
5. Delete an existing subscription

The server is treated as the authoritative source for subscription information. Clients can query the endpoint by specifying the datetime from which they want to fetch changes to ensure they only fetch information that is relevant to them since their last sync.

Note: To ensure the best compatibility with REST, a bulk DELETE action is not supported. The client can instead send multiple DELETE queries to the endpoint to remove several subscriptions.

• Subscriptions endpoint
  • Authentication
  • GUID authority
  • Get all subscriptions
    • Parameters
    • Server-side behavior
    • Example request
    • Example 200 response
  • Get a single subscription
    • Server-side behavior
    • Example request
    • Example 200 response
  • Add new subscriptions
    • Parameters
    • Server-side behavior
    • Example request
    • Example 200 response
  • Update an existing subscription
    • Parameters
    • Example request
    • Example 200 response
  • Delete an existing subscription
    • Server-side behavior
    • Example request
    • Example 200 response

Authentication

All endpoints use Oauth 2 to authenticate. The exact mechanism is still open for discussion. Users create Oauth tokens by authenticating with an account on the server. When they query the endpoints, they are able to interact only with information relating to their account.

GUID authority

GUIDs are a feature of the Podcasting 2.0 specification which uniquely identify the podcast. To handle this and ensure that this API complies with the Podcasting 2.0 spec, all subscription entries must be assigned a GUID.

For the best compatibility, the following sources should be used in order of preference:

1. The guid field in the source RSS feed. This should not change at any point in the future and should be considered authoritative
2. The podcastGuid field returned by the Podcast Index's /podcasts/byfeedurl endpoint. This should only be queried if:
   1. The server administrator opts into using this service and provides an API key
   2. The RSS feed doesn't contain a guid field, possibly due to being out-of-date
3. A server-assigned guid that is generated only if neither of the above sources contain any information

Get all subscriptions

GET /subscriptions

This endpoint enables clients to return all subscription information relating to the authenticated user. It returns the following information:

Field	Data type	Required?	Description
feed_url	String	Yes	The URL of the podcast RSS feed
guid	String<UUID>	Yes	The globally unique ID of the podcast
is_subscribed	Boolean	Yes	Whether the user is subscribed to the podcast or not
subscription_changed	Datetime	No	The date on which the is_subscribed field was last updated. Presented in ISO 8601 format
guid_changed	Datetime	No	The date on which the podcast's GUID was last updated. Presented in ISO 8601 format
new_guid	String<UUID>	No	The new GUID associated with the podcast

Parameters

The client can attach an optional since parameter in the path This is a Datetime parameter that specifies the range of data the endpoint will return. The endpoint will only return objects in which either the subscription_changed or guid_changed fields were updated after the datetime specified in the since parameter.

GET /subscriptions?since=2022-04-23T18:25:43.511Z

Note: If no since parameter is provided, the server returns all subscription information. Only the latest entries should be returned (see server-side behavior below).

Server-side behavior

If the entry contains a new_guid, the server should return the newest guid associated with the entry. For example: if a subscription has received 2 new guids, the server should follow these entries and only return the last one it finds. This ensures the client has the most up-to-date entry for the subscription.

Example request

curl -X 'GET' \
  '/subscriptions?since=2022-04-23T18%3A25%3A34.511Z' \
  -H 'accept: application/json'

Example 200 response

{
   "subscriptions": [
      {
         "feed_url": "https://example.com/rss1",
         "guid": "31740ac6-e39d-49cd-9179-634bcecf4143",
         "is_subscribed": true,
         "guid_changed": "2022-09-21T10:25:32.411Z",
         "new_guid": "8d1f8f09-4f50-4327-9a63-639bfb1cbd98"
      },
      {
         "feed_url": "https://example.com/rss2",
         "guid": "968cb508-803c-493c-8ff2-9e397dadb83c",
         "is_subscribed": false,
         "subscription_changed": "2022-04-24T17:53:21.573Z"
      }
   ]
}

Get a single subscription

GET /subscriptions/{guid}

This endpoint returns subscription information relating to a specific subscription for the authenticated user. It returns the following information:

Field	Data type	Required?	Description
feed_url	String	Yes	The URL of the podcast RSS feed
guid	String<UUID>	Yes	The globally unique ID of the podcast
is_subscribed	Boolean	Yes	Whether the user is subscribed to the podcast or not
subscription_changed	Datetime	Yes	The date on which the is_subscribed field was last updated. Presented in ISO 8601 format
guid_changed	Datetime	No	The date on which the podcast's GUID was last updated. Presented in ISO 8601 format
new_guid	String<UUID>	No	The new GUID associated with the podcast

Server-side behavior

If the entry contains a new_guid, the server should return the newest guid associated with the entry. For example: if a subscription has received 2 new guids, the server should follow these entries and only return the last one it finds. This ensures the client has the most up-to-date entry for the subscription.

Example request

curl --location '/subscriptions/968cb508-803c-493c-8ff2-9e397dadb83c'

Example 200 response

{
   "feed_url": "https://example.com/rss2",
   "guid": "968cb508-803c-493c-8ff2-9e397dadb83c",
   "is_subscribed": true
}

Add new subscriptions

POST /subscriptions

This endpoint enables clients to add new subscriptions to the system for the authenticated user. It returns the following information:

Field	Data type	Required?	Description
feed_url	String	Yes	The URL of the podcast RSS feed
guid	String<UUID>	Yes	The globally unique ID of the podcast
is_subscribed	Boolean	Yes	Whether the user is suscribed to the podcast or not
subscription_changed	Datetime	Yes	The date on which the is_subscribed field was last updated. Presented in ISO 8601 format

Parameters

The client must provide an array of feed_urls in the request body to add them to the system.

{
   "feed_urls": [
       "https://example.com/rss1",
       "https://example.com/rss2",
       "https://example.com/rss3",
       "https://example.com/rss4"
   ]
}

Server-side behavior

When new feeds are posted to the server, the server must perform the following actions:

1. Search the RSS feed for an existing guid field and save this as the guid if it exists
2. If Podcast Index searching is enabled, query the /podcasts/byfeedurl endpoint for the feed in question. If there is a response and it contains a podcastGuid field, save this value as the guid
3. If no existing guid is found, generate a new one and assign it to the entry's guid field.
4. If the feed already exists in the system, do the following:
   1. If the user is already subscribed to the feed, do nothing
   2. If the user is unsubscribed from the feed, set the is_subscribed field to true

Example request

curl --location '/subscriptions' \
--header 'Content-Type: application/json' \
--data '{
   "feed_urls": [
       "https://example.com/rss1",
       "https://example.com/rss2",
       "https://example.com/rss3",
       "https://example.com/rss4"
   ]
}'

Example 200 response

{
   "subscriptions": [
      {
         "feed_url": "https://example.com/rss1",
         "guid": "8d1f8f09-4f50-4327-9a63-639bfb1cbd98",
         "is_subscribed": true,
         "subscription_changed": "2023-02-23T14:00:00.000Z"
      },
      {
         "feed_url": "https://example.com/rss2",
         "guid": "968cb508-803c-493c-8ff2-9e397dadb83c",
         "is_subscribed": true,
         "subscription_changed": "2023-02-23T14:00:00.000Z"
      },
      {
         "feed_url": "https://example.com/rss3",
         "guid": "e672c1f4-230d-4ab4-99d3-390a9f835ec1",
         "is_subscribed": true,
         "subscription_changed": "2023-02-23T14:00:00.000Z"
      },
      {
         "feed_url": "https://example.com/rss4",
         "guid": "2d8bb39b-8d34-48d4-b223-a0d01eb27d71",
         "is_subscribed": true,
         "subscription_changed": "2023-02-23T14:00:00.000Z"
      },
   ]
}

Update an existing subscription

PATCH /subscriptions/{guid}

This endpoint allows clients to update information about a subscription. It returns the following information:

Field	Data type	Required?	Description
new_feed_url	String	No	The URL of the podcast RSS feed. Only returned if the feed_url field was updated by the request
is_subscribed	Boolean	No	Whether the user is suscribed to the podcast or not. Only returned if the is_subscribed field was updated by the request
subscription_changed	Datetime	No	The date on which the is_subscribed field was last updated. Presented in ISO 8601 format. Only returned if the is_subscribed field was updated by the request
guid_changed	Datetime	No	The date on which the podcast's GUID was last updated. Presented in ISO 8601 format. Only returned if the guid field was updated by the request
new_guid	String<UUID>	No	The new GUID associated with the podcast. Only returned if the guid field was updated by the request

Parameters

The client must provide an the following parameters in the request body:

Parameter	Data type	Required?	Desciption
new_feed_url	String	No	The URL of the new RSS feed for the subscription
new_guid	String <UUID>	No	The new GUID of the podcast
is_subscribed	Boolean	No	Whether the user is suscribed to the podcast or not

Example request

curl --location --request PATCH '/subscriptions/2d8bb39b-8d34-48d4-b223-a0d01eb27d71' \
--header 'Content-Type: application/json' \
--data '{
    "new_feed_url": "https://example.com/rss5",
    "new_guid": "965fcecf-ce04-482b-b57c-3119b866cc61",
    "is_subscribed": false
}'

Example 200 response

{
   "new_feed_url": "https://example.com/rss5",
   "is_subscribed": false,
   "subscription_changed": "2023-02-23T14:41:00.000Z",
   "guid_changed": "2023-02-23T14:41:00.000Z",
   "new_guid": "965fcecf-ce04-482b-b57c-3119b866cc61"
}

Delete an existing subscription

DELETE /subscriptions/{guid}

This endpoint allows clients to mark a feed as deleted. This prevents the server from updating the feed in the background and prevents the server returning any information such as playback positions relating to the given associated feed. It returns the following information:

Field	Data type	Required?	Description
feed_url	String	Yes	The URL of the podcast RSS feed
guid	String<UUID>	Yes	The globally unique ID of the podcast
is_deleted	Boolean	Yes	Whether the feed is deleted or not

Server-side behavior

When a DELETE request is received, the server must do the following:

1. Set the is_deleted flag in the subscriptions table to true
2. Set the is_deleted flag on any previous entries in the subscriptions table relating to the feed to true

Example request

curl --location --request POST '/subscriptions/2d8bb39b-8d34-48d4-b223-a0d01eb27d71'

Example 200 response

{
   "feed_url": "https://example.com/rss5",
   "guid": "2d8bb39b-8d34-48d4-b223-a0d01eb27d71",
   "is_deleted": true
}

[Sporiff]

Just a note: this doesn't include the data flow or database structure for this information. I'll write that up separately.

[keunes]

Do we need to describe the database structure? Though tempting - isn't that up to each implementing party?

For example for the is_deleted status: I guess we should focus on the functional expectation (for reason X, the data is expected to remain in place while endpoints will no longer return anything)

[georgkrause]

The database structure should be left at the discretion of the implementers, it doesn't really matter for our spec how the data is stored and different implementations might have different needs and restrictions. Lets mind our own business :)

[keunes]

Actually we do have some basic database info/expectations: https://openpodcastapi.github.io/pr-previews/pr-26/subscriptions/index.html#database-schema

It's broad enough and makes sense, to be honest. Would you agree @georgkrause?

[georgkrause]

I don't think thats a good idea. What we care about is the API to allow clients and servers to communicate with each other in a standardized way. We don't care how data is stored. For example Funkwhale probably stores all the information already, if this is part of the spec we would rename database fields OR don't comply with the spec. We can add this as an example but declare it to be non-normative.

[keunes]

On the other hand; how can you communicate in a standardised way if you don't agree on concepts, like what 'deleted' means. Or what if you receive a string when you expect an integer.

if this is part of the spec we would rename database fields OR don't comply with the spec

Right. I don't think it's the intention to specify what something needs to be called in the database, but rather (as said) what we communicate. Maybe we can change the phrasing from 'Database scheme' to 'Data terminology' or something, and from 'Nullable' to 'Null possibly returned' or sth? @Sporiff?

[georgkrause]

Some first-read quick feedback on a formal level:

• I think we need to introduce pagination into the list endpoint.

• We must be more careful with the key words for requirements

• The exact mechanism is still open for discussion.

  I'd like to avoid wordings like this. The spec should be written in a way that is independent from the time someone might read it. I guess its much better to state this is out of scope for this draft and we can add a reference to the specification of our oauth usage later on.

[keunes]

We must be more careful with the key words for requirements

You mean "we MUST be more careful" ? ;)

The spec should be written in a way that is independent from the time someone might read it. I guess its much better to state this is out of scope for this draft and we can add a reference to the specification of our oauth usage later on.

If with 'this draft' you refer to the draft spec for the Subscriptions endpoint, then I agree. (Though we'll probably want to describe it before 'finalising' the v 1 or 0.1 specs.)

[georgkrause]

I think a huge issue we haven't covered yet needs to be solved:

The GUID in the RSS-Feed has the highest priority. This means when I create a new subscription using the /subscriptions endpoint sending the feed url in the body, the server must fetch the feed in order to get the GUID. This might be not possible (network issues, slow response) but we already need a GUID which we can return.

A possible solution would be to try to fetch the Feed, extract the GUID and everything is fine. Only if this doesn't work in a short amount of time, we generate an GUID. The server may then try to fetch the Feed in the background from time to time in order to update if required.

I just checked the proposal again and I think there is no specification for fetching the Feed URL anyways, which probably should be changed.

[georgkrause]

The problem here is the timing. The users wants to subscribe to a new podcast, in any client. Can the client subscribe without telling the server? If yes, it needs to synchronise in the background and its not too much of an issue, but then we cannot make use of the error message (or we need to define client behaviour more precisely).

If syncing is part of the subscription activity itself, the client does not want to wait forever until the server made another http request.

Maybe its better to generate a GUID on our own in the first place and fetch the feed later on which might cause a forwarding to a new GUID, so slightly more data but much easier to implement and likely a way better user experience

[keunes]

I just checked the proposal again and I think there is no specification for fetching the Feed URL anyways, which probably should be changed.

The server should be able to work without ever fetching any feed, in order to support a fully decentralised model.

In the context of subscriptions, if a server doesn't fetch feeds, can't we just rely on the client to send a GUID along? As long as it respects the same creation rules, it should probably be fine. Therefore, I think simply add the 'GUID received from client' to the list, before @Sporiff's point 1, would do (?)

Then if no GUID is received and the feed server is inaccessible, a temporary GUID might work. At the same time - do we really expect servers will use the GUID as their internal unique identifier? If not - is it a major issue if a subscription is without a GUID for a short while?

Can the client subscribe without telling the server?

This should certainly be possible. AntennaPod shouldn't be 'blocked' if the server isn't available.

[georgkrause]

I vaguely remember we had some discussions about this, but not sure if we came to an agreement.

I agree we should make sure the client can act independently from the server.

The server should be able to work without ever fetching any feed, in order to support a fully decentralised model.

@keunes I don't really understand why fetching the feed is defeating the decentralization. Also, if we want to use the authoritative GUID, we probably want to reassure its correctly sent by the client.

At the same time - do we really expect servers will use the GUID as their internal unique identifier?

I don't think we expect them, but it MAY be used and anyways, the GUID is the resource ID used to query the info.

If not - is it a major issue if a subscription is without a GUID for a short while?

It is, since the GUID is the resource identifier you use to read/write information to/from the API.

[keunes]

I don't really understand why fetching the feed is defeating the decentralization.

The question is: who fetches the feed? If it's only the server that can do it, we have a problem. The client should be able to add a podcast while a server is down. We did discuss and I think we properly addressed it in the specs now (to be confirmed).

if we want to use the authoritative GUID, we probably want to reassure its correctly sent by the client.

Didn't we say that a) the client should follow the same GUID steps as the server and b) clients should be trusted (because they are expected to follow the same GUID steps? (Also to be checked.)
Edit: for point a) we said that the client only is allowed to pass on a GUID if it comes from the feed: https://openpodcastapi.github.io/pr-previews/pr-26/subscriptions/add-new.html#request-parameters

It is, since the GUID is the resource identifier you use to read/write information to/from the API.

If the client doesn't communicate with any server, it's not a problem. If the client communicates with the server for the first time about a specific feed, I believe we expect the server to return the GUID (also tbc).

[georgkrause]

Jeah I think we already discussed all of it and it just wasn't reflected here.

The client can pass the GUID, if none is given we use the feed url to generate one and then we can fetch the feed and eventually redirect the old guid to the new one. thats fine!

If the client doesn't communicate with any server, it's not a problem. If the client communicates with the server for the first time about a specific feed, I believe we expect the server to return the GUID (also tbc).

Jeah, so we have a GUID in all cases and we are fine :)

[JonOfUs]

I'm not sure in which case the DELETE request should be used. Should it always be used if a user unsubscribes from a podcast, in combination with a PATCH request changing the subscription status? Or should podcast clients have the option to unsubscribe from and to delete a podcast separately?

edit: the problem that could occur is that the state is_deleted will be difficult to sync to other devices if the subscription is excluded from updates and maybe even syncs. Or does the flag only disable background feed updates and behaves more like a 'disable'/'pause' flag?

[keunes]

What is a deletion in Antennapod's world? How is it different to an unsubscribe event?

Currently: physical delete. But to fulfil the feature request, we might conceive that it is given a 'special tag'.

An Unsubscribe event in AntennaPod speak I think would be turning off the 'Keep updated' switch (all data maintained, feed no longer refreshed).

But maybe @ByteHamster has different ideas.

As for how we would reflect it, the actual subscription GET request could include the is_deleted flag using the same logic as the others if it's something useful, where we return it when it has changed since the last call.

How does the server produce the 'changed since' list again? By keeping a datestamp on a given podcast, right? If so, then in order to provide the 'changed since' for that (since deleted) subscription, we'd need to keep the subscription in the database with that flag. Meaning the data can't be physically deleted. Right?

[georgkrause]

The example is fully valid and we should keep the is_deleted, however the implementers should decide on how long to keep the data. If they decide to delete after 7 days, people might need to resolve sync issues when a decide is offline for a longer period of time.

[keunes]

I believe we discussed this at the previous meeting. From what I remember, the tombstone can be kept indefinitely as they will only contain the guid(s).

[georgkrause]

Its just that we cannot really write the obligation to store data "indefinitely" into our specification, so I'd rather advice people to do so and tell them that the kind of issue you described above might occur if they delete tombstones.

[JonOfUs]

That sounds good. This sounds like a good usage case for the keyword 'SHOULD' ;)

SHOULD - This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.