Pattern to replace soft-delete

[rofrankel]

@toumorokoshi mentioned having an idea for a design pattern that could serve as an alternative to soft-delete, letting us further standardize Delete. He kindly offered to draft an AEP describing that pattern.

[toumorokoshi]

collecting a list of places to remove references:

• Clean up AIP-132: List #108 (comment)
• 135#soft-delete

[toumorokoshi]

The background

As of 2025-09-13, the aeps have a pattern in which one can "soft delete" a
resource - where it not actually removed from a collection, but is actually
flagged as deleted. This allows the resource to be recoverable later.

The soft delete pattern, I believe, exists to serve the purpose of situations
where recovery is desired, such as most situations where persistent data is
stored:

• A database, where recreating the resource does not restore any valuble data
  stored.
• A filesystem, where a file may need to be recovered after the fact.

The implementation of soft delete today

The current implementation of AEPs' soft delete boils down to the following:

1. a new custom method, undelete, on the resource.
2. an added field show_deleted to the List standard method, where resources
   are hidden unless specified.
3. an added field show_deleted to the Get standard method, where the resource will
   return 410 without it.

There are a few other details (such as the usage of expiry_date) that are
additional, but do not force any additional fields on the standard methods.

The problems: soft delete is not compatible with declarative clients

Declarative clients expect consistent resource-oriented operations on each
resource - for simple CRUD on a resource, declarative clients are able to easily
map each operation to the resource lifecycle.

graph LR
  exists
  not_exists
  exists -- "update" --> exists
  not_exists -- "create" --> exists
  exists -- "delete" --> not_exists

Loading

For soft delete, where would that fit in? Effectively it creates another state - soft deleted.

graph LR
  exists
  soft_deleted
  not_exists
  not_exists -- "create" --> exists
  exists -- "update" --> exists
  soft_delete -- "undelete" --> exists
  soft_delete -- "expiry hit" --> not_exists

Loading

But this may also result in weird cases like:

A resource "create" throwing an error because the resource is soft-deleted, and
therefore exists. But a get would return a 404.

Proposed alternative

Since the primary use case is about backing up and enabling restoration of data,
the proposal would be to replace these augmentations on the primary resource, to
adding a second resource for "backups" or "snapshots".

The pattern would look something like:

/database/foo
/database-snapshots/foo/snapshots/latest

The database-snapshot could be support a custom method, restore, that would be
similar to undelete and bring an old resource back.

This would be similar to the resource revision pattern -
where a design pattern is introduced by adding a new resource, not by augmenting
standard methods.

[thegagne]

We discussed on the call 3 options:

1. Some sort of state field in the resource itself to mark it as soft deleted.
2. A shadow collection of some nature (such as snapshots as mentioned above)
3. Using the pattern of resource revisions.

These all have their pros and cons.

After the call, I ran some of this past ChatGPT and it came to similar conclusions that we had, but it argued a bit harder for the first option as the most straightforward, and citing how Kubernetes resources have metadata.deletionTimestamp, that mark something as in the process of being deleted.

Adapting this idea to AEP patterns, AEP-148 already has delete_time and purge_time fields defined.

This still leaves scenarios that need addressing:

1. How to un-delete something - probably just apply it with empty delete_time?
2. What should the behavior be when re-creating something that has been soft deleted? Should it be allowed? I think not as it does not respect the soft delete. It should be forcibly deleted first. You could use apply however to undelete it and update it.
3. How to force deletion? Possibly query param for force_delete=true?
4. How to list with or without soft deleted items? Which is the default?
5. Whether or not soft-delete is the default action for a collection, or it's opt-in at delete time? I think I favor it being set only at the collection level (opt-out of soft delete, or take the default behavior for the collection)
6. How to document whether a collection implements soft delete by default?

The other options may still be worth considering, I was just thinking through this option a bit further.

[toumorokoshi]

Discussion:

• we can adjust the existing pattern where create would overwrite the existing resource.

• @kindermoumoute feels that the soft deleted resource pattern is better.

• @rambleraptor says:

  • if you have multiple backups, use resource revisions.

use cases:

• list soft-deleted resources.
• restore a soft-deleted resource
• restore from a backup to a new resource.

[thegagne]

Answering my own questions from our discussion:

1. We could use the :undelete custom method, but I selfishly like the apply option above because AWS API Gateway is still challenging for custom methods
2. Re-creating should overwrite a soft-deleted resource, and that soft-deleted resource should be deleted/overwritten and no longer available for restore.
3. We did not speak to forced deletion, this should probably be addressed.
4. Listing without soft-deleted should be the default. A query parameter could be added for show_deleted or something to that effect.
   5 and 6. We did not address how to document collection delete default behavior, or opting-in or out, but this should be addressed as well.

Workflow considerations

Create

• If soft delete exists, it would be overwritten.
• The backend system may need to implement a delete & create behind the scenes, depending on how the resource is implemented.

Update/Apply

• These should 404 by default if the resource is soft deleted.
• Should you be able to pass in a query parameter, or use a custom method to update a soft deleted thing?
• If we use a query poaram to allow manipulation of soft deleted resource, is that the path to undelete via removing the delete_time and purge_time fields?

Get

• Should 404 by default for resources that are soft deleted.
• Should 200 and return resource when query param is added (something like ?show_deleted=true
• One thing I don't like about this is that you are adding a query parameter, which is typically used for filtering results of a collection, but now it exists on a single resource. Maybe it should be a custom method instead.

List

• By default, soft deleted resources are not listed
• Query parameter ?show_deleted=true may show them, or possibly this gets included in the CEL filter?
• Would it be desirable to have a list of ONLY deleted items? A mixed list? Both options?

Delete

• Do we still return a 204 no-content result here, or do we return the resource with additional fields for delete_time and purge_time? Changing 204 to 200 might be a breaking change for clients.

[kindermoumoute]

restore from a backup to a new resource.

In our approach a DatabaseBackup is a resource of its own with its own lifecycle. Thus the restore action is considered as any custom method:

rpc RestoreDatabaseBackup(RestoreDatabaseBackupRequest) returns (DatabaseBackup)

This backup can be restored on any non-deleted database.
My preference for infrastructure operation is to use a custom method, weither is CancelDeletion during the deletion grace period, or RestoreBackup. Can it be generalized to any asynchronously deleted resource? Maybe 🤔

restore a soft-deleted resource

I feel I would only implement this kind of method for a resource that is synchronously deleted, such as an IAM policy. And in that case, it would keep the same ID and get/list the deleted resources with a show_deleted=true parameter. Also should the deleted nature of the resource be displayed in the resource full path somehow?

If we use a query poaram to allow manipulation of soft deleted resource, is that the path to undelete via removing the delete_time and purge_time fields?
Do we still return a 204 no-content result here, or do we return the resource with additional fields for delete_time and purge_time?

Update and Delete should just return 404 IMO.

[thegagne]

Update and Delete should just return 404 IMO.

Sorry, for this I was talking about the create and then [soft] delete pattern. If the delete operation produces a soft-deleted resource, do we return that information in the delete response? This is different from a delete in a collection that does not offer soft-delete. I think we could return the normal no content response, which is maybe not as user-intuitive, but is more machine consistent with normal operations.

[toumorokoshi]

We discussed this offline today in the sync. the general consensus was that we make the change to soft-delete as a sibling collection. Rationale:

1. removing it now allows us to remove a bunch of if/else casing on soft-deleted resources (all of the edge cases you raised @thegagne).
2. we can always re-introduce this pattern later.

Thoughts? @thegagne

[thegagne]

I'll come back to this next week when I have more time to consider it.

I agree that the if-then logic is problematic. I do think that there is need to support some pattern of "disabled but not deleted", which is probably different than soft delete.

A simple disabled: true sort of field could work here. They would show up in lists, but could be filtered out (but not by default).

I would want to better understand the mechanics of the sibling collection for soft delete.

How does it get backed up?
How does it get restored?

Are those both custom methods?