Bulk API for process stats

[stephanme]

Issue

There is no bulk api to fetch process stats of multiple processes at once.

Context

cf a in a space that contains ~800 apps takes ~5min. One reason is that for every app process a single CF API request (GET /v3/processes/<process-guid>/stats) is executed to fetch the process stats.

There are additional reasons for the long execution time like cloudfoundry/cli#2733.

Possible Fix

Implement a list endpoint to fetch process stats for multiple processes, e.g.

• GET /v3/processes/stats with query params similar to GET /v3/processes, or
• add an include=stats parameter to GET /v3/processes

For good performance, it would be desirable that the cloud controller can fetch the process stats from log-cache using a bulk api as well.

[svkrieger]

Usage of stats endpoint in CF CLI

• cf apps command uses the state to display how many instances are running
• cf app <app_name> displays various information like state and metrics of instances

Current stats endpoint

The current stats endpoint does the following:

• Fetch DesiredLRPs from Diego for the given process_guid (metric tag (for filtering LogCache envelopes by process_guid) and isolation segment is being used/ can be removed I think, as this info is also available in the CCDB)
• Metrics are being fetched from LogCache using the GET API
• ActualLRPs are being fetched from Diego, which contain the state and network information etc.

Current limitations of downstream services

Diego

Currently we can only fetch ActualLRPs for one process, not for several.

LogCache

• Using the GET API we can fetch metrics for all processes and their instances for one app
• The PromQL API supports fetching one metric for many apps. We have 8 metrics though, so it would result in 8 requests to LogCache. Calls to this API are way more expensive compared to Diego API.

Possible improvements

Bulk endpoint to fetch state of instances for a whole space

To improve the performance of the cf apps command we could offer a bulk endpoint, which supports fetching the state (and the other info from Diego) for all instances in one space.

GET /v3/processes/state?space_guids=<space_guid>,...

• The user can specify space_guids as query parameter
• This could be extended in the future by app_guids, process_guids query parameters if needed
• Response contains info from Diego and CCDB, but not LogCache
• Prerequisite is a bulk endpoint for ActualLRPs in Diego

Bulk endpoint for stats for all instances of an app

This endpoint would be rather for convenience as usually apps only have one or two processes (based on my experience).

GET /v3/apps/:guid/processes/stats

• By binding this endpoint to one app, we do not need a bulk endpoint for metrics from LogCache, as we can already fetch metrics for all processes for an app.
• Endpoint would show exactly the same data as the current stats endpoint, but including the process_guid.
• Prerequisite is also a bulk endpoint for ActualLRPs

[svkrieger]

As the Diego folks already implemented the endpoint for fetching ActualLRPs for several process_guids and that resolves our prerequisites, I'd like to get a discussion going about the state endpoint. I'd prefer to offer only fetching the state (and all other data from the actual lrp) for ONE space and not several. This would limit the load we put on CC and Diego. Also there is currently only a use case by the cf cli, which needs the state of all processes for a whole space for the cf apps command.

I see two options where we could put such an endpoint:

Option 1: /v3/processes/state?space_guid=...

[+] Could be extended in the future with query parameters app_guids and process_guids

[-] Query parameter space_guid (singular) does not exist yet in the API

Option 2: /v3/spaces/:space_guid/processes/state

[+] Very clear that state can be only fetched for a whole space

[-] We should also serve /v3/spaces/:space_guid/processes then, as that does not exist yet

Payload

For the payload I'd suggest we offer data similar to the current stats endpoint, but without the fields provided by logcache and adding 2 guids (process, app).

{
  "resources": [
    {
      "type": "web",
      "process_guid": "some-process-guid",              # NEW
      "app_guid": "some-app-guid",                      # NEW
      "index": 0,
      "instance_guid": "77e31085-3d74-4518-70d7-d07f",
      "state": "RUNNING",
      "routable": true,
      "host": "10.0.16.18",
      "instance_internal_ip": "10.255.235.67",
      "uptime": 92329,
      "fds_quota": 16384,
      "isolation_segment": null,
      "details": null,
      "instance_ports": [
        {
          "external": 0,
          "internal": 8080,
          "external_tls_proxy_port": 61000,
          "internal_tls_proxy_port": 61001
        },
        {
          "external": 0,
          "internal": 8080,
          "external_tls_proxy_port": null,
          "internal_tls_proxy_port": 61443
        },
        {
          "external": 0,
          "internal": 2222,
          "external_tls_proxy_port": 61001,
          "internal_tls_proxy_port": 61002
        }
      ],
    }
  ]
}

[philippthun]

Here is a (new) proposal for the new endpoint leveraging the bulk API for ActualLRPs provided by Diego.

As ActualLRPs are instances of processes, the most natural naming of this (pseudo) resource would be "instances" or "process instances".

To show instances for a specific process, the following endpoint is proposed: GET /v3/processes/:guid/instances

A single "process instances" resource looks like this (GET /v3/processes/bae2eca5-2935-40bb-9e25-986445cb771d/instances):

{
  "process_guid": "bae2eca5-2935-40bb-9e25-986445cb771d",
  "instances": [
    {
      "index": 0,
      "state": "RUNNING",
      "uptime": 712
    },
    {
      "index": 1,
      "state": "RUNNING",
      "uptime": 598
    },
    {
      "index": 2,
      "state": "RUNNING",
      "uptime": 599
    }
  ],
  "created_at": "2025-11-18T14:54:41Z",
  "updated_at": "2025-11-18T14:56:41Z",
  "links": {
    "self": {
      "href": "https://api.cf.mydomain.com/v3/processes/bae2eca5-2935-40bb-9e25-986445cb771d/instances"
    },
    "process": {
      "href": "https://api.cf.mydomain.com/v3/processes/bae2eca5-2935-40bb-9e25-986445cb771d"
    }
  }
}

As the "process instances" resource does not have an own GUID, the (parent) process_guid is shown.
For now only the following is returned: index, state and uptime. This can be enhanced by other fields from the ActualLRP, if needed.

The proposal for the bulk endpoint to show instances for multiple processes is: GET /v3/processes/instances

This is a new pattern for the v3 api, i.e. a list of sub-resources for a list of resources. The only alternative I see is defining "process instances" as top-level resource (GET /v3/process-instances), but the hierarchical nature of GET /v3/processes/instances provides a clearer picture of the relationship from my point of view.

Similar to GET /v3/processes without any filtering parameters showing all processes, GET /v3/processes/instances would show all instances of all processes. Instances are grouped by processes and pagination as well as ordering and filtering by created_at or updated_at timestamps are based on processes as well.

The endpoint supports similar filtering parameters as its parent endpoint:

GET /v3/processes	GET /v3/processes/instances
guids (i.e. process guids)	process_guids
app_guids	same
space_guids	same
organization_guids	same
types	no (do we need this?)

All process instances for a given space (the cli use case, i.e. cf apps) can be fetched by e.g. GET /v3/processes/instances?space_guids=09b6945b-9bfd-4c71-88a2-c61febf7756d.

The result looks as follows:

{
  "pagination": {
    "total_results": 3,
    "total_pages": 1,
    "first": {
      "href": "https://api.cf.mydomain.com/v3/processes/instances?page=1&per_page=50&space_guids=09b6945b-9bfd-4c71-88a2-c61febf7756d"
    },
    "last": {
      "href": "https://api.cf.mydomain.com/v3/processes/instances?page=1&per_page=50&space_guids=09b6945b-9bfd-4c71-88a2-c61febf7756d"
    },
    "next": null,
    "previous": null
  },
  "resources": [
    {
      "process_guid": "bae2eca5-2935-40bb-9e25-986445cb771d",
      "instances": [
        {
          "index": 0,
          "state": "RUNNING",
          "uptime": 712
        },
        {
          "index": 1,
          "state": "RUNNING",
          "uptime": 598
        },
        {
          "index": 2,
          "state": "RUNNING",
          "uptime": 599
        }
      ],
      "created_at": "2025-11-18T14:54:41Z",
      "updated_at": "2025-11-18T14:56:41Z",
      "links": {
        "self": {
          "href": "https://api.cf.mydomain.com/v3/processes/bae2eca5-2935-40bb-9e25-986445cb771d/instances"
        },
        "process": {
          "href": "https://api.cf.mydomain.com/v3/processes/bae2eca5-2935-40bb-9e25-986445cb771d"
        }
      }
    },
    {
      "process_guid": "019251d2-f592-4049-949d-57429e5a456c",
      "instances": [
        {
          "index": 0,
          "state": "RUNNING",
          "uptime": 557
        }
      ],
      "created_at": "2025-11-18T14:57:17Z",
      "updated_at": "2025-11-18T14:57:23Z",
      "links": {
        "self": {
          "href": "https://api.cf.mydomain.com/v3/processes/019251d2-f592-4049-949d-57429e5a456c/instances"
        },
        "process": {
          "href": "https://api.cf.mydomain.com/v3/processes/019251d2-f592-4049-949d-57429e5a456c"
        }
      }
    },
    {
      "process_guid": "eebc94fa-3e74-4d51-8f29-7f7ee8e0539d",
      "instances": [
        {
          "index": 0,
          "state": "RUNNING",
          "uptime": 539
        }
      ],
      "created_at": "2025-11-18T14:57:35Z",
      "updated_at": "2025-11-18T14:57:42Z",
      "links": {
        "self": {
          "href": "https://api.cf.mydomain.com/v3/processes/eebc94fa-3e74-4d51-8f29-7f7ee8e0539d/instances"
        },
        "process": {
          "href": "https://api.cf.mydomain.com/v3/processes/eebc94fa-3e74-4d51-8f29-7f7ee8e0539d"
        }
      }
    }
  ]
}

[philippthun]

@Gerg - Could you please have a look at the proposal? Thanks!

[Gerg]

Some thoughts:

In a vacuum, I think GET /v3/processes/:guid/instances looks good.

There is some prior art to returning non-resource data from pseudo-resource endpoints.

For example:

GET /v3/apps/:app_guid/environment_variables

{
  "var": {
    "RAILS_ENV": "production"
  },
  "links": {
    "...": "..."
  }
}

In that case, I think you can drop the process_guid, created_at, and updated_at fields.

There are also currently a couple endpoints that already match the /v3/processes/:guid/instances path:

1. DELETE /v3/processes/:guid/instances/:index
2. DELETE /v3/apps/:guid/processes/:type/instances/:index

I think it would be pretty easy to implement a GET /v3/processes/:guid/instances/:index also, though we probably don't have a use case for that currently.

---

I'm less excited about GET /v3/processes/instances.

but the hierarchical nature of GET /v3/processes/instances provides a clearer picture of the relationship

A similar hierarchy exists with orgs/spaces, but we don't model it as GET /v3/organizations/spaces.

Similar to GET /v3/processes without any filtering parameters showing all processes, GET /v3/processes/instances would show all instances of all processes. Instances are grouped by processes and pagination as well as ordering and filtering by created_at or updated_at timestamps are based on processes as well.

A couple things here make me uneasy:

1. GET .../instances isn't returning a list of instance resources. Instead it's returning bundles of instances, grouped by process with some process data mixed in.
2. Unlike the above endpoint, it's returning something that kind of looks like a regular API resource , but doesn't exactly match (e.g. they have a process_guid instead of guid field).

---

Some potential options/suggestions:

1. Call them something like process_instance_bundles instead of instances, to make it clearer what is being returned.
2. Convert instances into a first-class resource. ActualLRPs have instance_guids and since timestamps, so we could populate the guid and *_at fields from those. So something like:

   GET /v3/instances
   

   {
     "pagination": "...",
     "resources": [
           {
             "guid": "a6989e9f-8899-447d-48ac-46a9",
             "created_at": "2025-12-01T21:05:48Z",
             "updated_at": "2025-12-01T21:05:48Z",
             "index": 0,
             "state": "RUNNING",
             "uptime": 712,
             "relationships": {
                 "process": {
                   "data": {
                     "guid": "ae422b5b-ef8e-4000-83c4-1a09a993891a"
                   }
                 }
             }
           }
       ]
   }

3. Add a query parameter to endpoints that return processes to embed instances in the process response. Something like:

   GET /v3/processes?space_guids=:space_guid&embed_instances=true
   

   {
     "pagination": "...",
     "resources": [
       {
         "guid": "ae422b5b-ef8e-4000-83c4-1a09a993891a",
         "created_at": "2025-12-01T21:05:48Z",
         "updated_at": "2025-12-01T21:06:10Z",
         "version": "30aa35de-8783-4651-bee4-0128da233fb8",
         "type": "web",
         "instances": [
           {
             "index": 0,
             "state": "RUNNING",
             "uptime": 712
           },
           {
             "index": 1,
             "state": "RUNNING",
             "uptime": 598
           },
           {
             "index": 2,
             "state": "RUNNING",
             "uptime": 599
           }
         ],
         "...": "..."
       }
     ]
   }

[philippthun]

Thanks for your thoughts on the bulk endpoint @Gerg. I also wasn't entirely satisfied with that approach, so I appreciate your suggestions.

Introducing instances as a first-class resource (GET /v3/instances) would indeed be the cleanest conceptual model, and - as you wrote - the required fields (guid, created_at, updated_at) could be derived. However, I don't see a way to guarantee deterministic ordering for pagination: the timestamps exist only in the data returned from Diego. Cloud Controller itself only has knowledge of processes and uses process GUIDs to request instance data from Diego. Because of that, CC cannot maintain a stable global ordering across all instances.

Given this limitation, the most pragmatic option seems to be embedding instances within processes. It would be the first time v3 embeds something, but I think the justification is reasonable: embedding is only used for pseudo-resources that don't have their own list endpoint.

If we want to go in that direction, what do you think about a more generic parameter, e.g. embed=instances (similar to include)? This would also give us flexibility to embed other pseudo-resources in the future if needed (e.g. GET /v3/apps?embed=environment_variables).

Regarding the single-resource endpoint: I assume we would still keep the dedicated endpoint at GET /v3/processes/:guid/instances (without process_guid, created_at and updated_at as you pointed out), right?

[Gerg]

Given this limitation, the most pragmatic option seems to be embedding instances within processes. It would be the first time v3 embeds something, but I think the justification is reasonable: embedding is only used for pseudo-resources that don't have their own list endpoint.

Sounds good to me!

If we want to go in that direction, what do you think about a more generic parameter, e.g. embed=instances (similar to include)? This would also give us flexibility to embed other pseudo-resources in the future if needed (e.g. GET /v3/apps?embed=environment_variables).

I can think of three main options here:

1. Add a one-off embed_instances parameter (flexible on name).
   • This is the simplest, since it doesn't introduce/modify any API-wide concepts.
   • If we do end up adding something more generic, we can deprecate this param and make it syntactic sugar on top of whatever we build.
2. Add a new API-wide embed parameter for pseudo-resources.
   • This is the most potentially flexible and could be extended to support other things like embedding environment variables in apps.
   • On the other hand, it's maybe over-engineering to add a whole new concept if we are actually only going to use it once.
   • It is pretty similar to the existing include & fields params, so it may be confusing for API consumers to select which param to use.
   • Users may then want to include non-pseudo-resources also, which is then slipping us closer to re-implementing inline-relations-depth.
3. Extend the existing include parameter to also support pseudo resources (in addition to relationships).
   • There is nothing to say that we can't extend the current API styleguide/spec to make the include param more powerful.
   • This may be more intuitive for people who already understand how include work and means we wouldn't need to introduce a new concept.
   • This would increase the conceptual complexity of how the include param works.
   • For the process list endpoint, we would probably need to add some more data so clients can link the included instances back to the listed processes.

Unless y'all have plans to add broader support for embed, my weak preference is for option (1). I'm also interested to hear what others think.

Regarding the single-resource endpoint: I assume we would still keep the dedicated endpoint at GET /v3/processes/:guid/instances (without process_guid, created_at and updated_at as you pointed out), right?

👍

[philippthun]

Extending the existing include parameter could look like this:

GET /v3/processes?include=instances

{
  "pagination": "...",
  "resources": [
    {
      "guid": "process-guid",
      "...": "...",
      "relationships": {
        "instances": {
          "data": [
            { "guid": "instance-guid-1" },
            { "guid": "instance-guid-2" }
          ]
        },
        "...": "..."
      },
      "links": "..."
    }
  ],
  "included": {
    "instances": [
      {
        "guid": "instance-guid-1",
        "index": 0,
        "state": "RUNNING",
        "uptime": 712
      },
      {
        "guid": "instance-guid-2",
        "index": 1,
        "state": "RUNNING",
        "uptime": 598
      }
    ]
  }
}

• This approach fits reasonably well into the existing include concept and clients could resolve a process's instances by their GUIDs as they do with other included resources.
• However, since each process can have multiple instances, the combined list of included instances could become quite large. There has been past discussion about how pagination for included resources might work, but because these pseudo-resources do not exist in the database, implementing pagination would be difficult. On the other hand, instance objects are fairly small, so returning all of them - even when the count exceeds per_page - might still be acceptable.
• Ordering is another open question. Other included resources are ordered by created_at + GUID. In this case, the "natural" ordering is the process ordering plus the instance index. Should we preserve that, or instead impose ordering by instance GUID to match existing conventions?

Regarding options 1 and 2 (embed_instances=true vs. embed=instances), I don't see a substantial difference. Even if we never introduce additional embeddable pseudo-resources, I prefer embed=instances because it aligns better with our existing parameter patterns. I don't think we should treat this as a new API-wide concept.

[philippthun]

We will implement the following new endpoints / parameters:

• GET /v3/processes/:guid/instances
  • The node containing the data will be named resources (not instances).
  • There will be links (self + process).
  • There won't be pagination.

{
  "resources": [
    {
      "index": 0,
      "state": "RUNNING",
      "uptime": 712
    },
    {
      "index": 1,
      "state": "RUNNING",
      "uptime": 598
    }
  ],
  "links": {
    "self": "...",
    "process": "..."
  }
}

• GET /v3/processes/:guid?embed=instances
  • The resources array from GET /v3/processes/:guid/instances will be embedded as instances.

{
  "guid": "process-guid",
  "...": "..."
  "instances": [
    {
      "index": 0,
      "state": "RUNNING",
      "uptime": 712
    },
    {
      "index": 1,
      "state": "RUNNING",
      "uptime": 598
    }
  ],
  "relationships": "...",
  "links": "..."
}

• GET /v3/processes?embed=instances
  • For every process the list of instances will be embedded.

{
  "pagination": "...",
  "resources": [
    {
      "guid": "process-guid",
      "...": "..."
      "instances": [
        {
          "index": 0,
          "state": "RUNNING",
          "uptime": 712
        },
        {
          "index": 1,
          "state": "RUNNING",
          "uptime": 598
        }
      ],
      "relationships": "...",
      "links": "..."
    }
  ]
}