feat: Add metric aggregation configuration for hydrating system_labels

[adityastic]

This PR takes over #416

This commit addresses the need to access GCP Monitoring's system_labels metadata
(like machine_type) which are not returned by default through the APIs. By adding
support for metric aggregation configuration, users can now explicitly instruct
the API to include system labels in the response.

Related issues which could use this feature:

• How to get instance_name in metrics from agent.googleapis.com/agent/* #78
• Export Resource Metadata Labels #115

Changes:

• System Labels Access: GCP Monitoring's system_labels metadata (e.g., machine_type)
  are not returned by default through the APIs
• Metrics Enhancement: Enables better visibility with system labels

New Features:

• Metric Aggregation Configuration: Add support for specifying custom aggregation
  options per metric prefix using --monitoring.metrics-with-aggregations flag
• System Labels Integration: Automatically include system labels from Stackdriver
  metadata in exported metrics
• Enhanced Configuration Format: Support for alignment period, cross-series
  reducer, group-by fields, and per-series aligner configuration

Technical Implementation:

• Add MetricAggregationConfig struct to handle aggregation configuration
• Implement aggregation logic in monitoring collector with proper API integration
• Add JSON unmarshaling for system labels from Stackdriver metadata
• Extend command-line interface with new aggregation configuration flag

Configuration Format:

The new aggregation configuration uses the format:
metric_name:alignment_period:cross_series_reducer:group_by_fields:per_series_aligner

Example for machine_type access:
compute.googleapis.com/reservation/used:60s:REDUCE_SUM:resource.label."resource_container",resource.label."location",resource.label."reservation_id",metadata.system_labels."machine_type",metric.label."reserved_resource_type":ALIGN_MEAN

Use Case Example:

To expose machine_type dimension for reservation metrics:

--monitoring.metrics-with-aggregations="compute.googleapis.com/reservation/used:60s:REDUCE_SUM:resource.label.\"resource_container\",resource.label.\"location\",resource.label.\"reservation_id\",metadata.system_labels.\"machine_type\",metric.label.\"reserved_resource_type\":ALIGN_MEAN"

Documentation:

• Update README.md with new configuration option and usage examples

[adityastic]

@SuperQ mind giving it a review! Thanks!

[terryyanko]

@adityastic Thank you for this! Would it be possible to grab the user defined labels as part of this PR as well? 🙏 https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries#MonitoredResourceMetadata

[kgeckhart]

👋 @adityastic is this something that requires all the extra configuration for system_labels?

Looking at the docs, https://cloud.google.com/monitoring/alerts/doc-variables#null-values, it seems as though the requirement is that the system labels be used in a filter. Could someone configure the system labels they want directly and we tack on a system_label!="" to the query?

I think we could have a separate conversation regarding introducing support for aggregations/reduce configuration. If we go forward with the current implementation I don't see many people being able to take advantage of system_labels due to the complexity required to configure their support. I would very much like to see a scrape result from your example config to see how the metrics types and names are treated. Applying any of the reduce functions seems like it would dramatically change the context of the metric but that won't be reflected in the metric name.

[Misfits09]

👋 @adityastic is this something that requires all the extra configuration for system_labels?

Looking at the docs, cloud.google.com/monitoring/alerts/doc-variables#null-values, it seems as though the requirement is that the system labels be used in a filter. Could someone configure the system labels they want directly and we tack on a system_label!="" to the query?

I think we could have a separate conversation regarding introducing support for aggregations/reduce configuration. If we go forward with the current implementation I don't see many people being able to take advantage of system_labels due to the complexity required to configure their support. I would very much like to see a scrape result from your example config to see how the metrics types and names are treated. Applying any of the reduce functions seems like it would dramatically change the context of the metric but that won't be reflected in the metric name.

After some experimenting on my end adding a filter on metada.system_labels does not work out of the box, as the data needs to be aggregated/aligned... I got this error on my end with a basic filter metadata.system_labels.name != ""

time=2025-12-12T15:36:36.783Z level=ERROR source=monitoring_collector.go:239 msg="Error while getting Google Stackdriver Monitoring metrics" project_id=fifteen-data-team err="googleapi: Error 400: Field f
ilter had an invalid value of \"metric.type=\"agent.googleapis.com/disk/percent_used\" AND (metadata.system_labels.name != \"\")\": Resource metadata filters are only allowed on aligned metrics., badReque
st"