[DO NOT MERGE] SAES-110: APW validation timing experiment

[davidfeng-datadog]

Draft PR — do not merge.

Experimental PR to measure APW validation timing for a single integration. Tracked in SAES-110.

Touches only sym/README.md so the review-deployment flow produces a lean tarball containing only the sym collection.

Will be closed once the tarball is captured.

[cit-pr-commenter-54b7da]

Monitor Template Quality Assessment

52 monitors analyzed across 9 integrations.

• 33 monitors have missing sections
• Most common missing sections: WHY, IMPACT, HOW_TO_TROUBLESHOOT

Monitors with missing sections

Integration	Monitor	Missing Sections	Suggested Links
kubernetes	[Kubernetes] Monitor Kubernetes Deployments Replica Pods	IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS	- Logs

• Infrastructure
• Metrics Explorer (kubernetes_state.deployment.replicas_desired)
• Metrics Explorer (kubernetes_state.deployment.replicas_available) |
  | kubernetes | [Kubernetes] Monitor Unschedulable Kubernetes Nodes | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes_state.node.status) |
  | kubernetes | [Kubernetes] Pod {{pod_name.name}} is CrashloopBackOff on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes_state.container.status_report.count.waiting) |
  | kubernetes | [Kubernetes] Pod {{pod_name.name}} is ImagePullBackOff on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes_state.container.status_report.count.waiting) |
  | kubernetes | [Kubernetes] Pod {{pod_name.name}} is OOMKilled on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes.containers.state.terminated) |
  | kubernetes | [Kubernetes] Monitor Kubernetes Failed Pods in Namespaces | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes_state.pod.status_phase) |
  | kubernetes | [Kubernetes] Monitor Kubernetes Pods Restarting | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes.containers.restarts) |
  | kubernetes | [Kubernetes] Monitor Kubernetes Statefulset Replicas | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (kubernetes_state.statefulset.replicas_desired)
• Metrics Explorer (kubernetes_state.statefulset.replicas_ready) |
  | nginx | [NGINX] 4xx Errors higher than usual | WHY | - Logs
• Infrastructure
• Metrics Explorer (nginx.upstream.peers.responses.4xx) |
  | postgres | [Postgres] Number of connections is approaching connection limit on {{host.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (postgresql.percent_usage_connections) |
  | postgres | [Postgres] Replication delay is abnormally high on {{host.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (postgresql.replication_delay) |
  | redis | [Redis] High memory consumption | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (redis.mem.used)
• Metrics Explorer (redis.mem.maxmemory) |
  | amazon-rds | [AWS] RDS CPU utilization is high | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (aws.rds.cpuutilization) |
  | amazon-rds | [AWS] RDS Anomaly in database connections | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (aws.rds.database_connections) |
  | amazon-rds | [AWS] RDS Storage utilization is high | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
• Infrastructure
• Metrics Explorer (aws.rds.free_storage_space)
• Metrics Explorer (aws.rds.total_storage_space) |
  | amazon-sqs | Queue {{queuename.name}} has a high number of visible messages | WHY | - Logs
• Infrastructure
• Metrics Explorer (aws.sqs.approximate_number_of_messages_visible) |
  | apm | Abnormal change in p75 latency for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | Watchdog Anomaly: {{event.title}} for {{service}} | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: event-v2 alert |
  | apm | New issues found for $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | High p95 latency for $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | Abnormal change in Apdex for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | Low traffic on $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | New issues found for {{service}} on {{env}} | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: error-tracking alert |
  | apm | No data for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | apm | Increased error rate for $service.value in $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
  | real-user-monitoring | View Error Rate is high for view {{[view.name].name}} | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name} by {view.name' (line 1, column 53). |
  | real-user-monitoring | Interaction to Next Paint is high for view {{[view.name].name}} | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name} by {view.name' (line 1, column 49). |
  | real-user-monitoring | App Startup Time is high | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name}' (line 1, column 35). |
  | real-user-monitoring | Crash-Free Session Rate is low | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name}.as_count() / ' (line 1, column 38). |
  | real-user-monitoring | Hang Rate is high | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name}.as_count(), 2' (line 1, column 50). |
  | system | Disk latency is high for host {{host.name}} on device {{device.name}} | WHY | - Logs
• Infrastructure
• Metrics Explorer (system.io.await) |
  | system | Network traffic (sent) is high on host {{host.name}} | WHY | - Logs
• Infrastructure
• Metrics Explorer (system.net.bytes_sent) |
  | system | Host $host.value not reporting data | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: service check |

[github-actions]

This pull request has not been updated for more than 21 days. If there are no updates to this PR within 7 days, it will be closed. If you'd like to re-open this PR after it's been closed, you can start from the latest master branch or pull the latest changes into your branch and create a new pull request.