Peak EWMA load balancer (contrib) randomly segfaults -- (Envoy via Istio-Proxy 1.29.0)

[frittentheke]

Title: Peak EWMA load balancer (contrib) randomly segfaults -- (Envoy via Istio-Proxy 1.29.0)

Description:

I certainly understand that the Istio-Proxy is a custom Envoy build and that Peak EWMA is a contrib module on top of that. I clarified with the envoy-security contact (@ggreenway), that this crash issue is OK to report publicly.

It seems to me that this NOT related to traffic since I can run load of requests across different upstream targets with no issues.
At first glance it seems like random crashes. Maybe (!) they are related to the constant Pod and target churn happening in Kubernetes cluster. It's just a sandbox cluster, so no production traffic affected.

But that also means I gladly run more test if they help (you).

Repro steps:

1. I wanted to try the contrib loadbalancer module for Peak EWMA, added via PR contrib: implement Peak EWMA load balancing policy #40653 by @rroblak

2. Since we are running Istio, I proposed to add the PeakEWMA load balancer via Add Peak EWMA load balancer (contrib) istio/proxy#6690, which was merged and is now part of Istio-Proxy >=1.29.0.

3. I upgraded our Istio sandbox to 1.29.0 (which means it runs https://github.com/istio/proxy/releases/tag/1.29.0).

4. Since Peak EWMA is not yet directly configurable via Istio configuration I added an EnvoyFilter which I attached as YAML.

5. After a while I noticed some container restarts and with Envoy crashing and throwing strack traces.

Config:

EnvoyFilter

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: peak-ewma-lb
  namespace: istio-ingress
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.peak_ewma
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.peak_ewma.v3alpha.PeakEwmaConfig
  - applyTo: CLUSTER
    match:
      context: GATEWAY
    patch:
      operation: MERGE
      value:
        load_balancing_policy:
          policies:
          - typed_extension_config:
              name: envoy.clusters.lb.peak_ewma
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.load_balancing_policies.peak_ewma.v3alpha.PeakEwma
  workloadSelector:
    labels:
      istio: ingressgateway-external

This configuration itself works and I can see peak_ewma is configured for
all clusters as well as I can see the related stats being produced:

[...]


{
     "cluster": {
      "@type": "[type.googleapis.com/envoy.config.cluster.v3.Cluster](http://type.googleapis.com/envoy.config.cluster.v3.Cluster)",
      "name": "outbound|80||httpbin.sandbox.svc.cluster.local",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {},
        "initial_fetch_timeout": "0s",
        "resource_api_version": "V3"
       },
       "service_name": "outbound|80||httpbin.sandbox.svc.cluster.local"
      },
      "connect_timeout": "10s",
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 4294967295,
         "max_pending_requests": 4294967295,
         "max_requests": 4294967295,
         "max_retries": 4294967295
        }
       ]
      },
      "metadata": {
       "filter_metadata": {
        "istio": {
         "services": [
          {
           "host": "httpbin.sandbox.svc.cluster.local",
           "name": "httpbin",
           "namespace": "sandbox"
          }
         ]
        }
       }
      },
      "common_lb_config": {},
      "alt_stat_name": "outbound|80||httpbin.sandbox.svc.cluster.local;",
      "filters": [
       {
        "name": "istio.metadata_exchange",
        "typed_config": {
         "@type": "[type.googleapis.com/udpa.type.v1.TypedStruct](http://type.googleapis.com/udpa.type.v1.TypedStruct)",
         "type_url": "[type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange](http://type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange)",
         "value": {
          "protocol": "istio-peer-exchange",
          "enable_discovery": true
         }
        }
       }
      ],
      "load_balancing_policy": {
       "policies": [
        {
         "typed_extension_config": {
          "name": "envoy.clusters.lb.peak_ewma",
          "typed_config": {
           "@type": "[type.googleapis.com/envoy.extensions.load_balancing_policies.peak_ewma.v3alpha.PeakEwma](http://type.googleapis.com/envoy.extensions.load_balancing_policies.peak_ewma.v3alpha.PeakEwma)"
          }
         }
        }
       ]
      },


[...]

Logs:

[...]

{"level":"info","time":"2026-02-16T19:47:35.946536Z","msg":"FLAG: --log_as_json=\"true\""}                                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:35.946539Z","msg":"FLAG: --log_caller=\"\""}                                                                                                                                           
{"level":"info","time":"2026-02-16T19:47:35.946542Z","msg":"FLAG: --log_output_level=\"default:info\""}                                                                                                                         
{"level":"info","time":"2026-02-16T19:47:35.946544Z","msg":"FLAG: --log_stacktrace_level=\"default:none\""}                                                                                                                     
{"level":"info","time":"2026-02-16T19:47:35.946553Z","msg":"FLAG: --log_target=\"[stdout]\""}                                                                                                                                   
{"level":"info","time":"2026-02-16T19:47:35.946557Z","msg":"FLAG: --meshConfig=\"./etc/istio/config/mesh\""}                                                                                                                    
{"level":"info","time":"2026-02-16T19:47:35.946560Z","msg":"FLAG: --outlierLogPath=\"\""}                                                                                                                                       
{"level":"info","time":"2026-02-16T19:47:35.946562Z","msg":"FLAG: --profiling=\"true\""}                                                                                                                                        
{"level":"info","time":"2026-02-16T19:47:35.946565Z","msg":"FLAG: --proxyComponentLogLevel=\"misc:error\""}                                                                                                                     
{"level":"info","time":"2026-02-16T19:47:35.946568Z","msg":"FLAG: --proxyLogLevel=\"warning\""}                                                                                                                                 
{"level":"info","time":"2026-02-16T19:47:35.946571Z","msg":"FLAG: --serviceCluster=\"istio-proxy\""}                                                                                                                            
{"level":"info","time":"2026-02-16T19:47:35.946574Z","msg":"FLAG: --stsPort=\"0\""}                                                                                                                                             
{"level":"info","time":"2026-02-16T19:47:35.946576Z","msg":"FLAG: --templateFile=\"\""}                                                                                                                                         
{"level":"info","time":"2026-02-16T19:47:35.946578Z","msg":"FLAG: --tokenManagerPlugin=\"\""}                                                                                                                                   
{"level":"info","time":"2026-02-16T19:47:35.946582Z","msg":"FLAG: --vklog=\"0\""}                                                                                                                                               
{"level":"info","time":"2026-02-16T19:47:35.946585Z","msg":"Version 1.29.0-2300e2458ab713c2c514a58da1ea8b03343ada7e-Clean"}                                                                                                     
{"level":"info","time":"2026-02-16T19:47:35.946918Z","msg":"Proxy role","ips":["REDACTED"],"type":"router","id":"istio-ingressgateway-external-6d675cfb69-vwt6g.istio-ingress","domain":"istio-ingress.svc.cluster.local"}  
{"level":"info","time":"2026-02-16T19:47:35.947002Z","msg":"Apply proxy config from env {\"runtimeValues\":{\"overload.global_downstream_max_connections\":\"175000\"},\"gatewayTopology\":{\"forwardClientCertDetails\":\"SANIT
IZE_SET\",\"proxyProtocol\":{}},\"terminationDrainDuration\":\"150s\",\"proxyStatsMatcher\":{\"inclusionRegexps\":[\"listener\\\\..*_(80|443).*\",\"downstream_cx_proxy_proto_error\",\"http.*downstream_cx_.*\",\"http.*downstr
eam_rq_.*\",\".*overload.*\"]}}\n"}                                                                                                                                                                                             
{"level":"info","time":"2026-02-16T19:47:35.948319Z","msg":"Apply proxy config from annotation terminationDrainDuration: 150s\nruntimeValues:\n  \"overload.global_downstream_max_connections\": \"175000\"\n\nproxyStatsMatcher
:\n      inclusionRegexps:\n      - \"listener\\\\..*_(80|443).*\"\n      - \"downstream_cx_proxy_proto_error\"\n      - \"http.*downstream_cx_.*\"\n      - \"http.*downstream_rq_.*\"\n      - \".*overload.*\"\n"}           
{"level":"info","time":"2026-02-16T19:47:35.948471Z","msg":"cpu limit detected as 2, setting concurrency"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:35.948786Z","msg":"Effective config: binaryPath: /usr/local/bin/envoy\nconcurrency: 2\nconfigPath: ./etc/istio/proxy\ncontrolPlaneAuthPolicy: MUTUAL_TLS\ndiscoveryAddress: istiod.isti
o-system.svc:15012\ndrainDuration: 45s\ngatewayTopology:\n  forwardClientCertDetails: SANITIZE_SET\n  proxyProtocol: {}\nproxyAdminPort: 15000\nproxyStatsMatcher:\n  inclusionRegexps:\n  - listener\\..*_(80|443).*\n  - downs
tream_cx_proxy_proto_error\n  - http.*downstream_cx_.*\n  - http.*downstream_rq_.*\n  - .*overload.*\nruntimeValues:\n  overload.global_downstream_max_connections: \"175000\"\nserviceCluster: istio-proxy\nstatNameLength: 189
\nstatusPort: 15020\nterminationDrainDuration: 150s\n"}
{"level":"info","time":"2026-02-16T19:47:35.948802Z","msg":"JWT policy is third-party-jwt"}                                                                                                                                     
{"level":"info","time":"2026-02-16T19:47:35.948805Z","msg":"using credential fetcher of JWT type in cluster.local trust domain"}                                                                                                
{"level":"info","time":"2026-02-16T19:47:37.022934Z","msg":"Opening status port 15020"}                                                                                                                                         
{"level":"info","time":"2026-02-16T19:47:37.022954Z","msg":"Starting default Istio SDS Server"}                                                                                                                                 
{"level":"info","time":"2026-02-16T19:47:37.022979Z","msg":"CA Endpoint istiod.istio-system.svc:15012, provider Citadel"}                                                                                                       
{"level":"info","time":"2026-02-16T19:47:37.023000Z","msg":"Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem"}                                                                       
{"level":"info","time":"2026-02-16T19:47:37.023836Z","scope":"xdsproxy","msg":"Initializing with upstream address \"istiod.istio-system.svc:15012\" and cluster \"Kubernetes\""}                                                
{"level":"info","time":"2026-02-16T19:47:37.024561Z","scope":"sds","msg":"Starting SDS grpc server"}                                                                                                                            
{"level":"info","time":"2026-02-16T19:47:37.026781Z","scope":"sds","msg":"Starting SDS server for workload certificates, will listen on \"var/run/secrets/workload-spiffe-uds/socket\""}                                        
{"level":"info","time":"2026-02-16T19:47:37.026544Z","msg":"Pilot SAN: [istiod.istio-system.svc]"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:47:37.029600Z","msg":"Starting proxy agent"}                                                                                                                                              
{"level":"info","time":"2026-02-16T19:47:37.029920Z","msg":"Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disab
le-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --config-yaml {\"memory_allocator_manager\":{\"bytes_to_release\":50000000,\"memory_release_interval\":{\"second
s\":1}}} --concurrency 2]"}                                                                                                                                                                                                     
{"level":"info","time":"2026-02-16T19:47:37.112604Z","scope":"cache","msg":"generated new workload certificate","resourceName":"default","latency":"88.079151ms","ttl":"23h59m59.887400729s"}                                   
{"level":"info","time":"2026-02-16T19:47:37.112831Z","scope":"cache","msg":"Root cert has changed, start rotating root cert"}                                                                                                   
{"level":"info","time":"2026-02-16T19:47:37.113011Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":"23h59m59.886990522s"}                                                                             
{"level":"info","time":"2026-02-16T19:47:37.135253Z","scope":"xdsproxy","msg":"connected to delta upstream XDS server: istiod.istio-system.svc:15012","id":1}                                                                   
{"level":"info","time":"2026-02-16T19:47:37.244926Z","scope":"ads","msg":"ADS: new connection for node:1"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.250482Z","scope":"cache","msg":"adding watcher for file certificate /etc/ssl/certs/ca-certificates.crt"}                                                                            
{"level":"info","time":"2026-02-16T19:47:37.250515Z","scope":"cache","msg":"read certificate from file","resource":"file-root:system"}                                                                                          
{"level":"info","time":"2026-02-16T19:47:37.282428Z","scope":"ads","msg":"ADS: new connection for node:2"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.282722Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":"23h59m59.71728127s"}                                                                              
{"level":"info","time":"2026-02-16T19:47:37.285485Z","scope":"ads","msg":"ADS: new connection for node:3"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.285666Z","scope":"cache","msg":"returned workload certificate from cache","ttl":"23h59m59.714336457s"}                                                                              
{"level":"info","time":"2026-02-16T19:47:39.596913Z","msg":"Readiness succeeded in 3.653890211s"}                                                                                                                               
{"level":"info","time":"2026-02-16T19:47:39.597419Z","msg":"Envoy proxy is ready"}                                                                                                                                              
{"time":"2026-02-16T19:51:19.254197Z","level":"critical","scope":"envoy backtrace","msg":"Caught Segmentation fault, suspect faulting address 0x0","caller":"external/envoy/source/server/backtrace.h:129","thread":"14"}       
{"time":"2026-02-16T19:51:19.254252Z","level":"critical","scope":"envoy backtrace","msg":"Backtrace (use tools/stack_decode.py to get line numbers):","caller":"external/envoy/source/server/backtrace.h:113","thread":"14"}    
{"time":"2026-02-16T19:51:19.254257Z","level":"critical","scope":"envoy backtrace","msg":"Envoy version: 1fdadc301aaaa0326255f82bd27387a0e101c164/1.37.1-dev/Clean/RELEASE/BoringSSL","caller":"external/envoy/source/server/bac
ktrace.h:114","thread":"14"}                                                                                                                                                                                                    
{"time":"2026-02-16T19:51:19.254260Z","level":"critical","scope":"envoy backtrace","msg":"Address mapping: 564bf84b2000-564bfbaf3000 /usr/local/bin/envoy","caller":"external/envoy/source/server/backtrace.h:116","thread":"14"
}                                                                                                                                                                                                                               
{"time":"2026-02-16T19:51:19.281722Z","level":"critical","scope":"envoy backtrace","msg":"#0: [0x7fe8afeea330]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                          
{"time":"2026-02-16T19:51:19.286032Z","level":"critical","scope":"envoy backtrace","msg":"#1: event_process_active_single_queue [0x564bfb4d06ac]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}        
{"time":"2026-02-16T19:51:19.290486Z","level":"critical","scope":"envoy backtrace","msg":"#2: event_base_loop [0x564bfb4cf421]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                          
{"time":"2026-02-16T19:51:19.295114Z","level":"critical","scope":"envoy backtrace","msg":"#3: Envoy::Server::InstanceBase::run() [0x564bfa872fb7]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}       
{"time":"2026-02-16T19:51:19.298902Z","level":"critical","scope":"envoy backtrace","msg":"#4: Envoy::MainCommonBase::run() [0x564bfa8195aa]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}             
{"time":"2026-02-16T19:51:19.302649Z","level":"critical","scope":"envoy backtrace","msg":"#5: Envoy::MainCommon::main() [0x564bfa819ab4]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                
{"time":"2026-02-16T19:51:19.309888Z","level":"critical","scope":"envoy backtrace","msg":"#6: main [0x564bf854cf1c]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                     
{"time":"2026-02-16T19:51:19.310508Z","level":"critical","scope":"envoy backtrace","msg":"#7: [0x7fe8afecf1ca]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                          
{"time":"2026-02-16T19:51:19.310547Z","level":"critical","scope":"envoy backtrace","msg":"#8: __libc_start_main [0x7fe8afecf28b]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                        
{"time":"2026-02-16T19:51:19.315180Z","level":"critical","scope":"envoy backtrace","msg":"#9: _start [0x564bf84b202a]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                   
{"level":"error","time":"2026-02-16T19:51:19.323634Z","msg":"Envoy exited with error: signal: segmentation fault (core dumped)"}                                                                                                
{"level":"info","time":"2026-02-16T19:51:19.323863Z","scope":"ads","msg":"ADS: \"\" 1 terminated"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:51:19.324239Z","scope":"ads","msg":"ADS: \"\" 3 terminated"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:51:19.324418Z","scope":"ads","msg":"ADS: \"\" 2 terminated"} 

[...]
{"level":"info","time":"2026-02-16T19:47:35.948802Z","msg":"JWT policy is third-party-jwt"}                                                                                                                                     
{"level":"info","time":"2026-02-16T19:47:35.948805Z","msg":"using credential fetcher of JWT type in cluster.local trust domain"}                                                                                                
{"level":"info","time":"2026-02-16T19:47:37.022934Z","msg":"Opening status port 15020"}                                                                                                                                         
{"level":"info","time":"2026-02-16T19:47:37.022954Z","msg":"Starting default Istio SDS Server"}                                                                                                                                 
{"level":"info","time":"2026-02-16T19:47:37.022979Z","msg":"CA Endpoint istiod.istio-system.svc:15012, provider Citadel"}                                                                                                       
{"level":"info","time":"2026-02-16T19:47:37.023000Z","msg":"Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem"}                                                                       
{"level":"info","time":"2026-02-16T19:47:37.023836Z","scope":"xdsproxy","msg":"Initializing with upstream address \"istiod.istio-system.svc:15012\" and cluster \"Kubernetes\""}                                                
{"level":"info","time":"2026-02-16T19:47:37.024561Z","scope":"sds","msg":"Starting SDS grpc server"}                                                                                                                            
{"level":"info","time":"2026-02-16T19:47:37.026781Z","scope":"sds","msg":"Starting SDS server for workload certificates, will listen on \"var/run/secrets/workload-spiffe-uds/socket\""}                                        
{"level":"info","time":"2026-02-16T19:47:37.026544Z","msg":"Pilot SAN: [istiod.istio-system.svc]"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:47:37.029600Z","msg":"Starting proxy agent"}                                                                                                                                              
{"level":"info","time":"2026-02-16T19:47:37.029920Z","msg":"Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disab
{"level":"info","time":"2026-02-16T19:47:37.112604Z","scope":"cache","msg":"generated new workload certificate","resourceName":"default","latency":"88.079151ms","ttl":"23h59m59.887400729s"}                                   
{"level":"info","time":"2026-02-16T19:47:37.112831Z","scope":"cache","msg":"Root cert has changed, start rotating root cert"}                                                                                                   
{"level":"info","time":"2026-02-16T19:47:37.113011Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":"23h59m59.886990522s"}                                                                             
{"level":"info","time":"2026-02-16T19:47:37.135253Z","scope":"xdsproxy","msg":"connected to delta upstream XDS server: istiod.istio-system.svc:15012","id":1}                                                                   
{"level":"info","time":"2026-02-16T19:47:37.244926Z","scope":"ads","msg":"ADS: new connection for node:1"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.250482Z","scope":"cache","msg":"adding watcher for file certificate /etc/ssl/certs/ca-certificates.crt"}                                                                            
{"level":"info","time":"2026-02-16T19:47:37.250515Z","scope":"cache","msg":"read certificate from file","resource":"file-root:system"}                                                                                          
{"level":"info","time":"2026-02-16T19:47:37.282428Z","scope":"ads","msg":"ADS: new connection for node:2"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.282722Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":"23h59m59.71728127s"}                                                                              
{"level":"info","time":"2026-02-16T19:47:37.285485Z","scope":"ads","msg":"ADS: new connection for node:3"}                                                                                                                      
{"level":"info","time":"2026-02-16T19:47:37.285666Z","scope":"cache","msg":"returned workload certificate from cache","ttl":"23h59m59.714336457s"}                                                                              
{"level":"info","time":"2026-02-16T19:47:39.596913Z","msg":"Readiness succeeded in 3.653890211s"}                                                                                                                               
{"level":"info","time":"2026-02-16T19:47:39.597419Z","msg":"Envoy proxy is ready"}                                                                                                                                              
{"time":"2026-02-16T19:51:19.254197Z","level":"critical","scope":"envoy backtrace","msg":"Caught Segmentation fault, suspect faulting address 0x0","caller":"external/envoy/source/server/backtrace.h:129","thread":"14"}       
{"time":"2026-02-16T19:51:19.254252Z","level":"critical","scope":"envoy backtrace","msg":"Backtrace (use tools/stack_decode.py to get line numbers):","caller":"external/envoy/source/server/backtrace.h:113","thread":"14"}    
{"time":"2026-02-16T19:51:19.254257Z","level":"critical","scope":"envoy backtrace","msg":"Envoy version: 1fdadc301aaaa0326255f82bd27387a0e101c164/1.37.1-dev/Clean/RELEASE/BoringSSL","caller":"external/envoy/source/server/bac
{"time":"2026-02-16T19:51:19.254260Z","level":"critical","scope":"envoy backtrace","msg":"Address mapping: 564bf84b2000-564bfbaf3000 /usr/local/bin/envoy","caller":"external/envoy/source/server/backtrace.h:116","thread":"14"
{"time":"2026-02-16T19:51:19.281722Z","level":"critical","scope":"envoy backtrace","msg":"#0: [0x7fe8afeea330]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                          
{"time":"2026-02-16T19:51:19.286032Z","level":"critical","scope":"envoy backtrace","msg":"#1: event_process_active_single_queue [0x564bfb4d06ac]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}        
{"time":"2026-02-16T19:51:19.290486Z","level":"critical","scope":"envoy backtrace","msg":"#2: event_base_loop [0x564bfb4cf421]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                          
{"time":"2026-02-16T19:51:19.295114Z","level":"critical","scope":"envoy backtrace","msg":"#3: Envoy::Server::InstanceBase::run() [0x564bfa872fb7]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}       
{"time":"2026-02-16T19:51:19.298902Z","level":"critical","scope":"envoy backtrace","msg":"#4: Envoy::MainCommonBase::run() [0x564bfa8195aa]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}             
{"time":"2026-02-16T19:51:19.302649Z","level":"critical","scope":"envoy backtrace","msg":"#5: Envoy::MainCommon::main() [0x564bfa819ab4]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                
{"time":"2026-02-16T19:51:19.309888Z","level":"critical","scope":"envoy backtrace","msg":"#6: main [0x564bf854cf1c]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                     
{"time":"2026-02-16T19:51:19.310508Z","level":"critical","scope":"envoy backtrace","msg":"#7: [0x7fe8afecf1ca]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                          
{"time":"2026-02-16T19:51:19.310547Z","level":"critical","scope":"envoy backtrace","msg":"#8: __libc_start_main [0x7fe8afecf28b]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                        
{"time":"2026-02-16T19:51:19.315180Z","level":"critical","scope":"envoy backtrace","msg":"#9: _start [0x564bf84b202a]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                   
{"level":"error","time":"2026-02-16T19:51:19.323634Z","msg":"Envoy exited with error: signal: segmentation fault (core dumped)"}                                                                                                
{"level":"info","time":"2026-02-16T19:51:19.323863Z","scope":"ads","msg":"ADS: \"\" 1 terminated"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:51:19.324239Z","scope":"ads","msg":"ADS: \"\" 3 terminated"}                                                                                                                              
{"level":"info","time":"2026-02-16T19:51:19.324418Z","scope":"ads","msg":"ADS: \"\" 2 terminated"}   

I switched to debug logs now and will try to provide more context (what happens before the crashes).

Call Stack:
I hope the stack from the access logs above helps. Please instruct me what to provide otherwise.

[ggreenway]

Based on the backtrace, my best guess is it's an issue in the shutdown code here:

envoy/contrib/peak_ewma/load_balancing_policies/source/peak_ewma_lb.cc

Line 98 in 5608b86

main_dispatcher_.post([timer = std::move(aggregation_timer_)]() mutable {

. Since stopping the timer is posted, that posted message may run after the timer runs, if both are scheduled in the same iteration of the event loop. The timer needs to be canceled immediately, not posted.

[frittentheke]

Based on the backtrace, my best guess is it's an issue in the shutdown code here:

envoy/contrib/peak_ewma/load_balancing_policies/source/peak_ewma_lb.cc

Line 98 in 5608b86
main_dispatcher_.post(timer = std::move(aggregation_timer_) mutable {
. Since stopping the timer is posted, that posted message may run after the timer runs, if both are scheduled in the same iteration of the event loop. The timer needs to be canceled immediately, not posted.

Wow @ggreenway thanks for the lightning fast analysis. Would you be able to provide a bugfix based on this already then?
Can I provide any more data that would help you?

We are rather keen on gaining some experience with the advantages of using a PEWMA algo for load balancing, but in this state the crashes happen a little too often ;-)

But since I am running this in a sandbox environment I could do some basic smoke tests of the fix.

[ggreenway]

I don't have time to work on this now, but hopefully someone else can pick this up.

[rroblak]

I'll look into this tomorrow. Thank you both for pushing this forward.

[frittentheke]

@rroblak I have a bit more log from two crashes and what happend just before ...

{"time":"2026-02-17T04:01:17.777857Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:18.016645Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:18.019452Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:18.105087Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:18.133620Z","level":"debug","scope":"envoy conn_handler","msg":"new connection from 10.250.97.217:40664","caller":"external/envoy/source/common/listener_manager/active_tcp_listener.cc:162","thread":"23","ConnectionId":"7611"}                                          
{"time":"2026-02-17T04:01:18.133689Z","level":"debug","scope":"envoy http","msg":"new stream","caller":"external/envoy/source/common/http/conn_manager_impl.cc:417","thread":"23","ConnectionId":"7611"}                                                                                             
{"time":"2026-02-17T04:01:18.133741Z","level":"debug","scope":"envoy http","msg":"request headers complete (end_stream=true):\n':authority', '100.64.110.178:15021'\n':path', '/healthz/ready'\n':method', 'GET'\n'user-agent', 'kube-probe/1.34+'\n'accept', '*/*'\n'connection', 'close'\n","caller
":"external/envoy/source/common/http/conn_manager_impl.cc:1334","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                                                                                                                                               
{"time":"2026-02-17T04:01:18.133752Z","level":"debug","scope":"envoy http","msg":"request end stream timestamp recorded","caller":"external/envoy/source/common/http/conn_manager_impl.cc:1317","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                               
{"time":"2026-02-17T04:01:18.133776Z","level":"debug","scope":"envoy connection","msg":"current connecting state: false","caller":"external/envoy/source/common/network/connection_impl.h:101","thread":"23","ConnectionId":"7611"}                                                                  
{"time":"2026-02-17T04:01:18.133801Z","level":"debug","scope":"envoy router","msg":"cluster 'agent' match for URL '/healthz/ready'","caller":"external/envoy/source/common/router/router.cc:544","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                              
{"time":"2026-02-17T04:01:18.133842Z","level":"debug","scope":"envoy router","msg":"router decoding headers:\n':authority', '100.64.110.178:15021'\n':path', '/healthz/ready'\n':method', 'GET'\n':scheme', 'http'\n'user-agent', 'kube-probe/1.34+'\n'accept', '*/*'\n'x-forwarded-proto', 'http'\n'
x-request-id', 'dd26c38f-a907-4cda-81cd-b1a10f9a333a'\n'x-envoy-expected-rq-timeout-ms', '15000'\n","caller":"external/envoy/source/common/router/router.cc:830","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                                              
{"time":"2026-02-17T04:01:18.133862Z","level":"debug","scope":"envoy pool","msg":"using existing fully connected connection","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:330","thread":"23","ConnectionId":"3"}                                                               
{"time":"2026-02-17T04:01:18.133870Z","level":"debug","scope":"envoy pool","msg":"creating stream","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:247","thread":"23","ConnectionId":"3"}                                                                                         
{"time":"2026-02-17T04:01:18.133881Z","level":"debug","scope":"envoy router","msg":"pool ready","caller":"external/envoy/source/common/router/upstream_request.cc:599","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                                        
{"time":"2026-02-17T04:01:18.133904Z","level":"debug","scope":"envoy client","msg":"encode complete","caller":"external/envoy/source/common/http/codec_client.cc:150","thread":"23","ConnectionId":"3"}                                                                                              
{"time":"2026-02-17T04:01:18.134145Z","level":"debug","scope":"envoy client","msg":"response complete","caller":"external/envoy/source/common/http/codec_client.cc:137","thread":"23","ConnectionId":"3"}                                                                                            
{"time":"2026-02-17T04:01:18.134322Z","level":"debug","scope":"envoy router","msg":"upstream headers complete: end_stream=true","caller":"external/envoy/source/common/router/router.cc:1704","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                 
{"time":"2026-02-17T04:01:18.134354Z","level":"debug","scope":"envoy router","msg":"Executing cleanup(): resetting retry_state_ and disabling timers","caller":"external/envoy/source/common/router/router.cc:1150","thread":"23"}                                                                   
{"time":"2026-02-17T04:01:18.134428Z","level":"debug","scope":"envoy http","msg":"closing connection due to connection close header","caller":"external/envoy/source/common/http/conn_manager_impl.cc:1963","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                   
{"time":"2026-02-17T04:01:18.134532Z","level":"debug","scope":"envoy http","msg":"encoding headers via codec (end_stream=true):\n':status', '200'\n'date', 'Tue, 17 Feb 2026 04:01:18 GMT'\n'content-length', '0'\n'x-envoy-upstream-service-time', '0'\n'server', 'envoy'\n'connection', 'close'\n",
"caller":"external/envoy/source/common/http/conn_manager_impl.cc:2019","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                                                                                                                                        
{"time":"2026-02-17T04:01:18.134547Z","level":"debug","scope":"envoy http","msg":"Codec completed encoding stream.","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2135","thread":"23","ConnectionId":"7611","StreamId":"12136597456474396864"}                                    
{"time":"2026-02-17T04:01:18.134557Z","level":"debug","scope":"envoy router","msg":"Executing cleanup(): resetting retry_state_ and disabling timers","caller":"external/envoy/source/common/router/router.cc:1150","thread":"23"}                                                                   
{"time":"2026-02-17T04:01:18.134564Z","level":"debug","scope":"envoy connection","msg":"closing data_to_write=143 type=0","caller":"external/envoy/source/common/network/connection_impl.cc:181","thread":"23","ConnectionId":"7611"}                                                                
{"time":"2026-02-17T04:01:18.134569Z","level":"debug","scope":"envoy connection","msg":"setting delayed close timer with timeout 1000 ms","caller":"external/envoy/source/common/network/connection_impl_base.cc:51","thread":"23","ConnectionId":"7611"}                                            
{"time":"2026-02-17T04:01:18.134579Z","level":"debug","scope":"envoy pool","msg":"response complete","caller":"external/envoy/source/common/http/http1/conn_pool.cc:61","thread":"23","ConnectionId":"3"}                                                                                            
{"time":"2026-02-17T04:01:18.134584Z","level":"debug","scope":"envoy pool","msg":"destroying stream: 0 active remaining, readyForStream false, currentUnusedCapacity 1","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:282","thread":"23","ConnectionId":"3"}                    
{"time":"2026-02-17T04:01:18.135315Z","level":"debug","scope":"envoy connection","msg":"write flush complete","caller":"external/envoy/source/common/network/connection_impl.cc:859","thread":"23","ConnectionId":"7611"}                                                                            
{"time":"2026-02-17T04:01:18.135328Z","level":"debug","scope":"envoy connection","msg":"closing socket: 1","caller":"external/envoy/source/common/network/connection_impl.cc:313","thread":"23","ConnectionId":"7611"}                                                                               
{"time":"2026-02-17T04:01:18.135351Z","level":"debug","scope":"envoy conn_handler","msg":"adding to cleanup list","caller":"external/envoy/source/common/listener_manager/active_stream_listener_base.cc:125","thread":"23","ConnectionId":"7611"}                                                   
{"time":"2026-02-17T04:01:18.135357Z","level":"debug","scope":"envoy connection","msg":"Not closing conn, socket is not open","caller":"external/envoy/source/common/network/connection_impl.cc:148","thread":"23","ConnectionId":"7611"}                                                            
{"time":"2026-02-17T04:01:18.364444Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:19.013838Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T04:01:19.188577Z","level":"debug","scope":"envoy http2","msg":"Http2Visitor: remaining data payload: 457, stream_id: 1, end_stream: false","caller":"external/envoy/source/common/http/http2/codec_impl.cc:1965","thread":"15","ConnectionId":"7448"}                            
{"time":"2026-02-17T04:01:19.188634Z","level":"debug","scope":"envoy config","msg":"Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.cluster.v3.Cluster at version 2026-02-17T04:01:19Z/126","caller":"external/envoy/source/extensions/config_subscription/grpc/new_grpc_mux_imp
l.cc:158","thread":"15"}                                                                                                                                                                                                                                                                             
{"time":"2026-02-17T04:01:19.188725Z","level":"info","scope":"envoy upstream","msg":"cds: response indicates 1 added/updated cluster(s), 1 removed cluster(s); applying changes","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:31","thread":"15"}                                
{"time":"2026-02-17T04:01:19.188804Z","level":"debug","scope":"envoy upstream","msg":"cds: add/update cluster 'BlackHoleCluster' skipped","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:63","thread":"15"}                                                                       
{"time":"2026-02-17T04:01:19.188861Z","level":"debug","scope":"envoy init","msg":"ClusterImplBase destroyed","caller":"external/envoy/source/common/init/watcher_impl.cc:31","thread":"15"}                                                                                                          
{"time":"2026-02-17T04:01:19.188899Z","level":"debug","scope":"envoy init","msg":"init manager Cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local destroyed","caller":"external/envoy/source/common/init/watcher_impl.cc:31","thread":"15"}                                          
{"time":"2026-02-17T04:01:19.188920Z","level":"debug","scope":"envoy upstream","msg":"removing cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:822","thread":"15"}                                       
{"time":"2026-02-17T04:01:19.188960Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"23"}                                   
{"time":"2026-02-17T04:01:19.188974Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"15"}                                   
{"time":"2026-02-17T04:01:19.189036Z","level":"debug","scope":"envoy upstream","msg":"cds: remove cluster 'outbound|80||deployment-svc.kuberhealthy.svc.cluster.local'","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:76","thread":"15"}                                         
{"time":"2026-02-17T04:01:19.189490Z","level":"info","scope":"envoy upstream","msg":"cds: added/updated 0 cluster(s) (skipped 1 unmodified cluster(s)); removed 1 cluster(s)","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:84","thread":"15"}                                   
{"time":"2026-02-17T04:01:19.189222Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"22"}                                   
{"time":"2026-02-17T04:01:19.191123Z","level":"debug","scope":"envoy config","msg":"Delta config for type.googleapis.com/envoy.config.cluster.v3.Cluster accepted with 1 resources added, 1 removed","caller":"external/envoy/source/extensions/config_subscription/grpc/delta_subscription_state.cc:
276","thread":"15"}                                                                                                                                                                                                                                                                                  
{"time":"2026-02-17T04:01:19.192629Z","level":"critical","scope":"envoy backtrace","msg":"Caught Segmentation fault, suspect faulting address 0x0","caller":"external/envoy/source/server/backtrace.h:129","thread":"15"}                                                                            
{"time":"2026-02-17T04:01:19.192675Z","level":"critical","scope":"envoy backtrace","msg":"Backtrace (use tools/stack_decode.py to get line numbers):","caller":"external/envoy/source/server/backtrace.h:113","thread":"15"}                                                                         
{"time":"2026-02-17T04:01:19.192962Z","level":"critical","scope":"envoy backtrace","msg":"Envoy version: 1fdadc301aaaa0326255f82bd27387a0e101c164/1.37.1-dev/Clean/RELEASE/BoringSSL","caller":"external/envoy/source/server/backtrace.h:114","thread":"15"}                                         
{"time":"2026-02-17T04:01:19.192968Z","level":"critical","scope":"envoy backtrace","msg":"Address mapping: 55bd46e20000-55bd4a461000 /usr/local/bin/envoy","caller":"external/envoy/source/server/backtrace.h:116","thread":"15"}                                                                    
{"time":"2026-02-17T04:01:19.223665Z","level":"critical","scope":"envoy backtrace","msg":"#0: [0x7f53dfd49330]","caller":"external/envoy/source/server/backtrace.h:123","thread":"15"}                                                                                                               
{"time":"2026-02-17T04:01:19.226438Z","level":"critical","scope":"envoy backtrace","msg":"#1: event_process_active_single_queue [0x55bd49e3e6ac]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                             
{"time":"2026-02-17T04:01:19.228956Z","level":"critical","scope":"envoy backtrace","msg":"#2: event_base_loop [0x55bd49e3d421]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                               
{"time":"2026-02-17T04:01:19.231018Z","level":"critical","scope":"envoy backtrace","msg":"#3: Envoy::Server::InstanceBase::run() [0x55bd491e0fb7]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                            
{"time":"2026-02-17T04:01:19.232888Z","level":"critical","scope":"envoy backtrace","msg":"#4: Envoy::MainCommonBase::run() [0x55bd491875aa]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                  
{"time":"2026-02-17T04:01:19.234985Z","level":"critical","scope":"envoy backtrace","msg":"#5: Envoy::MainCommon::main() [0x55bd49187ab4]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                     
{"time":"2026-02-17T04:01:19.236879Z","level":"critical","scope":"envoy backtrace","msg":"#6: main [0x55bd46ebaf1c]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                                          
{"time":"2026-02-17T04:01:19.236918Z","level":"critical","scope":"envoy backtrace","msg":"#7: [0x7f53dfd2e1ca]","caller":"external/envoy/source/server/backtrace.h:123","thread":"15"}                                                                                                               
{"time":"2026-02-17T04:01:19.236932Z","level":"critical","scope":"envoy backtrace","msg":"#8: __libc_start_main [0x7f53dfd2e28b]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                             
{"time":"2026-02-17T04:01:19.238968Z","level":"critical","scope":"envoy backtrace","msg":"#9: _start [0x55bd46e2002a]","caller":"external/envoy/source/server/backtrace.h:121","thread":"15"}                                                                                                        
{"level":"info","time":"2026-02-17T04:01:19.243126Z","scope":"ads","msg":"ADS: \"\" 2 terminated"}                                                                                                                                                                                                   
{"level":"info","time":"2026-02-17T04:01:19.243199Z","scope":"ads","msg":"ADS: \"\" 1 terminated"}                                                                                                                                                                                                   
{"level":"info","time":"2026-02-17T04:01:19.243208Z","scope":"ads","msg":"ADS: \"\" 3 terminated"}                                                                                                                                                                                                   
{"level":"error","time":"2026-02-17T04:01:19.244523Z","msg":"Envoy exited with error: signal: segmentation fault (core dumped)"}  



{"time":"2026-02-17T08:09:11.324336Z","level":"debug","scope":"envoy conn_handler","msg":"adding to cleanup list","caller":"external/envoy/source/common/listener_manager/active_stream_listener_base.cc:125","thread":"22","ConnectionId":"308"}                                                    
{"time":"2026-02-17T08:09:11.324373Z","level":"debug","scope":"envoy connection","msg":"Not closing conn, socket is not open","caller":"external/envoy/source/common/network/connection_impl.cc:148","thread":"22","ConnectionId":"308"}                                                             
{"time":"2026-02-17T08:09:11.349184Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T08:09:11.351449Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:11.438911Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:11.695433Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T08:09:12.344761Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T08:09:12.776930Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:13.015554Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:13.018201Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:13.103805Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"23"}                                                        
{"time":"2026-02-17T08:09:13.321941Z","level":"debug","scope":"envoy conn_handler","msg":"new connection from 10.250.97.217:50710","caller":"external/envoy/source/common/listener_manager/active_tcp_listener.cc:162","thread":"22","ConnectionId":"309"}                                           
{"time":"2026-02-17T08:09:13.322010Z","level":"debug","scope":"envoy http","msg":"new stream","caller":"external/envoy/source/common/http/conn_manager_impl.cc:417","thread":"22","ConnectionId":"309"}                                                                                              
{"time":"2026-02-17T08:09:13.322064Z","level":"debug","scope":"envoy http","msg":"request headers complete (end_stream=true):\n':authority', '100.64.110.178:15021'\n':path', '/healthz/ready'\n':method', 'GET'\n'user-agent', 'kube-probe/1.34+'\n'accept', '*/*'\n'connection', 'close'\n","caller
{"time":"2026-02-17T08:09:13.322075Z","level":"debug","scope":"envoy http","msg":"request end stream timestamp recorded","caller":"external/envoy/source/common/http/conn_manager_impl.cc:1317","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                                
{"time":"2026-02-17T08:09:13.322098Z","level":"debug","scope":"envoy connection","msg":"current connecting state: false","caller":"external/envoy/source/common/network/connection_impl.h:101","thread":"22","ConnectionId":"309"}                                                                   
{"time":"2026-02-17T08:09:13.322124Z","level":"debug","scope":"envoy router","msg":"cluster 'agent' match for URL '/healthz/ready'","caller":"external/envoy/source/common/router/router.cc:544","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                               
{"time":"2026-02-17T08:09:13.322175Z","level":"debug","scope":"envoy router","msg":"router decoding headers:\n':authority', '100.64.110.178:15021'\n':path', '/healthz/ready'\n':method', 'GET'\n':scheme', 'http'\n'user-agent', 'kube-probe/1.34+'\n'accept', '*/*'\n'x-forwarded-proto', 'http'\n'
{"time":"2026-02-17T08:09:13.322195Z","level":"debug","scope":"envoy pool","msg":"using existing fully connected connection","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:330","thread":"22","ConnectionId":"3"}                                                               
{"time":"2026-02-17T08:09:13.322200Z","level":"debug","scope":"envoy pool","msg":"creating stream","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:247","thread":"22","ConnectionId":"3"}                                                                                         
{"time":"2026-02-17T08:09:13.322211Z","level":"debug","scope":"envoy router","msg":"pool ready","caller":"external/envoy/source/common/router/upstream_request.cc:599","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                                                         
{"time":"2026-02-17T08:09:13.322233Z","level":"debug","scope":"envoy client","msg":"encode complete","caller":"external/envoy/source/common/http/codec_client.cc:150","thread":"22","ConnectionId":"3"}                                                                                              
{"time":"2026-02-17T08:09:13.322527Z","level":"debug","scope":"envoy client","msg":"response complete","caller":"external/envoy/source/common/http/codec_client.cc:137","thread":"22","ConnectionId":"3"}                                                                                            
{"time":"2026-02-17T08:09:13.322543Z","level":"debug","scope":"envoy router","msg":"upstream headers complete: end_stream=true","caller":"external/envoy/source/common/router/router.cc:1704","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                                  
{"time":"2026-02-17T08:09:13.322567Z","level":"debug","scope":"envoy router","msg":"Executing cleanup(): resetting retry_state_ and disabling timers","caller":"external/envoy/source/common/router/router.cc:1150","thread":"22"}                                                                   
{"time":"2026-02-17T08:09:13.322577Z","level":"debug","scope":"envoy http","msg":"closing connection due to connection close header","caller":"external/envoy/source/common/http/conn_manager_impl.cc:1963","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                    
{"time":"2026-02-17T08:09:13.322583Z","level":"debug","scope":"envoy http","msg":"encoding headers via codec (end_stream=true):\n':status', '200'\n'date', 'Tue, 17 Feb 2026 08:09:13 GMT'\n'content-length', '0'\n'x-envoy-upstream-service-time', '0'\n'server', 'envoy'\n'connection', 'close'\n",
{"time":"2026-02-17T08:09:13.322593Z","level":"debug","scope":"envoy http","msg":"Codec completed encoding stream.","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2135","thread":"22","ConnectionId":"309","StreamId":"11034755504060836678"}                                     
{"time":"2026-02-17T08:09:13.322600Z","level":"debug","scope":"envoy router","msg":"Executing cleanup(): resetting retry_state_ and disabling timers","caller":"external/envoy/source/common/router/router.cc:1150","thread":"22"}                                                                   
{"time":"2026-02-17T08:09:13.322606Z","level":"debug","scope":"envoy connection","msg":"closing data_to_write=143 type=0","caller":"external/envoy/source/common/network/connection_impl.cc:181","thread":"22","ConnectionId":"309"}                                                                 
{"time":"2026-02-17T08:09:13.322609Z","level":"debug","scope":"envoy connection","msg":"setting delayed close timer with timeout 1000 ms","caller":"external/envoy/source/common/network/connection_impl_base.cc:51","thread":"22","ConnectionId":"309"}                                             
{"time":"2026-02-17T08:09:13.322617Z","level":"debug","scope":"envoy pool","msg":"response complete","caller":"external/envoy/source/common/http/http1/conn_pool.cc:61","thread":"22","ConnectionId":"3"}                                                                                            
{"time":"2026-02-17T08:09:13.322620Z","level":"debug","scope":"envoy pool","msg":"destroying stream: 0 active remaining, readyForStream false, currentUnusedCapacity 1","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:282","thread":"22","ConnectionId":"3"}                    
{"time":"2026-02-17T08:09:13.323189Z","level":"debug","scope":"envoy connection","msg":"write flush complete","caller":"external/envoy/source/common/network/connection_impl.cc:859","thread":"22","ConnectionId":"309"}                                                                             
{"time":"2026-02-17T08:09:13.323225Z","level":"debug","scope":"envoy connection","msg":"closing socket: 1","caller":"external/envoy/source/common/network/connection_impl.cc:313","thread":"22","ConnectionId":"309"}                                                                                
{"time":"2026-02-17T08:09:13.323255Z","level":"debug","scope":"envoy conn_handler","msg":"adding to cleanup list","caller":"external/envoy/source/common/listener_manager/active_stream_listener_base.cc:125","thread":"22","ConnectionId":"309"}                                                    
{"time":"2026-02-17T08:09:13.323269Z","level":"debug","scope":"envoy connection","msg":"Not closing conn, socket is not open","caller":"external/envoy/source/common/network/connection_impl.cc:148","thread":"22","ConnectionId":"309"}                                                             
{"time":"2026-02-17T08:09:13.362990Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T08:09:14.012313Z","level":"debug","scope":"envoy filter","msg":"proxy_protocol: New connection accepted","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:228","thread":"22"}                                                        
{"time":"2026-02-17T08:09:14.035277Z","level":"debug","scope":"envoy main","msg":"flushing stats","caller":"external/envoy/source/server/server.cc:246","thread":"14"}                                                                                                                               
{"time":"2026-02-17T08:09:14.334140Z","level":"debug","scope":"envoy config","msg":"Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.cluster.v3.Cluster at version 2026-02-17T08:09:14Z/111","caller":"external/envoy/source/extensions/config_subscription/grpc/new_grpc_mux_imp
{"time":"2026-02-17T08:09:14.334366Z","level":"info","scope":"envoy upstream","msg":"cds: response indicates 1 added/updated cluster(s), 1 removed cluster(s); applying changes","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:31","thread":"14"}                                
{"time":"2026-02-17T08:09:14.334408Z","level":"debug","scope":"envoy upstream","msg":"cds: add/update cluster 'BlackHoleCluster' skipped","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:63","thread":"14"}                                                                       
{"time":"2026-02-17T08:09:14.334481Z","level":"debug","scope":"envoy init","msg":"ClusterImplBase destroyed","caller":"external/envoy/source/common/init/watcher_impl.cc:31","thread":"14"}                                                                                                          
{"time":"2026-02-17T08:09:14.334744Z","level":"debug","scope":"envoy init","msg":"init manager Cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local destroyed","caller":"external/envoy/source/common/init/watcher_impl.cc:31","thread":"14"}                                          
{"time":"2026-02-17T08:09:14.334769Z","level":"debug","scope":"envoy upstream","msg":"removing cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:822","thread":"14"}                                       
{"time":"2026-02-17T08:09:14.334793Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"14"}                                   
{"time":"2026-02-17T08:09:14.334845Z","level":"debug","scope":"envoy upstream","msg":"cds: remove cluster 'outbound|80||deployment-svc.kuberhealthy.svc.cluster.local'","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:76","thread":"14"}                                         
{"time":"2026-02-17T08:09:14.334847Z","level":"info","scope":"envoy upstream","msg":"cds: added/updated 0 cluster(s) (skipped 1 unmodified cluster(s)); removed 1 cluster(s)","caller":"external/envoy/source/common/upstream/cds_api_helper.cc:84","thread":"14"}                                   
{"time":"2026-02-17T08:09:14.334809Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"23"}                                   
{"time":"2026-02-17T08:09:14.334819Z","level":"debug","scope":"envoy upstream","msg":"removing TLS cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local","caller":"external/envoy/source/common/upstream/cluster_manager_impl.cc:826","thread":"22"}                                   
{"time":"2026-02-17T08:09:14.335251Z","level":"debug","scope":"envoy config","msg":"Delta config for type.googleapis.com/envoy.config.cluster.v3.Cluster accepted with 1 resources added, 1 removed","caller":"external/envoy/source/extensions/config_subscription/grpc/delta_subscription_state.cc:
{"time":"2026-02-17T08:09:14.335755Z","level":"critical","scope":"envoy backtrace","msg":"Caught Segmentation fault, suspect faulting address 0x0","caller":"external/envoy/source/server/backtrace.h:129","thread":"14"}                                                                            
{"time":"2026-02-17T08:09:14.335804Z","level":"critical","scope":"envoy backtrace","msg":"Backtrace (use tools/stack_decode.py to get line numbers):","caller":"external/envoy/source/server/backtrace.h:113","thread":"14"}                                                                         
{"time":"2026-02-17T08:09:14.335821Z","level":"critical","scope":"envoy backtrace","msg":"Envoy version: 1fdadc301aaaa0326255f82bd27387a0e101c164/1.37.1-dev/Clean/RELEASE/BoringSSL","caller":"external/envoy/source/server/backtrace.h:114","thread":"14"}                                         
{"time":"2026-02-17T08:09:14.335823Z","level":"critical","scope":"envoy backtrace","msg":"Address mapping: 5594af52f000-5594b2b70000 /usr/local/bin/envoy","caller":"external/envoy/source/server/backtrace.h:116","thread":"14"}                                                                    
{"time":"2026-02-17T08:09:14.373606Z","level":"critical","scope":"envoy backtrace","msg":"#0: [0x7f7acffa1330]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                                                                                               
{"time":"2026-02-17T08:09:14.376456Z","level":"critical","scope":"envoy backtrace","msg":"#1: event_process_active_single_queue [0x5594b254d6ac]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                             
{"time":"2026-02-17T08:09:14.379109Z","level":"critical","scope":"envoy backtrace","msg":"#2: event_base_loop [0x5594b254c421]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                               
{"time":"2026-02-17T08:09:14.381450Z","level":"critical","scope":"envoy backtrace","msg":"#3: Envoy::Server::InstanceBase::run() [0x5594b18effb7]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                            
{"time":"2026-02-17T08:09:14.383498Z","level":"critical","scope":"envoy backtrace","msg":"#4: Envoy::MainCommonBase::run() [0x5594b18965aa]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                  
{"time":"2026-02-17T08:09:14.385486Z","level":"critical","scope":"envoy backtrace","msg":"#5: Envoy::MainCommon::main() [0x5594b1896ab4]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                     
{"time":"2026-02-17T08:09:14.387356Z","level":"critical","scope":"envoy backtrace","msg":"#6: main [0x5594af5c9f1c]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                                          
{"time":"2026-02-17T08:09:14.387413Z","level":"critical","scope":"envoy backtrace","msg":"#7: [0x7f7acff861ca]","caller":"external/envoy/source/server/backtrace.h:123","thread":"14"}                                                                                                               
{"time":"2026-02-17T08:09:14.387433Z","level":"critical","scope":"envoy backtrace","msg":"#8: __libc_start_main [0x7f7acff8628b]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                             
{"time":"2026-02-17T08:09:14.389404Z","level":"critical","scope":"envoy backtrace","msg":"#9: _start [0x5594af52f02a]","caller":"external/envoy/source/server/backtrace.h:121","thread":"14"}                                                                                                        
{"level":"info","time":"2026-02-17T08:09:14.396641Z","scope":"ads","msg":"ADS: \"\" 2 terminated"}                                                                                                                                                                                                   
{"level":"error","time":"2026-02-17T08:09:14.397001Z","msg":"Envoy exited with error: signal: segmentation fault (core dumped)"}                                                                                                                                                                     
{"level":"info","time":"2026-02-17T08:09:14.397027Z","scope":"ads","msg":"ADS: \"\" 1 terminated"}                                                                                                                                                                                                   
{"level":"info","time":"2026-02-17T08:09:14.397031Z","scope":"ads","msg":"ADS: \"\" 3 terminated"} 

So it seems it's the configuration change / removal of cluster outbound|80||deployment-svc.kuberhealthy.svc.cluster.local which led up to the crash / segfault.

[rroblak]

@frittentheke Please see #43526.