upstream ext_proc: option to wait for connection establishment

[guydc]

Title: upstream ext-proc: option to wait for connection establishment

Description:
Upstream HTTP filters are not guaranteed to execute after upstream connection establishment: #10455 (comment).

Upstream HTTP filters may require information about the upstream peer (such as Peer Cert Subject/IP) and application-layer data from the request, to determine if a request may be proxied or blocked, especially in DFP use cases.

Ext-Proc filters can expose an option to wait for upstream connection establishment before sending headers and attributes to the ext-proc server, to ensure that this info is available.

[guydc]

cc @yanjunxiang-google

[yanjunxiang-google]

Could you please describe the use cases in more detail about why this option is needed? @guydc

[yanjunxiang-google]

Adding more ext_proc code owners @tyxia @yanavlasov

[guydc]

Hi @yanjunxiang-google !

One use case is extensibility for upstream Network/TLS validations. For example:

• Verifying non-standard custom certificate organizational extensions
• Performing standard validations not covered by Envoy/BoringSSL by design, like OCSP and Dynamic CRL revocation checks, etc.
• Implementing outbound IP deny/allow lists.

Typically, this would be solved with network filters/transport sockets. However, currently, cluster network filters don't seem to support common extensibility options like WASM, Dynamic Modules or Ext-Proc and aren't commonly implemented in general.

It should be noted that a network filter needs to be registered in code as an upstream network filter before usage. Currently, there are no upstream network filters available in Envoy out of the box.

Perhaps supporting a dual network ext-proc filter could be a long term option here.

Upstream HTTP filters do support this form of extensibility, and also have the added benefit of being able to perform such validations post connection establishment (which could be relevant for enforcing policy changes on long-lived connections). However, they lack:

• A guarantee that logic would execute only after connection was established, ensuring all traffic is subject to validations.
• An explicit way to destroy connections that fail validation.

Another use case is implementing validations that require both application-level info (e.g. user identity) and upstream network info (e.g. list of IP addresses and that this user is allowed to access), especially when using Envoy as a forward proxy.

I believe that addressing the connection context issue should be possible, based on the comments in the linked discussion. There's a sketch here: cb4ee20 that I'm willing to work on, if the use case is found to be relevant.

Some related issues that can be solved with a connection-aware ext-proc:

• Support for upstream network filter #21178
• TLS certificate validation: how to asynchronously fetch CRL using CRL DP present in cert #19142

[yanjunxiang-google]

Thanks for the info @guydc Checking your prototyping linked above, it looks like you put the ext_proc/upstrea-filter-chain-execution on hold and wait for the upstream connection establishment finished then continue the upstream filter chain execution.
A couple of questions: 1) can we add an Envoy config option to make the upstream HTTP filter chain be executed after the upstream connection established? and 2) Can we do this in a generic place, like router filter, and not hooked into a specific HTTP filter?

[guydc]

Thanks @yanjunxiang-google. I see your point about making this a generic capability of the router, allowing other filters to benefit from upstream connection context.
I'll look into the option the you mentioned. @yanavlasov - as a code owner for the router, does that make sense for you?

[tyxia]

Please note router config is not only place where upstream filter chain can be defined. Upstream filter chain can also be created at Cluster creation time, as part of HttpProtocolOptions config.

From a quick scan, generalizing this feature to benefit all upstream filter makes sense to me. And having it in common place also SGTM. If it is hard to find the correct common place, we probably can also consider exposing the common API to let each filter stop, which can also achieve the purpose of generalization.