Failed to do authentication for a microservice with Istio Service Mesh when trying to reach Azure Key Vault

[zgeorgijevacc1]

Description of the environment:
We have a microservices running in Azure Kubernetes Services (AKS), which communicates with Azure Key Vault to obtain certain secrets from there. We use Azure Spring plugin to reach Azure Key Vault. For authentication we use an Azure service principal. Everything was work smoothly.
Description of the issue:
Recently we added Istio Service Mesh and implemented a Control Egress Traffic Istio for Azure Key Vault:
_apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: azurekeyvault
spec:
hosts:

• azurevaultxxxxx.vault.azure.net
  ports:
• number: 443
  name: https
  protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: azurekeyvault
spec:
hosts:

• azurevaultxxxxx.vault.azure.net
  tls:
• match:
  • port: 443
    sni_hosts:
    • azurevaultxxxxx.vault.azure.net
      route:
  • destination:
    host: azurevaultxxxxx.vault.azure.net
    port:
    number: 443
    weight: 100_

Now, we are getting an exception ("Failed to do authentication") and here is the log:
java.lang.IllegalStateException: Failed to configure KeyVault property source
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:72) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:22) ~[azure-spring-boot-2.0.6.jar!/:na]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:184) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:170) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:358) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.3.RELEASE.jar!/:2.0.3.RELEASE]
at foa3k.configserver.ConfigServerApplication.main(ConfigServerApplication.java:12) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]
Caused by: java.lang.IllegalStateException: Failed to do authentication.
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:48) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.getAuthenticationCredentials(KeyVaultCredentials.java:113) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials.access$100(KeyVaultCredentials.java:27) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$2.authenticate(KeyVaultCredentials.java:81) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.followUpRequest(RetryAndFollowUpInterceptor.java:284) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:152) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.azure.keyvault.authentication.KeyVaultCredentials$1.intercept(KeyVaultCredentials.java:59) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[client-runtime-1.6.0.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[okhttp-3.8.1.jar!/:na]
at okhttp3.RealCall.execute(RealCall.java:69) ~[okhttp-3.8.1.jar!/:na]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[retrofit-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40) ~[adapter-rxjava-2.4.0.jar!/:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[adapter-rxjava-2.4.0.jar!/:na]
at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10247) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.Observable.subscribe(Observable.java:10214) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[rxjava-1.2.0.jar!/:1.2.0]
at rx.observables.BlockingObservable.single(BlockingObservable.java:341) ~[rxjava-1.2.0.jar!/:1.2.0]
at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecrets(KeyVaultClientImpl.java:2951) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.KeyVaultClient.listSecrets(KeyVaultClient.java:911) ~[azure-keyvault-1.0.0.jar!/:1.0.0]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.createOrUpdateHashMap(KeyVaultOperation.java:88) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultOperation.(KeyVaultOperation.java:43) ~[azure-spring-boot-2.0.6.jar!/:na]
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:62) ~[azure-spring-boot-2.0.6.jar!/:na]
... 22 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_181]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_181]
at com.microsoft.azure.keyvault.spring.AzureKeyVaultCredential.doAuthenticate(AzureKeyVaultCredential.java:45) ~[azure-spring-boot-2.0.6.jar!/:na]
... 68 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[na:1.8.0_181]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_181]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[na:1.8.0_181]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[na:1.8.0_181]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) ~[na:1.8.0_181]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) ~[na:1.8.0_181]
at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:148) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:86) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:85) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:880) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:68) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:176) ~[adal4j-1.3.0.jar!/:1.3.0]
at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:165) ~[adal4j-1.3.0.jar!/:1.3.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]

Based on our analysis of the source code available from GitHub, it appears that very likely failure happens during attempt to authenticate HTTP Istio proxy (see RetryAndFollowUpInterceptor.java:284), not microservice itself.
So, we are really trying to get clarity why Istio proxy is not properly handled here.
Also, how Azure Key Vault treats proxies and how they should be configured when client behind them uses an Azure service principal for authentication.
For clarity we tried other non Azure Key Vault related http calls and they are going smoothly (without any problems). We have problem only when our microservice tries to reach Azure Key Vault by using Azure Spring plugin.

[blackstorm]

same error

java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by: java.lang.IllegalStateException: Failed to configure KeyVault property source
Caused by: java.lang.IllegalStateException: Failed to do authentication.
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target