OCPBUGS-77773: fix backend server health check for DCM enabled

[jcmoraisjr]

Without DCM, router configure single replica backends without health check. This saves a bit of cpu and network io, since a single failing replica failing health check will continue to disrupt the service. This works without DCM because whenever a scaling out happens, haproxy is reloaded with new configurations enabling health check on all the replicas, including the first one.

This is not working with DCM because the code simply add the new replica on an empty slot, ignoring the status of the other ones. In the end the new replica has health check enabled and the first one continues with health check disabled.

Options to change this behavior includes adding api calls to enable the first replica, or instead always enable it by default if DCM is enabled. The difficulty with the former is the amount of changes needed to make in the codebase: identify first replica without health check, del+add api calls we don't use (yet!!). The later seems a natural path since it is only for DCM enabled, and changes behavior only on single replica deployments, which should be the exception. Moreover, the benefit of having health check enabled in a simple way far outweight the small increase on cpu and network.

That said there are two distinct changes happening in this PR:

• Removing the criteria for active endpoint, now health check is always enabled if DCM is enabled;
• Adding inter keyword in the server-template, so the dynamically added server will use the same health check interval of the other replicas.

https://issues.redhat.com/browse/OCPBUGS-77773

[openshift-ci-robot]

@jcmoraisjr: This pull request references Jira Issue OCPBUGS-77773, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @ShudiLi

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Without DCM, router configure single replica backends without health check. This saves a bit of cpu and network io, since a single failing replica failing health check will continue to disrupt the service. This works without DCM because whenever a scaling out happens, haproxy is reloaded with new configurations enabling health check on all the replicas, including the first one.

This is not working with DCM because the code simply add the new replica on an empty slot, ignoring the status of the other ones. In the end the new replica has health check enabled and the first one continues with health check disabled.

Options to change this behavior includes adding api calls to enable the first replica, or instead always enable it by default if DCM is enabled. The difficulty with the former is the amount of changes needed to make in the codebase: identify first replica without health check, del+add api calls we don't use (yet!!). The later seems a natural path since it is only for DCM enabled, and changes behavior only on single replica deployments, which should be the exception. Moreover, the benefit of having health check enabled in a simple way far outweight the small increase on cpu and network.

That said there are two distinct changes happening in this PR:

• Removing the criteria for active endpoint, now health check is always enabled if DCM is enabled;
• Adding inter keyword in the server-template, so the dynamically added server will use the same health check interval of the other replicas.

https://issues.redhat.com/browse/OCPBUGS-77773

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rfredette for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jcmoraisjr]

/test e2e-aws-serial-2of2

[openshift-ci]

@jcmoraisjr: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[jcmoraisjr]

/close

Closing on behalf of #747

[openshift-ci]

@jcmoraisjr: Closed this PR.

Details

In response to this:

/close

Closing on behalf of #747

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@jcmoraisjr: This pull request references Jira Issue OCPBUGS-77773. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Without DCM, router configure single replica backends without health check. This saves a bit of cpu and network io, since a single failing replica failing health check will continue to disrupt the service. This works without DCM because whenever a scaling out happens, haproxy is reloaded with new configurations enabling health check on all the replicas, including the first one.

This is not working with DCM because the code simply add the new replica on an empty slot, ignoring the status of the other ones. In the end the new replica has health check enabled and the first one continues with health check disabled.

Options to change this behavior includes adding api calls to enable the first replica, or instead always enable it by default if DCM is enabled. The difficulty with the former is the amount of changes needed to make in the codebase: identify first replica without health check, del+add api calls we don't use (yet!!). The later seems a natural path since it is only for DCM enabled, and changes behavior only on single replica deployments, which should be the exception. Moreover, the benefit of having health check enabled in a simple way far outweight the small increase on cpu and network.

That said there are two distinct changes happening in this PR:

• Removing the criteria for active endpoint, now health check is always enabled if DCM is enabled;
• Adding inter keyword in the server-template, so the dynamically added server will use the same health check interval of the other replicas.

https://issues.redhat.com/browse/OCPBUGS-77773

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.