docs: PrivateLink troubleshooting for NLB security group enforcement#36623
Open
materializekatrina wants to merge 1 commit into
Open
docs: PrivateLink troubleshooting for NLB security group enforcement#36623materializekatrina wants to merge 1 commit into
materializekatrina wants to merge 1 commit into
Conversation
kay-kim
reviewed
May 19, 2026
| **Remarks**: | ||
|
|
||
| a. Network Load Balancers do not have associated security groups. Therefore, the security groups for your targets must use IP addresses to allow traffic. | ||
| a. Network Load Balancers do not have associated security groups by default. Therefore, the security groups for your targets must use IP addresses to allow traffic. |
Contributor
There was a problem hiding this comment.
Double-checking ... this is pre-change content ... but since we're here:
The "Therefore, " sentence , is this the same or different as point b) below?
Will also ask Justin.
|
|
||
| b. You can't use the security groups for the clients as a source in the security groups for the targets. Therefore, the security groups for your targets must use the IP addresses of the clients to allow traffic. For more details, check the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html). | ||
|
|
||
| c. If you have associated a [security group with your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-security-groups.html) and enabled **Enforce inbound rules on PrivateLink traffic**, the NLB's inbound rules will be applied to traffic arriving from Materialize's VPC endpoint. In that case, the source IPs are private IPs from Materialize's VPC — **not** the IPs of any client (e.g., `kcat` or a Kafka client running from a workstation or bastion host) that you might use to verify connectivity to the brokers directly. Inbound rules that only permit your own test traffic will silently block Materialize, even when your own connectivity tests succeed. To resolve this, either disable **Enforce inbound rules on PrivateLink traffic**, or add inbound rules to the NLB's security group that permit the relevant ports (each broker's listener port and the health check port) from a source that covers Materialize's VPC endpoint traffic. |
Contributor
There was a problem hiding this comment.
So ... we seem to be modifying content shown on
https://preview.materialize.com/materialize/36623/ingest-data/network-security/privatelink/
Did we want to update the content in https://preview.materialize.com/materialize/36623/ingest-data/postgres/amazon-rds/#b-optional-configure-network-security page and such?
|
|
||
| b. You can't use the security groups for the clients as a source in the security groups for the targets. Therefore, the security groups for your targets must use the IP addresses of the clients to allow traffic. For more details, check the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html). | ||
|
|
||
| c. If you have associated a [security group with your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-security-groups.html) and enabled **Enforce inbound rules on PrivateLink traffic**, the NLB's inbound rules will be applied to traffic arriving from Materialize's VPC endpoint. In that case, the source IPs are private IPs from Materialize's VPC — **not** the IPs of any client (e.g., `kcat` or a Kafka client running from a workstation or bastion host) that you might use to verify connectivity to the brokers directly. Inbound rules that only permit your own test traffic will silently block Materialize, even when your own connectivity tests succeed. To resolve this, either disable **Enforce inbound rules on PrivateLink traffic**, or add inbound rules to the NLB's security group that permit the relevant ports (each broker's listener port and the health check port) from a source that covers Materialize's VPC endpoint traffic. |
Contributor
There was a problem hiding this comment.
Also, maybe make it leaner? And perhaps list the disabling to be the second bullet point(?) since it seems kind of permissive for all inbound traffic and not just materialize's traffic.
If you have associated a security group with your Network Load Balancer and
enabled **Enforce inbound rules on PrivateLink traffic**, the security group's
inbound rules also apply to traffic from Materialize's VPC endpoint. Any traffic
not explicitly permitted, including Materialize's, will be silently blocked.
To resolve this either:
- Add inbound rules to the NLB's security group that permit the listener port
and the health check port from a source covering Materialize's VPC endpoint
traffic, or
- Disable **Enforce inbound rules on PrivateLink traffic**.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Customer hit a silent PrivateLink failure:
psqlfrom their bastion worked, but Materialize source creation failed. Their NLB security group had Enforce inbound rules on PrivateLink traffic enabled with rules that only allowed their own test IPs — traffic from Materialize's VPC endpoint arrives with different source IPs, so it was silently dropped.Our docs don't mention this, and still claim "NLBs do not have associated security groups" (no longer true since AWS added the feature in 2023).
Slack thread: https://materializeinc.slack.com/archives/C08K5GHRV1S/p1779207642559229
Changes
In each of the three PrivateLink setup shortcodes (PostgreSQL, MySQL, Kafka):