7.2.0 release documentation #1014
+497
−4
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,209 @@ | ||
| --- | ||
| title: Path MTU Discovery Enhancements | ||
|
Contributor
There was a problem hiding this comment. Since there does not exist a document for Path MTU for the SSR, I would suggest calling this document Path MTU Discover, and leave off the Enhancements. We should use this document to describe overall PMTU behavior, including the enhancements. The only area in our docs that covers PMTU is concepts_machine_communication. I would leave that content there, but provide a link to this new document. It might be worthwhile to duplicate some or all of the other content here. |
||
| sidebar_label: Path MTU Discovery Enhancements | ||
| --- | ||
|
|
||
| #### Version History | ||
|
|
||
| | Release | Modification | | ||
| | ------- | ------------ | | ||
| | 7.2.0 | Feature introduced | | ||
|
Contributor
There was a problem hiding this comment. Untrue. PMTU has been around for a long time. Perhaps from the beginning? Might be better off removing this version history table. |
||
|
|
||
| ## Overview | ||
|
|
||
| The SSR performs Path MTU Discovery (PMTUD) along the overlay to determine the correct maximum transmission unit (MTU) for each peer path. By default, this test runs every ten minutes. If a change in the underlay reduces the available path MTU between two SSRs, the new value is not discovered until the next PMTUD cycle. Additionally, existing sessions continue to use the previous MTU value until the next time those sessions are rebuilt. | ||
|
|
||
| Devices in the underlay may report an ICMP Destination Unreachable / Fragmentation Needed (type 3, code 4) error, referred to here as a _TooBig_ packet, to indicate they could not forward a packet due to an undersized MTU. Prior to SSR 7.2.0, these messages were forwarded to the correct endpoint, but the SSR itself did not act on the MTU value contained in the message, leaving existing sessions with an incorrect PMTU. | ||
|
|
||
| SSR 7.2.0 introduces two complementary enhancements to address these gaps: | ||
|
|
||
| 1. **Underlay ICMP reaction** — When the SSR receives a TooBig packet from the underlay, it updates the affected overlay flow and generates a corrected TooBig packet toward the original packet sender, allowing the sender to adjust its segment size. | ||
|
|
||
| 2. **Session Refresh** - The flow which was traversed to trigger the TooBig response from the underlay is now updated to use the MTU reported in the TooBig packet. | ||
|
|
||
| For TCP flows, setting `enforced-mss automatic` on the egress `network-interface` is the recommended complement to these features. It adjusts the TCP MSS advertised at the interface boundary to avoid fragmentation in the first place. See [Configuration](#configuration) for details. | ||
|
|
||
| ## How The SSR Reacts to Underlay ICMP TooBig Messages | ||
|
|
||
| The following sequence illustrates what happens when the underlay path MTU changes after a session is already established. | ||
|
|
||
| ### Initial State | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant Client | ||
| participant Hub as Hub SSR | ||
| participant R1 as Spoke SSR | ||
| participant Server | ||
|
|
||
| Client->>Hub: Data (MTU 1500) | ||
| Hub->>R1: SVR overlay packet (MTU 1500) | ||
| R1->>Server: Data (MTU 1500) | ||
| Note over Hub,R1: Underlay MTU = 1500. Session PMTU on both SSRs = 1500. | ||
| ``` | ||
|
|
||
| The client and server are communicating through two peering SSRs over the overlay. The PMTU is consistent at 1500 across all hops, and both SSRs have applied an MTU of 1500 to the forward flow actions for this session. | ||
|
|
||
| ### Underlay MTU Drops — First TooBig Received by Hub | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant Client | ||
| participant Hub as Hub SSR | ||
| participant R2 as Underlay Device | ||
| participant R1 as Spoke SSR | ||
| participant Server | ||
|
|
||
| Note over R2,R1: Underlay MTU between Hub and R2 drops to 1300 | ||
| Hub->>R2: SVR packet hub-WAN to spoke-WAN (over 1300 bytes) | ||
| R2-->>Hub: ICMP TooBig type 3 code 4, reported MTU = 1300 | ||
| Note over Hub: DivertedPacketHandler finds reverse flow. Updates Hub-to-Spoke PMTU to 1300. | ||
|
There was a problem hiding this comment. We should emphasis that this only affects this particular flow. A different session using this same path does not realize the PMTU reported to this session. It also does not persist, meaning if the session is modified for some other reason (FIB update, device flap, etc), the egress MTU is 'reset' back to the latest at that time based on the most recent BFD-based PMTU detection test, or interface MTU if no such test has been done |
||
| Hub-->>Server: New TooBig toward Server | ||
| Note over Server: Server adjusts MSS if TCP-capable | ||
| ``` | ||
|
|
||
| When R2 (an underlay device) cannot forward an oversized packet, it sends a TooBig packet to the Hub's WAN interface. The SSR processes this message and does the following: | ||
|
|
||
| 1. It extracts the encapsulated IP header from the TooBig body to identify the affected overlay session. | ||
| 2. It finds the reverse flow using that header and updates the Hub → Spoke forward flow's PMTU to the value reported by the underlay. | ||
|
There was a problem hiding this comment. Pedantic, but it doesnt have to be a reverse flow. It's whatever flow was traversed to elicit the TooBig response. In the context of a TCP download used throughout this doc, it would likely be a reverse flow |
||
| 3. It constructs a new TooBig packet directed toward the original packet sender (the Server), so the server's TCP stack can reduce its MSS. | ||
|
|
||
| :::note | ||
| The MTU value propagated in the new TooBig packet reflects the underlay-reported value. On paths with encryption, HMAC, FEC, or BFD tunneling overhead, the effective usable MTU will be lower than the raw underlay value. The SSR accounts for these overheads when setting the MTU on forward flow actions. | ||
| ::: | ||
|
|
||
| ## Fabric Fragmentation and Oversize Packet Behavior | ||
|
|
||
| When the PMTU on an overlay (SVR/fabric) path is lower than the MTU of the segment immediately preceding the Hub, packets larger than the PMTU will require fragmentation along the overlay. The SSR always fragments fabric packets when necessary, even when the incoming packet carries the Don't Fragment (DF) bit. This preserves packet delivery but prevents the sender from learning about the smaller path MTU and adjusting its segment size. | ||
|
There was a problem hiding this comment. IPv6 traffic also is supposed to never fragment at a router, but like the IPv4 with dont-frag, we do anyways when its an SVR path |
||
|
|
||
| :::note | ||
| For TCP traffic, setting `enforced-mss automatic` on the egress `network-interface` is the most reliable way to avoid this scenario. When set, the SSR rewrites the TCP MSS at the interface boundary to match the session MTU (including the path MTU for SVR sessions). This is commonly known as `MSS Clamping` and is not the default; it must be explicitly configured. | ||
| ::: | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Enabling Oversize Fabric Packet Behavior | ||
|
|
||
| #### On a `network-interface` | ||
|
|
||
| ``` | ||
| config | ||
| authority | ||
| router <router-name> | ||
| node <node-name> | ||
| device-interface <device-interface-name> | ||
| network-interface <network-interface-name> | ||
| oversize-fabric-packet-behavior true | ||
|
Contributor
There was a problem hiding this comment. is |
||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| ``` | ||
|
|
||
| #### On a `service-policy` | ||
|
|
||
| ``` | ||
| config | ||
| authority | ||
| service-policy <policy-name> | ||
| oversize-fabric-packet-behavior true | ||
| exit | ||
| exit | ||
| exit | ||
| ``` | ||
|
|
||
| ### Configuring `enforced-mss` (Recommended for TCP) | ||
|
|
||
| Set `enforced-mss` to `automatic` on egress interfaces to avoid fabric fragmentation for TCP traffic. The SSR calculates the correct MSS from the interface or path MTU for SVR sessions. | ||
|
|
||
| ``` | ||
| config | ||
| authority | ||
| router <router-name> | ||
| node <node-name> | ||
| device-interface <device-interface-name> | ||
| network-interface <network-interface-name> | ||
| enforced-mss automatic | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| ``` | ||
|
|
||
| ### Configuring PMTUD Interval | ||
|
|
||
| The PMTUD interval (how frequently the SSR probes each overlay path) is configurable at the router level and can be overridden per neighborhood or per adjacency. | ||
|
|
||
| ``` | ||
| config | ||
| authority | ||
| router <router-name> | ||
| path-mtu-discovery | ||
| enabled true | ||
| interval 600 | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| ``` | ||
|
|
||
| | Field | Default | Description | | ||
| | ----- | ------- | ----------- | | ||
| | `enabled` | `true` | Enables or disables PMTUD for this router. | | ||
| | `interval` | `600` | Seconds between PMTUD tests. Valid range: 1–86400. | | ||
|
|
||
| To override the interval for a specific adjacency: | ||
|
|
||
| ``` | ||
| config | ||
| authority | ||
| router <router-name> | ||
| node <node-name> | ||
| device-interface <device-interface-name> | ||
| network-interface <network-interface-name> | ||
| adjacency <ip-address> | ||
| path-mtu-discovery | ||
| enabled true | ||
| interval 300 | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| exit | ||
| ``` | ||
|
|
||
| ## Verification | ||
|
|
||
| Use `show peers` to confirm the currently discovered path MTU for each peer path: | ||
|
There was a problem hiding this comment. Nothing to correct here, just clarifying that this IS correct today. The I dont think there's visibility into that from a show command perspective. A new stat, |
||
|
|
||
| ```text | ||
| admin@node1.router1# show peers | ||
| Peer Node Network Interface Destination Status Hostname Path MTU | ||
| ------------------------ --------- ------------------- --------------- -------- ------------ ---------- | ||
| router2 node1 wan0 192.0.2.10 Up router2.lab 1300 | ||
| ``` | ||
|
|
||
| A `Path MTU` value of `0` indicates PMTUD is disabled or has not yet completed a test cycle. | ||
|
|
||
| A new stat, `stats/icmp/flow-mtu-updates`, provides a count of flows that have had their MTU updated at runtime via a TooBig packet. This counter is reset when the system resets (not persisted). | ||
|
|
||
| **need stat example | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - If the path MTU shown by `show peers` does not reflect the expected value, verify that `path-mtu-discovery > enabled` is `true` on both sides of the adjacency. | ||
| - If TCP sessions continue to fragment after configuring `enforced-mss automatic`, confirm the setting is applied to the correct egress interface and that both peers have completed a PMTUD cycle. | ||
|
|
||
| ## Related Topics | ||
|
|
||
| - [Concepts: Machine to Machine Communication](concepts_machine_communication.md) — path MTU discovery protocol details and BFD traffic patterns. | ||
| - [Configuration Reference Guide](config_reference_guide.md) — full parameter reference for `path-mtu-discovery`, `enforced-mss`, and `session-resiliency`. | ||
| - [Configuring Session Recovery Detection](config_session_recovery.md) — session health-check and flow rebuild mechanisms. | ||
| - [Configuring Forward Error Correction](config_forward_error_correction.md) — complementary resiliency feature for packet loss. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please provide at least a CLI output for IPv4