MTU and Jumbo Frames

Chapter 5.6 derived the 64-octet floor from a round-trip propagation time and showed that every term in the derivation was a physical measurement. The ceiling has no such derivation.

1500 octets is not a distance, not a time, and not a consequence of any other number in the frame. It was chosen — an engineering compromise about buffer memory, error-recovery cost and per-frame overhead made when memory was expensive and links were slow, and then frozen because everything on every network agreed on it.

Which makes the ceiling a fundamentally different kind of constraint from the floor, and the difference is the whole chapter.

A floor derived from physics is the same everywhere, needs no negotiation, and cannot be locally relaxed. A ceiling that is a convention can be raised — by agreement. And the agreement is not between two devices, or between a device and its driver. It is among every device on the path a frame will take, none of which know about each other.

What happens when that agreement is incomplete is the reason this chapter exists, and it is not an error report. It is a black hole: small frames pass, large frames disappear, and no counter on any device on the path records anything wrong.

1. Scope — What This Chapter Owns

Chapter 5.6 owns the floor and the padding that enforces it. It established the wire-occupancy accounting this chapter reuses, and the argument that per-frame work rather than bandwidth is the binding constraint.

This chapter owns the ceiling: where 1500 came from and why it is a different kind of number, the three standardised maxima and what distinguishes them, what raising the ceiling actually buys measured honestly, why the agreement's scope is a path rather than a device, and why a mismatch presents as silence rather than as an error.

It does not own: the check sequence appended after the payload, which is Chapter 5.8 — although Section 11 has to note that a longer frame changes what that check can promise. Nor the interframe gap, which is Chapter 5.9. Nor the buffering that a larger frame requires at each hop, which is a switching-architecture subject.

And it does not own path MTU discovery, which is a protocol above this layer. This chapter owns what a MAC can observe about the problem — which is less than the protocol above can, and available sooner.

The question this chapter answers that its neighbours do not: what exactly is being agreed when a network is configured for large frames, who are the parties, and what does a partial agreement look like from inside a device that is behaving correctly?

2. Where 1500 Came From, and Why It Is a Different Kind of Number

Run the same exercise Chapter 5.6 §2 ran on the floor, and notice that it fails.

There is no distance to start from. No propagation delay, no segment length, no signalling-rate conversion. Nothing about the physical medium sets an upper bound on how long a transmission may be.

What there was instead is a set of competing costs, each pulling in a different direction:

Receive buffering. A receiver must be able to hold a frame. At the time the number was chosen, memory was the dominant cost in an interface, and a ceiling on frame size is directly a ceiling on the buffer every station must provide.

Error recovery. A corrupted frame is discarded whole. The larger the frame, the more work is lost per error, and the higher the retransmission cost at a given error rate.

Fairness on a shared medium. Chapter 1.1's medium was shared, and a station transmitting a very long frame holds it for the duration. A ceiling bounds how long any one station can make the others wait.

Against those, the pressure upward: every frame carries fixed overhead — preamble, header, check sequence, gap — and the larger the payload, the smaller that overhead's share.

1500 is where those met, for the hardware of the time. Every one of the three downward pressures has weakened enormously since: memory is cheap, links are reliable enough that whole-frame loss is rare, and the medium is no longer shared. The number did not move.

3. Three Standard Maxima, and What the Extra Room Is For

Read the middle column as the actual design intent. None of the three increases what the client may send. The payload ceiling is 1500 throughout. What grows is space for octets that belong to somebody else — a tag inserted by a bridge, or prefixes and suffixes added by an encapsulation running over Ethernet.

That is the point of the envelope frame and it is routinely misread. Its 482 extra octets are not a bigger MTU. They exist so that a network carrying encapsulated traffic can add its headers without forcing the encapsulated payload below 1500 — which would otherwise mean every station inside the encapsulation had to be reconfigured to a smaller MTU, which is exactly the coordination problem this chapter is about.

And then the fourth box, which is not on the same footing as the other three. The commonly deployed 9000-octet "jumbo" frame is not standardised by IEEE 802.3. It is a widely honoured convention among vendors, and the specific value varies between implementations — some support 9000, some 9216, some other figures, and the differences are exactly large enough to produce a mismatch that only appears on the largest frames a workload happens to generate.

So the honest statement is that a network running jumbo frames is running on an agreement with no normative text behind it. That is workable — a great many high-performance networks do it — and it means the agreement has to be verified rather than assumed, which is what Section 12's detector is for.

4. What a Larger Frame Actually Buys

The claim usually made for jumbo frames is efficiency. The claim is true and much smaller than people expect, and the number that matters is a different one.

Wire efficiency, computed the way Chapter 5.6 §11 computed it — counting preamble, header, check sequence and interframe gap as the cost of carrying a payload:

Illustrative arithmetic, and read the last two rows against each other. Going from a standard frame to a six-times-larger one improves wire efficiency by about two percentage points. That is a real gain and it is not why anyone deploys jumbo frames.

The gain that matters is the frame rate. Carrying a fixed amount of data at 9000 octets per frame instead of 1500 takes one sixth as many frames — and every per-frame cost falls with it: address lookups, filter decisions, descriptor writes, interrupts, buffer allocations, and every software touch on the receive path.

So the honest summary is a ratio, not a percentage: jumbo frames buy roughly 2% more bandwidth and 6× less per-frame work, and it is the second number that decides whether they are worth the coordination.

And the costs are worth stating in the same units, because they are not symmetric with the benefits.

Buffering grows at every hop, not only at the endpoints. A device that accepts a 9000-octet frame must be able to hold one, per port, per direction, and store-and-forward devices must hold the whole frame before making a forwarding decision. That memory is provisioned in silicon and cannot be reconfigured, which is why Section 5's build-time check compares the configured ceiling against the buffer rather than trusting the configuration.

Latency for small traffic rises when large frames share the link. A 9000-octet frame occupies the wire six times longer than a 1500-octet one, and a small frame arriving behind it waits. This is Chapter 1.1's fairness argument reappearing on a link that is no longer shared — the medium is not contended, but the transmit queue is.

And the work lost per discarded frame rises in direct proportion. A corrupted frame is discarded whole, so at a fixed error rate the octets lost per unit time scale with frame size. On modern links the error rate is low enough that this is negligible; it was not negligible when 1500 was chosen, and it is the term that has changed most.

Read those three against the two benefits and the shape of the decision appears: the benefit is concentrated in per-frame work, the costs are concentrated in memory and in latency for other traffic, and none of them is a bandwidth argument on either side.

5. RTL 1 — Classifying a Frame Against a Ceiling That Is Local

// SYNTHESIZABLE.
//
// Classifies an arriving frame against three DIFFERENT limits, which are
// routinely collapsed into one and must not be:
//
//   STANDARD CEILING -- 1518 / 1522 / 2000, from the frame format. A frame
//                       above this is non-standard, whatever this device
//                       is configured to accept.
//
//   LOCAL CEILING    -- what THIS device is configured and buffered for.
//                       May be far above the standard (jumbo) or, on a
//                       misconfigured device, far below its neighbours'.
//
//   BUFFER CAPACITY  -- what this device can physically hold. A local
//                       ceiling above it is a configuration that cannot be
//                       honoured, and the failure is a partial frame
//                       rather than a rejection.
//
// The three coincide on a default device, which is why the collapse
// survives testing and fails in a jumbo deployment.
package frame_max_pkg;
 
  // NORMATIVE. Maximum basic frame, including the check sequence and
  // excluding preamble, delimiter and gap.
  localparam int unsigned MAX_BASIC    = 1518;
  // NORMATIVE. One tag.
  localparam int unsigned MAX_QTAGGED  = 1522;
  // NORMATIVE. Envelope frame: 482 octets above the basic frame, for
  // encapsulation prefixes and suffixes.
  localparam int unsigned MAX_ENVELOPE = 2000;
 
  typedef enum logic [2:0] {
    SZ_RUNT,        // below the floor -- Chapter 5.6 owns this
    SZ_NORMAL,      // within the basic frame
    SZ_TAGGED,      // above basic, within Q-tagged
    SZ_ENVELOPE,    // above Q-tagged, within envelope
    SZ_JUMBO,       // above every standard maximum, within the local one
    SZ_OVER_LOCAL   // above the local ceiling: THIS DEVICE will not take it
  } size_class_e;
 
endpackage
 
module frame_size_classifier
  import frame_max_pkg::*;
  import frame_size_pkg::*;
#(
  // Configured, not fixed. A default deployment sets this to MAX_BASIC;
  // a jumbo deployment sets it to whatever every device on the path
  // agreed on -- an agreement this module cannot verify (Section 12).
  parameter int unsigned LOCAL_CEILING = MAX_BASIC,
  // What the receive buffer can actually hold. Independent of the
  // configured ceiling, and the assertion below is why.
  parameter int unsigned BUFFER_OCTETS = MAX_BASIC
) (
  input  logic        clk,
  input  logic        rst_n,
 
  input  logic        frame_done,
  input  logic [15:0] frame_octets,
 
  output logic        class_valid,
  output size_class_e size_class,
  // Named for what it means: this device's limit, not the frame's fault.
  output logic        over_local_ceiling,
  // Above every STANDARD maximum. Independent of local configuration, so
  // it means the same thing on every device on the path.
  output logic        over_standard
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      class_valid        <= 1'b0;
      size_class         <= SZ_NORMAL;
      over_local_ceiling <= 1'b0;
      over_standard      <= 1'b0;
    end else begin
      class_valid <= frame_done;
 
      if (frame_done) begin
        over_local_ceiling <= (frame_octets > 16'(LOCAL_CEILING));
        over_standard      <= (frame_octets > 16'(MAX_ENVELOPE));
 
        // Order matters: the local ceiling is checked FIRST, because a
        // device configured below the standard must not report a frame as
        // SZ_NORMAL and then drop it. A classification that disagrees with
        // the disposition is worse than no classification.
        if (frame_octets > 16'(LOCAL_CEILING))        size_class <= SZ_OVER_LOCAL;
        else if (frame_octets < 16'(MIN_FRAME_OCTETS)) size_class <= SZ_RUNT;
        else if (frame_octets <= 16'(MAX_BASIC))       size_class <= SZ_NORMAL;
        else if (frame_octets <= 16'(MAX_QTAGGED))     size_class <= SZ_TAGGED;
        else if (frame_octets <= 16'(MAX_ENVELOPE))    size_class <= SZ_ENVELOPE;
        else                                           size_class <= SZ_JUMBO;
      end
    end
  end
 
  // A configured ceiling above the physical buffer is not a stricter
  // configuration -- it is an UNHONOURABLE one. The device accepts a frame
  // it cannot hold, and the failure appears as truncation mid-frame rather
  // than as a rejection at the end.
  // synopsys translate_off
  a_ceiling_fits_buffer: assert final (LOCAL_CEILING <= BUFFER_OCTETS)
    else $fatal(1, "local ceiling exceeds receive buffer -- frames will truncate");
  // synopsys translate_on
 
endmodule

Classification: synthesizable.

What it teaches: that over_local_ceiling and over_standard mean different things and must not be one signal. over_standard is a property of the frame and every device on the path agrees about it. over_local_ceiling is a property of this device's configuration, and the device next to it may disagree — which is the entire mechanism of a path mismatch. A single oversize output makes the two indistinguishable in every counter and every log downstream.

Deliberately simplified: it classifies a completed frame. A cut-through path must decide before the frame ends, which means comparing a running count against the ceiling and aborting mid-frame — a different structure with the same three limits.

Production implication: the build-time assertion on LOCAL_CEILING against BUFFER_OCTETS catches a configuration error that is otherwise invisible until a large frame arrives. A ceiling above the buffer is not "accepting more" — it is accepting frames the device will truncate, and truncation produces a bad check sequence, which reports as a link error. The link is fine; the configuration was arithmetically impossible.

6. "Giant" Is Two Different Findings

The word covers two situations that a single counter merges, and separating them is the difference between a five-minute diagnosis and a week.

A frame above every standard maximum is a frame that should not exist. Whatever produced it is emitting non-standard frames, and no configuration on this device makes that acceptable. The finding is about a peer.

A frame above this device's local ceiling may be entirely standard, and entirely correct, and simply larger than this device was configured to accept. The finding is about this device, or about the difference between it and its neighbour.

The two have opposite responses. The first is escalated to whoever operates the sender. The second is fixed by changing a number on this device — or, more often, by discovering that this device is the only one on the path that still has the default.

And in a jumbo deployment the second is overwhelmingly the common case, which is why a design that reports both as "giants received" produces a counter that is almost always describing a local misconfiguration and is almost always read as describing a remote fault.

7. RTL 2 — A Giant Detector That Says Whose Problem It Is

// SYNTHESIZABLE.
//
// Splits "giant" into the two findings of Section 6, and adds the piece
// that turns a count into a diagnosis: the SIZE of the offending frames.
//
// A count of giants says a limit was exceeded. The DISTRIBUTION of their
// sizes says by how much, and that is what identifies the cause:
//
//   sizes clustered just above the local ceiling -> a neighbour configured
//       slightly higher. Usually the 1500-against-1518 confusion, or a tag
//       this device did not budget for.
//   sizes clustered at a common jumbo value      -> a neighbour running
//       jumbo while this device is not. The cluster names the value.
//   sizes scattered and enormous                 -> genuinely malformed
//       frames, or a length that was never bounded upstream.
module giant_detector
  import frame_max_pkg::*;
#(
  parameter int unsigned LOCAL_CEILING = MAX_BASIC,
  parameter int unsigned CNT_W = 32
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        clear,
 
  input  logic        class_valid,
  input  logic [15:0] frame_octets,
  input  logic        over_local_ceiling,
  input  logic        over_standard,
 
  // Above the local ceiling but WITHIN every standard maximum: a correct
  // frame this device declined. A local configuration finding.
  output logic [CNT_W-1:0] c_over_local_only,
  // Above every standard maximum: a peer emitting non-standard frames.
  output logic [CNT_W-1:0] c_over_standard,
 
  // The observed extremes. These survive `clear`, because they describe
  // what the neighbour is capable of sending rather than a window.
  output logic [15:0]      largest_seen,
  output logic [15:0]      smallest_rejected,
  output logic             extremes_valid
);
 
  function automatic logic [CNT_W-1:0] bump(input logic [CNT_W-1:0] v);
    bump = (&v) ? v : (v + 1'b1);
  endfunction
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_over_local_only <= '0;
      c_over_standard   <= '0;
      largest_seen      <= '0;
      smallest_rejected <= '1;
      extremes_valid    <= 1'b0;
    end else begin
      if (clear) begin
        c_over_local_only <= '0;
        c_over_standard   <= '0;
        // extremes deliberately survive.
      end
 
      if (class_valid) begin
        if (frame_octets > largest_seen) largest_seen <= frame_octets;
 
        if (over_local_ceiling) begin
          // The SMALLEST frame this device rejected is the most useful
          // single number on the page: it is an upper bound on where the
          // local ceiling would have to move to stop rejecting anything.
          if (!extremes_valid || (frame_octets < smallest_rejected)) begin
            smallest_rejected <= frame_octets;
            extremes_valid    <= 1'b1;
          end
 
          if (over_standard) c_over_standard   <= bump(c_over_standard);
          else               c_over_local_only <= bump(c_over_local_only);
        end
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it teaches: that smallest_rejected is the actionable number and a count is not. A count says frames are being dropped. The smallest rejected size says exactly what the ceiling would have to be to stop dropping them — and, read the other way, how far the neighbour's configuration is from this one. A device with LOCAL_CEILING at 1518 and smallest_rejected at 1522 has a tag problem; the same device with smallest_rejected at 9018 has a neighbour running jumbo.

Deliberately simplified: two extremes rather than a distribution. Section 8's histogram supplies the distribution, and the two are complementary — the extreme is available immediately and from a single frame, the distribution needs traffic.

Production implication: largest_seen is tracked over all frames, not only rejected ones, and that is what makes it useful before anything breaks. A device whose largest seen frame has been creeping upward over weeks is on a path where something is being reconfigured, and the trend is visible while every frame is still within the ceiling. A counter that only moves after the first drop cannot warn; an extreme over all traffic can.

8. RTL 3 — The Size Distribution, and Why Buckets Beat an Average

// SYNTHESIZABLE INSTRUMENTATION.
//
// Frame size distributions are BIMODAL in almost every real deployment:
// a cluster of small frames (acknowledgements, control traffic) and a
// cluster at whatever the path MTU is. An AVERAGE frame size lands in the
// empty middle and describes no frame that was ever sent.
//
// The bucket edges are chosen to make the interesting boundaries visible
// rather than to be uniform -- a uniform histogram spends most of its
// buckets on sizes that never occur.
module frame_size_histogram
  import frame_max_pkg::*;
  import frame_size_pkg::*;
#(
  parameter int unsigned CNT_W = 32,
  parameter int unsigned NBUCKET = 8
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        clear,
 
  input  logic        class_valid,
  input  logic [15:0] frame_octets,
 
  output logic [CNT_W-1:0] bucket [NBUCKET],
  output logic [CNT_W-1:0] frames_total
);
 
  // Deliberately non-uniform. Buckets 1 and 2 straddle the 1518/1522
  // boundary at one-octet resolution, because that is where a tag-budget
  // error lives and a coarse bucket hides it entirely.
  function automatic int unsigned bucket_of(input logic [15:0] n);
    if      (n <  16'(MIN_FRAME_OCTETS)) bucket_of = 0;   // runts
    else if (n == 16'(MIN_FRAME_OCTETS)) bucket_of = 1;   // exactly at the floor
    else if (n <= 16'd511)               bucket_of = 2;   // small
    else if (n <  16'(MAX_BASIC))        bucket_of = 3;   // mid
    else if (n <= 16'(MAX_BASIC))        bucket_of = 4;   // exactly 1518
    else if (n <= 16'(MAX_QTAGGED))      bucket_of = 5;   // 1519..1522: tagged
    else if (n <= 16'(MAX_ENVELOPE))     bucket_of = 6;   // envelope range
    else                                 bucket_of = 7;   // above every standard
  endfunction
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < NBUCKET; i++) bucket[i] <= '0;
      frames_total <= '0;
    end else if (clear) begin
      for (int i = 0; i < NBUCKET; i++) bucket[i] <= '0;
      frames_total <= '0;
    end else if (class_valid) begin
      automatic int unsigned b = bucket_of(frame_octets);
      if (!(&bucket[b])) bucket[b] <= bucket[b] + 1'b1;
      if (!(&frames_total)) frames_total <= frames_total + 1'b1;
    end
  end
 
endmodule

Classification: synthesizable instrumentation.

What it teaches: that bucket edges are the design, not the counting. A uniform histogram over 0 to 9000 in eight buckets puts 1518 and 1522 in the same bucket — and those four octets are the difference between "this device does not budget for tags" and "everything is fine". The edges here are placed at the boundaries that carry meaning, and the buckets are consequently unequal in width by design.

Deliberately simplified: eight buckets and no jumbo resolution. A deployment running jumbo frames would add edges around its configured value, for the same reason buckets 4 and 5 exist here.

Production implication: bucket 1 — frames at exactly the floor — is separated from bucket 2 deliberately. Chapter 5.6 showed that a frame at exactly 64 octets is the padded case, and its count is the padding rate, measured without needing the transmitter's cooperation. A receiver can therefore estimate how much of its neighbour's traffic is floor rather than data, from size alone — the one part of Chapter 5.6 §12's measurement available from the outside.

9. RTL 4 — Efficiency as a Computed Quantity, Not a Rule of Thumb

// SYNTHESIZABLE INSTRUMENTATION.
//
// Computes the two numbers Section 4 argued should never be conflated:
//
//   wire efficiency -- payload octets as a fraction of everything the wire
//                      carried, INCLUDING preamble, delimiter and gap.
//                      Rises slowly with frame size and saturates.
//
//   frame rate      -- frames per window. Falls in direct proportion to
//                      frame size, and is what per-frame work is charged
//                      against.
//
// A design that reports only the first concludes that jumbo frames buy
// two percent. A design that reports both sees the factor of six.
module overhead_efficiency
  import frame_size_pkg::*;
#(
  parameter int unsigned CNT_W  = 40,
  parameter int unsigned WINDOW = 1_000_000,
  parameter int unsigned WIN_W  = $clog2(WINDOW + 1),
  // Octets on the wire that are not part of the frame: preamble and start
  // delimiter (8) plus the interframe gap (12). Chapter 5.9 owns the gap;
  // it is a parameter here because a design that omits it overstates its
  // own efficiency and never finds out.
  parameter int unsigned WIRE_EXTRA = 20
) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic        frame_done,
  input  logic [15:0] frame_octets,
  input  logic [15:0] payload_octets,   // client data only: pad excluded
 
  output logic [CNT_W-1:0] win_frames,
  output logic [CNT_W-1:0] win_payload,
  output logic [CNT_W-1:0] win_wire,
  // Per mille, integer. Section 12 of Chapter 5.6 explains why not percent.
  output logic [CNT_W-1:0] efficiency_permille,
  output logic             win_valid
);
 
  logic [WIN_W-1:0] win_q;
  logic [CNT_W-1:0] f_q, p_q, w_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      win_q <= '0; f_q <= '0; p_q <= '0; w_q <= '0;
      win_frames <= '0; win_payload <= '0; win_wire <= '0;
      efficiency_permille <= '0;
      win_valid <= 1'b0;
    end else begin
      win_valid <= 1'b0;
 
      if (frame_done) begin
        f_q <= f_q + 1'b1;
        p_q <= p_q + CNT_W'(payload_octets);
        // The wire cost includes what is NOT in the frame. Omitting it is
        // the standard way an efficiency figure comes out flattering.
        w_q <= w_q + CNT_W'(frame_octets) + CNT_W'(WIRE_EXTRA);
      end
 
      if (win_q == WIN_W'(WINDOW - 1)) begin
        win_frames  <= f_q;
        win_payload <= p_q;
        win_wire    <= w_q;
        efficiency_permille <= (w_q == '0) ? '0 : ((p_q * 1000) / w_q);
        win_valid   <= 1'b1;
        f_q <= '0; p_q <= '0; w_q <= '0; win_q <= '0;
      end else begin
        win_q <= win_q + 1'b1;
      end
    end
  end
 
endmodule

Classification: synthesizable instrumentation.

What it teaches: that an efficiency figure is only honest if it counts the octets that are not in the frame. Preamble, start delimiter and interframe gap occupy the wire and appear in no frame, so a calculation over frame octets alone reports a number that is always too high — and too high by an amount that grows as frames get smaller, which is precisely where the figure is being used to make a decision.

Deliberately simplified: WIRE_EXTRA is a constant 20. Chapter 5.9 shows the gap is not always exactly twelve octets, so a rigorous version takes the measured gap; the parameter makes the assumption visible rather than hiding it inside the sum.

Production implication: payload_octets excludes pad, which means this module depends on Chapter 5.6's stripper having separated them. Feeding it frame octets instead produces an efficiency figure that counts the floor's padding as useful data — and a link carrying mostly minimum-size frames would then report high efficiency while delivering almost nothing, which is the exact opposite of the truth.

10. The Unit of Agreement Is the Path

Every device in that figure is behaving correctly. Hop 2 is configured for standard frames and enforces standard frames, which is exactly what it was told to do. The sender is configured for jumbo frames and emits them, which is exactly what it was told to do. There is no defective device.

And no device computes the number in the lower box. Each one can see one link in each direction. The path maximum is the minimum over a set of values that no participant has access to, and there is no mechanism at this layer that gathers them.

So the agreement has a peculiar structure: it must be complete to be effective, and its completeness is not checkable from inside. A path is configured for jumbo frames by configuring every device individually, and there is no step at the end that verifies the set.

11. Why a Mismatch Is a Black Hole and Not an Error

Follow a 9000-octet frame into hop 2 and watch what each participant records.

Hop 2 drops it and counts it — as an oversize frame. That is a local counter, on a device nobody is looking at, whose name suggests a remote fault. It is very often the only record anywhere that anything happened.

The sender records nothing. The frame was transmitted successfully; the link came up, the check sequence was correct, and the interface counters increment normally. From the sender's side the frame left.

The receiver records nothing. No frame arrived, and a frame that does not arrive leaves no trace — the same absence-of-evidence property Chapter 5.4 §15 identified for a filter false negative.

And the small frames get through perfectly. Connectivity tests pass. Address resolution works. Control-plane traffic, which is small, is entirely unaffected. Every diagnostic that sends a small probe reports a healthy path, and the only traffic that fails is bulk transfer — which is to say, the traffic the jumbo configuration was deployed for.

That combination is the signature, and it is worth memorising as a shape rather than as a case: works for small transfers, hangs for large ones, no errors anywhere. It is produced by a size threshold somewhere on the path, and the threshold is a configuration rather than a fault.

And there is a reason it takes so long to find in practice, which is worth naming. The three places an engineer instinctively looks — the two endpoints and the physical link between them — are precisely the three places with no evidence. The endpoints are healthy and say so. The link is healthy and says so. The investigation converges on the physical layer because that is where an unexplained loss usually lives, and the physical layer is fine.

The evidence exists, on a device that is not part of the reported problem, in a counter whose name suggests somebody else's fault. Nothing routes the investigator to it, because from the endpoints' perspective that device is not implicated in anything.

Which is what makes the size sweep the right first move rather than a last resort. It does not require access to the intermediate devices, it does not require any counter to be read, and it converts a distributed absence of evidence into a single number — the size at which delivery stops. That number then names which device to look at, instead of the search naming it.

12. RTL 5 — Detecting the Mismatch From Observation, Not Configuration

// SYNTHESIZABLE DIAGNOSTIC.
//
// A device cannot verify a path agreement (Section 10). What it CAN do is
// report the evidence, and the evidence is a pair of observed maxima:
//
//   rx_observed_max -- the largest frame that actually arrived here. An
//                      upper bound on what the path from each peer will
//                      pass. It is EVIDENCE, not a limit.
//
//   tx_attempted_max-- the largest frame this device has sent. Compared
//                      against the peer's rx_observed_max (by management,
//                      not by this module), a gap between them localises
//                      the smaller ceiling to a specific hop.
//
// The asymmetry check is the diagnostic: a link on which large frames
// arrive but no large frames are acknowledged, or the reverse, has a
// one-directional ceiling -- which is common, because ceilings are
// configured per interface and interfaces are configured one at a time.
module mtu_mismatch_detector
  import frame_max_pkg::*;
#(
  parameter int unsigned LOCAL_CEILING = MAX_BASIC,
  parameter int unsigned CNT_W  = 32,
  // Frames that must be seen before an observed maximum is treated as
  // meaningful. A single large frame proves the path passed one; a
  // sustained absence of large frames proves nothing at all.
  parameter int unsigned CONFIDENCE_N = 1000
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
 
  input  logic        rx_done,
  input  logic [15:0] rx_octets,
  input  logic        rx_over_local,
 
  input  logic        tx_done,
  input  logic [15:0] tx_octets,
 
  output logic [15:0]      rx_observed_max,
  output logic [15:0]      tx_attempted_max,
  output logic [CNT_W-1:0] rx_frames,
 
  // High once enough traffic has been seen for rx_observed_max to mean
  // something. Without it, a quiet link reports an observed maximum of
  // zero and looks identical to a link that blocks everything large.
  output logic             observation_valid,
 
  // The actionable signal: this device is emitting frames larger than
  // anything it has ever received, on a link that has carried plenty of
  // traffic. Consistent with a smaller ceiling in the return direction.
  output logic             asymmetry_suspected,
 
  // Set when this device is the one doing the dropping. Distinguished
  // from the above because the responses differ: this one is fixed HERE.
  output logic             local_ceiling_is_binding
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rx_observed_max          <= '0;
      tx_attempted_max         <= '0;
      rx_frames                <= '0;
      observation_valid        <= 1'b0;
      asymmetry_suspected      <= 1'b0;
      local_ceiling_is_binding <= 1'b0;
    end else begin
      if (clear) begin
        // Observed maxima are NOT cleared. They are evidence about the
        // path, and a clear is a housekeeping act on this device.
        rx_frames <= '0;
      end
 
      if (rx_done) begin
        if (!(&rx_frames)) rx_frames <= rx_frames + 1'b1;
        if (rx_octets > rx_observed_max) rx_observed_max <= rx_octets;
        // This device rejected a frame the path was willing to carry, so
        // the binding constraint is here and the fix is here.
        if (rx_over_local) local_ceiling_is_binding <= 1'b1;
      end
 
      if (tx_done && (tx_octets > tx_attempted_max))
        tx_attempted_max <= tx_octets;
 
      observation_valid <= (rx_frames >= CNT_W'(CONFIDENCE_N));
 
      // Only meaningful once enough has been seen. The guard is the whole
      // reason this is a diagnostic rather than a noise source.
      asymmetry_suspected <= observation_valid &&
                             (tx_attempted_max > rx_observed_max) &&
                             (tx_attempted_max > 16'(MAX_BASIC));
    end
  end
 
endmodule

Classification: synthesizable diagnostic.

What it teaches: that an observed maximum is evidence and a configured maximum is an intention, and only the first can detect a mismatch. Reading back the configured ceiling from every device tells you what everyone meant; comparing observed maxima tells you what the path does. In the Figure 2 path, every device's configuration reads 9018 except one — and the one is the device nobody thought to check, because it was already in service when the change was made.

Deliberately simplified: asymmetry_suspected compares two unrelated maxima and will fire on a link whose traffic is genuinely one-sided. It is a hint that narrows an investigation, not a fault.

Production implication: observation_valid is the difference between a useful diagnostic and a nuisance. A quiet link reports an observed maximum of zero, which is indistinguishable from a link that drops everything large — and a design without the confidence guard raises the alarm on every interface that has not carried traffic yet, most notably immediately after boot. That is Chapter 5.4 §12's argument again: a diagnostic that fires during a known-quiet state trains its readers to ignore it.

13. Assertions — Local Properties Only, and One That Cannot Be Written

// ---------------------------------------------------------------------
// P1 -- The classification agrees with the disposition. A frame reported
// as SZ_NORMAL and then dropped is worse than an unclassified frame.
// ---------------------------------------------------------------------
property p_class_matches_disposition;
  @(posedge clk) disable iff (!rst_n)
    (class_valid && over_local_ceiling) |-> (size_class == SZ_OVER_LOCAL);
endproperty
a_class_matches_disposition: assert property (p_class_matches_disposition);
 
// ---------------------------------------------------------------------
// P2 -- over_standard is independent of local configuration. Every device
// on the path must agree about it, so it may not reference LOCAL_CEILING.
// ---------------------------------------------------------------------
property p_over_standard_is_absolute;
  @(posedge clk) disable iff (!rst_n)
    class_valid |-> (over_standard == ($past(frame_octets) > 16'(MAX_ENVELOPE)));
endproperty
a_over_standard_is_absolute: assert property (p_over_standard_is_absolute);
 
// ---------------------------------------------------------------------
// P3 -- A frame within every standard maximum is never reported as over
// the standard, whatever this device is configured for.
// ---------------------------------------------------------------------
property p_standard_frame_not_flagged;
  @(posedge clk) disable iff (!rst_n)
    (class_valid && $past(frame_octets) <= 16'(MAX_ENVELOPE)) |-> !over_standard;
endproperty
a_standard_frame_not_flagged: assert property (p_standard_frame_not_flagged);
 
// ---------------------------------------------------------------------
// P4 -- The two giant counters partition. A frame above the standard is
// also above a standard-configured local ceiling, so double-counting is
// the natural bug.
// ---------------------------------------------------------------------
property p_giant_counters_disjoint;
  @(posedge clk) disable iff (!rst_n)
    class_valid |=> ((c_over_standard + c_over_local_only) <= total_rejected);
endproperty
a_giant_counters_disjoint: assert property (p_giant_counters_disjoint);
 
// ---------------------------------------------------------------------
// P5 -- smallest_rejected is a real observed frame size, never a default
// or a sentinel leaking out.
// ---------------------------------------------------------------------
property p_smallest_rejected_is_real;
  @(posedge clk) disable iff (!rst_n)
    extremes_valid |-> (smallest_rejected > 16'(LOCAL_CEILING));
endproperty
a_smallest_rejected_is_real: assert property (p_smallest_rejected_is_real);
 
// ---------------------------------------------------------------------
// P6 -- Extremes are monotonic. A largest-seen that can fall is tracking
// a window, and describes nothing.
// ---------------------------------------------------------------------
property p_largest_seen_monotonic;
  @(posedge clk) disable iff (!rst_n)
    ##1 (largest_seen >= $past(largest_seen));
endproperty
a_largest_seen_monotonic: assert property (p_largest_seen_monotonic);
 
// ---------------------------------------------------------------------
// P7 -- Exactly one histogram bucket per frame.
// ---------------------------------------------------------------------
property p_one_bucket_per_frame;
  @(posedge clk) disable iff (!rst_n)
    class_valid |=> (bucket_sum == $past(bucket_sum) + 1);
endproperty
a_one_bucket_per_frame: assert property (p_one_bucket_per_frame);
 
// ---------------------------------------------------------------------
// P8 -- The 1518/1522 boundary is resolved at one-octet resolution. A
// 1519-octet frame must NOT land in the basic bucket.
// ---------------------------------------------------------------------
property p_tag_bucket_boundary;
  @(posedge clk) disable iff (!rst_n)
    (class_valid && $past(frame_octets) == 16'(MAX_BASIC) + 1)
      |=> (bucket_hit == 5);
endproperty
a_tag_bucket_boundary: assert property (p_tag_bucket_boundary);
 
// ---------------------------------------------------------------------
// P9 -- Efficiency counts the octets that are NOT in the frame. A wire
// total below the frame total means WIRE_EXTRA was dropped from the sum.
// ---------------------------------------------------------------------
property p_wire_exceeds_frames;
  @(posedge clk) disable iff (!rst_n)
    (win_valid && win_frames != '0)
      |-> (win_wire >= (win_frames * CNT_W'(WIRE_EXTRA)));
endproperty
a_wire_exceeds_frames: assert property (p_wire_exceeds_frames);
 
// ---------------------------------------------------------------------
// P10 -- Efficiency is a fraction. Above unity means payload octets are
// being counted that the wire did not carry -- pad included as payload
// is the usual cause.
// ---------------------------------------------------------------------
property p_efficiency_bounded;
  @(posedge clk) disable iff (!rst_n)
    win_valid |-> (efficiency_permille <= 1000);
endproperty
a_efficiency_bounded: assert property (p_efficiency_bounded);
 
// ---------------------------------------------------------------------
// P11 -- Payload never exceeds the frame that carried it.
// ---------------------------------------------------------------------
property p_payload_within_frame;
  @(posedge clk) disable iff (!rst_n)
    frame_done |-> (payload_octets <= frame_octets);
endproperty
a_payload_within_frame: assert property (p_payload_within_frame);
 
// ---------------------------------------------------------------------
// P12 -- No mismatch alarm before there is evidence. This is the guard
// that stops the diagnostic firing on every interface at boot.
// ---------------------------------------------------------------------
property p_no_alarm_without_evidence;
  @(posedge clk) disable iff (!rst_n)
    !observation_valid |-> !asymmetry_suspected;
endproperty
a_no_alarm_without_evidence: assert property (p_no_alarm_without_evidence);
 
// ---------------------------------------------------------------------
// P13 -- Observed maxima survive a counter clear: they are evidence about
// the path, not a measurement window.
// ---------------------------------------------------------------------
property p_observed_max_survives_clear;
  @(posedge clk) disable iff (!rst_n)
    clear |=> ($stable(rx_observed_max) && $stable(tx_attempted_max));
endproperty
a_observed_max_survives_clear: assert property (p_observed_max_survives_clear);
 
// ---------------------------------------------------------------------
// P14 -- local_ceiling_is_binding is sticky and means what it says: this
// device rejected a frame, so the fix is HERE.
// ---------------------------------------------------------------------
property p_binding_is_sticky;
  @(posedge clk) disable iff (!rst_n)
    local_ceiling_is_binding |=> local_ceiling_is_binding;
endproperty
a_binding_is_sticky: assert property (p_binding_is_sticky);
 
// ---------------------------------------------------------------------
// P15 -- The observed receive maximum never exceeds the local ceiling,
// because larger frames were rejected rather than received.
// ---------------------------------------------------------------------
property p_rx_max_within_ceiling;
  @(posedge clk) disable iff (!rst_n)
    observation_valid |-> (rx_observed_max <= 16'(LOCAL_CEILING));
endproperty
a_rx_max_within_ceiling: assert property (p_rx_max_within_ceiling);
 
// ---------------------------------------------------------------------
// P16 -- COVERAGE. A frame at exactly the local ceiling: accepted, and
// one octet from being rejected.
// ---------------------------------------------------------------------
c_exact_ceiling_seen: cover property (
  @(posedge clk) disable iff (!rst_n)
    (class_valid && $past(frame_octets) == 16'(LOCAL_CEILING))
);
 
// ---------------------------------------------------------------------
// P17 -- COVERAGE. A frame one octet above it, which is the only frame
// that distinguishes a correct comparison from an off-by-one.
// ---------------------------------------------------------------------
c_one_over_ceiling_seen: cover property (
  @(posedge clk) disable iff (!rst_n)
    (class_valid && $past(frame_octets) == 16'(LOCAL_CEILING) + 1)
);
 
// ---------------------------------------------------------------------
// P18 -- COVERAGE. Every bucket exercised. An untouched bucket is a size
// range the verification never produced, and bucket 5 is the one that
// goes untouched in every environment that forgets tags exist.
// ---------------------------------------------------------------------
c_all_buckets_seen: cover property (
  @(posedge clk) disable iff (!rst_n) all_buckets_nonzero
);

14. Verification — Twenty-Two Scenarios and a Test the Single-Link Testbench Cannot Run

15. Debugging — The Signature Is What Works, Not What Fails

Symptom — connectivity is fine, bulk transfers hang, no errors anywhere.

This shape is a size threshold until proven otherwise, and it is worth reaching for before any physical-layer hypothesis. Sweep the size: send at 1400, 1450, 1500, 1550, 1600 and find where it stops. The threshold is the answer, and it names the ceiling of the device that is dropping. If nothing is available to sweep with, read smallest_rejected from Section 7 on each device along the path — the one that has it set is the one enforcing the limit.

Symptom — a device reports "giants received" on a link where the neighbour is correctly configured.

Read Section 7's split first. c_over_standard means the neighbour really is emitting frames above every standard maximum, and that is a peer fault. c_over_local_only means this device's ceiling is below the neighbour's and the frames were entirely legitimate. In a jumbo deployment the second is almost always the case, and the counter's name is the reason the investigation starts in the wrong place.

Symptom — the mismatch appeared after a device was replaced, and its configuration matches the old one.

Check LOCAL_CEILING against the frame sizes actually seen, not against the configuration file. A ceiling expressed as an MTU (1500, the payload) and a ceiling expressed as a frame size (1518, or 1522 with a tag) differ by 18 or 22 octets, and a device that interprets a configured 1500 as a frame ceiling silently rejects every full-size standard frame. The signature is smallest_rejected = 1518 on a device configured for "1500", which reads as a contradiction and is a units error.

Symptom — jumbo frames were enabled and throughput did not improve.

Expected, if throughput was measured in bits per second. Section 4's arithmetic says the wire-efficiency gain is about two percentage points. The gain is a six-fold reduction in frame rate, and it shows up in receive-path CPU, interrupt counts and descriptor pressure — none of which a bandwidth test measures. Read win_frames from Section 9 before and after; if it did not fall by roughly the frame-size ratio, the large frames are not actually being sent.

Symptom — a device configured for jumbo frames truncates them.

Not a path problem. The configured ceiling exceeds the receive buffer, which Section 5's build-time assertion exists to catch and which is invisible at run time until a large frame arrives. The tell is that the truncated frames have a bad check sequence, so they are counted as link errors — and the link is fine.

Symptom — large frames pass in one direction and not the other.

Ceilings are configured per interface, and interfaces are configured one at a time. Section 12's asymmetry_suspected is built for exactly this, and the confirmation is to compare rx_observed_max at each end: the direction that works has a large one, the direction that does not has a small one, and the smaller value names the ceiling that is binding.

16. Common Misconceptions

"MTU is 1500, so the maximum frame is 1500."

The wrong model: one number describes both.

What it costs: a device configured with 1500 as a frame ceiling rejects every full-size standard frame, because a 1500-octet payload produces a 1518-octet frame. The symptom is a device that drops the most common large frame on the network while its configuration appears to match everyone else's.

The corrected model: MTU is the payload ceiling; the frame maximum includes the structure around it. 1500 payload gives 1518 basic, 1522 with one tag, and 2000 for an envelope frame — and every configuration interface must be read carefully to know which of the two it means.

"The maximum frame size is defined by the standard, like the minimum."

The wrong model: both ends of the range are equally normative.

What it costs: you assume a jumbo deployment is standards-compliant and interoperable by construction, and you have no reason to check the specific value each vendor supports. The mismatch that follows is between two devices that both believe they support jumbo frames.

The corrected model: three maxima are standardised — 1518, 1522, 2000. Jumbo frames are not. The commonly used 9000 is a convention, values differ between implementations, and a network running them is running on an agreement with no normative text behind it.

"Enabling jumbo frames on the two endpoints enables jumbo frames."

The wrong model: the agreement is between the devices that care.

What it costs: Section 11's black hole. Every device on the path enforces its own ceiling, and one that was in service before the change silently removes every large frame while reporting correct behaviour.

The corrected model: the unit of agreement is the path, and its completeness is not checkable from inside any device. The effective maximum is the minimum over a set of values no participant can see, which is why Section 12 reports observed maxima rather than configured ones.

"An oversize frame is an error."

The wrong model: a rejected frame indicates a fault at the sender.

What it costs: an investigation aimed at a neighbour that is behaving correctly, while the actual fix — a number on the device doing the counting — goes unconsidered. In a jumbo deployment this is the common case, not the exception.

The corrected model: two different findings share the word. Above every standard maximum is a peer fault. Above this device's ceiling is a statement about this device, and the frame may be entirely legitimate. Section 5 names the signals over_standard and over_local_ceiling precisely so downstream code cannot conflate them.

"Jumbo frames give you more bandwidth."

The wrong model: larger frames mean a higher useful bit rate.

What it costs: the change is justified by a number that barely moves, so it is judged a failure — and the benefit that did arrive is in a quantity nobody was measuring.

The corrected model: about 2% more bandwidth and about 6× less per-frame work. Wire efficiency at 1500 octets is already roughly 97.5%, so there is very little left to recover. What falls by a factor of six is the frame rate, and with it every lookup, descriptor, interrupt and buffer operation — which is what is usually running out.

17. Interview Reasoning

"Why is the maximum frame 1500 bytes?"

The weak answer treats it like the minimum and looks for a derivation. The answer that ends the topic says there is none: unlike the 64-octet floor, which is a round-trip propagation time in disguise, 1500 was chosen — a compromise among receive buffer cost, the work lost when a frame is discarded, and fairness on a shared medium, made when memory was expensive. The payoff is what follows from that difference: a derived floor cannot be relaxed by agreement, while a conventional ceiling can — which is exactly what makes it possible to get wrong.

"How would you debug a link where small packets work and large transfers hang?"

The trap is to reach for the physical layer. The strong answer names the shape first — works small, hangs large, no errors anywhere — and identifies it as a size threshold on the path, then proposes a size sweep rather than more sampling, because the failure has a threshold and thresholds are found by sweeping. The finishing detail is where the evidence lives: on an intermediate device, in a counter named for a remote fault, and the smallest rejected frame size names the ceiling exactly.

"Why can't a switch just tell the sender its MTU is too small?"

Because nothing in the frame format carries a reply, and the layer that could send one is above this. More precisely: the frame is destroyed by a device that is neither the sender nor the receiver, and the only thing it can do at this layer is count. Adding that this is a general property — the path maximum is the minimum of values no participant can see, so the agreement's completeness is not checkable from inside — shows the problem is understood structurally rather than as a missing feature.

"You're asked to assert that any frame the MAC accepts will reach its destination. What do you say?"

That the property is right and unwritable here, because deliverability is path-scoped and every signal in the design is link-scoped. The complete answer names the specific danger: the assertion does not fail, it passes vacuously in a single-link testbench, and a passing assertion next to a real outage is worse than no assertion, because the observability that would have found the problem then looks redundant. Then the constructive half: assert local properties that can genuinely fail, report observed maxima as evidence, and verify path behaviour in a testbench that actually contains two different ceilings.

18. Understanding Check

19. What's Next

The claim this chapter defended: the maximum frame size is an agreement whose scope is a path, enforced by devices whose scope is a link.

Unlike the floor, it was never derived from anything physical — it is where receive-buffer cost, per-error recovery and shared-medium fairness met for the hardware of 1980, and three maxima are standardised while the jumbo frames deployed everywhere are standardised nowhere. Raising it buys about two percentage points of wire efficiency and a six-fold reduction in frame rate, and only the second is worth the coordination.

And the coordination is the hard part, because no device can check it. Each one enforces its own ceiling correctly; the path's ceiling is the minimum over values no participant can see; and a partial agreement produces a black hole in which small frames pass, large frames vanish, the sender reports success, the receiver reports nothing, and the only trace is a counter on a device in the middle. A property about deliverability cannot be asserted from inside a MAC — it can only be evidenced, which is why Section 12 reports what it has observed rather than what it was configured with.

Chapter 5.8 — Frame Check Sequence takes the field that follows the payload, and it has a similar shape hiding in it. The frame check sequence is four octets, and the interesting question is not how it is computed — Chapter 6.2 owns that — but what range it covers.

Because the check is appended at one point in a transmitter and verified at one point in a receiver, everything between those two points is protected and everything outside them is not. An error in a buffer before the append, or in a buffer after the check, is invisible to a mechanism that appears to protect the whole frame. 5.8 pins down the two boundaries, shows what falls outside them, and prepares the question Chapter 6.1 answers: what a detector can promise, and what "detects errors" leaves out.

The full path is on the Ethernet curriculum index.