Ethernet · Module 8
Serialization Delay
A frame's size divided by the line rate — trivial arithmetic whose significance changes by four orders of magnitude, inverting latency budgets so that at 100 Gb/s one metre of cable outweighs an entire minimum-size frame.
Module 7 built the datapaths. This module asks how long they take, and it starts with the term that is most often left out of a latency budget entirely.
Serialization delay is the time to clock a frame onto the wire. A frame of n bits at a line rate of R bits per second takes n/R seconds to emit — and that is the whole of it. There is no subtlety in the formula and nothing to derive.
What is worth a chapter is what the number does across the rates this track has covered.
At 10 Mb/s a maximum-size frame takes 1.214 milliseconds to serialize. At 100 Gb/s the same frame takes 121.4 nanoseconds. Four orders of magnitude, and every other latency term in the system — propagation across a cable, a switch's forwarding decision, a driver's interrupt — changed by a very different factor, or by none at all.
So a latency budget is not a fixed set of proportions. It is a set of terms whose relative sizes are completely rearranged by the line rate — and a budget written when serialization dominated is wrong, in a direction people find surprising, when it no longer does.
1. Scope — What This Chapter Owns
This chapter owns serialization: the definition, the arithmetic, what the wire actually carries per frame, how the term behaves across rates, and how a design measures and budgets it.
It does not own propagation — Chapter 8.2 owns the physics of delay across a medium, and this chapter uses only the figure it will derive, about 5 ns per metre, to make the comparison above.
It does not own throughput. Serialization is a latency term: how long one frame takes. Chapter 8.3 asks how much data per unit time gets through, which is a different question with a different answer.
And it does not own the full latency decomposition — Chapter 8.4 puts every term together and assigns each to a stage.
It builds on Module 5's accounting. Chapter 5.6 §11 established that a minimum frame occupies 84 octet times on the wire once preamble and interframe gap are counted, and Chapter 5.9 established the gap itself. Serialization of a frame and occupancy of a wire slot are different quantities and Section 3 keeps them apart.
The question this chapter answers that its neighbours do not: how long does one frame take to put on the wire, and why does the answer's significance change so much more than the answer does?
2. The Arithmetic, and the Two Quantities It Produces
Serialization delay is the frame's bit count divided by the line rate. For a maximum-size basic frame — 1518 octets, so 12 144 bits — at 1 Gb/s:
12 144 bits / 1 000 000 000 bits per second = 12.144 microsecondsThat is the frame. But Chapter 5.6 §11 showed the wire carries more than the frame: eight octets of preamble and delimiter before it, and twelve octets of interframe gap after it. Those occupy the wire and belong in no frame.
So there are two quantities and they answer different questions:
| Quantity | What it is | What it answers |
|---|---|---|
| frame serialization | frame bits ÷ rate | how long this frame takes to emit |
| wire occupancy | (frame + 20 octets) ÷ rate | how long until the next frame may start |
Illustrative, at 1 Gb/s:
| Frame | Wire slot | Serialization | Occupancy | |
|---|---|---|---|---|
| minimum | 64 octets | 84 octets | 512 ns | 672 ns |
| maximum | 1518 octets | 1538 octets | 12.14 µs | 12.30 µs |
Read the minimum row. The overhead is 20 octets against a 64-octet frame — 31% more wire time than the frame itself takes. On the maximum row it is 20 against 1518, about 1.3%.
Which is why the two quantities must not be conflated. A latency budget wants serialization: how long before the last bit of this frame is out. A frame-rate calculation wants occupancy: how soon the next frame can follow. Using one where the other belongs is a 31% error on small frames and a rounding error on large ones — so it passes testing on bulk traffic and fails on the traffic that matters.
3. Four Orders of Magnitude, and What Does Not Move With Them
Work the two terms against each other, because the comparison is the whole insight.
Serialization scales exactly inversely with the rate. Ten times the rate, one tenth the time. Between 10 Mb/s and 100 Gb/s that is a factor of ten thousand.
Propagation does not scale at all. It is the medium's length divided by the signal's velocity — about 5 nanoseconds per metre in both copper and fibre, as Chapter 8.2 derives — and the line rate does not appear in that expression. A 100-metre link costs about 500 nanoseconds at every rate Ethernet has ever run at.
Illustrative comparison, for a 100-metre link and a maximum-size frame:
| Line rate | Serialization | Propagation (100 m) | Which dominates |
|---|---|---|---|
| 10 Mb/s | 1.214 ms | ~0.5 µs | serialization, by 2400× |
| 100 Mb/s | 121.4 µs | ~0.5 µs | serialization, by 240× |
| 1 Gb/s | 12.14 µs | ~0.5 µs | serialization, by 24× |
| 10 Gb/s | 1.214 µs | ~0.5 µs | serialization, by 2.4× |
| 100 Gb/s | 121.4 ns | ~0.5 µs | propagation, by 4× |
The crossover happens between 10 and 100 Gb/s for a maximum frame — and for a minimum frame it happened much earlier, because a minimum frame is 24 times smaller:
| Line rate | Minimum-frame serialization | Equivalent cable length |
|---|---|---|
| 1 Gb/s | 512 ns | 102 m |
| 10 Gb/s | 51.2 ns | 10.2 m |
| 100 Gb/s | 5.12 ns | 1.02 m |
The 1 Gb/s row is a coincidence worth remembering. A minimum-size frame's serialization at gigabit is 512 ns, and 512 ns of cable is 102 metres — essentially the 100-metre limit of a twisted-pair link. At gigabit, one minimum frame and one full-length copper link cost the same.
And the 100 Gb/s row is the one that reorders budgets. One metre of cable outweighs an entire minimum-size frame. A rack-to-rack hop of ten metres costs twice what ten maximum-size frames cost to serialize.
4. RTL 1 — Computing the Term
// SYNTHESIZABLE.
//
// Computes serialization time and wire occupancy for a frame, at the
// link's CURRENT rate.
//
// The rate is an input rather than a parameter because autonegotiation
// exists: a link that came up at 1 Gb/s may renegotiate to 100 Mb/s, and
// every time derived from the rate changes by a factor of ten at that
// moment. A design that fixed the rate at elaboration is silently wrong
// for the rest of the link's life.
//
// Times are reported in nanoseconds, computed as bit-times multiplied by
// a per-rate bit period. The multiply is by a table entry rather than a
// division, because a divider on this path is unnecessary: there are
// only a handful of rates.
package serdelay_pkg;
typedef enum logic [2:0] {
RATE_10M,
RATE_100M,
RATE_1G,
RATE_10G,
RATE_100G
} line_rate_e;
// Bit period in picoseconds. Exact for every rate Ethernet defines,
// because all of them are decimal powers of ten.
function automatic int unsigned bit_period_ps(input line_rate_e r);
case (r)
RATE_10M: bit_period_ps = 100_000; // 100 ns
RATE_100M: bit_period_ps = 10_000; // 10 ns
RATE_1G: bit_period_ps = 1_000; // 1 ns
RATE_10G: bit_period_ps = 100; // 100 ps
RATE_100G: bit_period_ps = 10; // 10 ps
default: bit_period_ps = 1_000;
endcase
endfunction
// Preamble, start delimiter and interframe gap: on the wire, in no
// frame (Chapter 5.6 §11 and Chapter 5.9).
localparam int unsigned WIRE_EXTRA_OCTETS = 20;
endpackage
module serialization_calc
import serdelay_pkg::*;
(
input logic clk,
input logic rst_n,
input line_rate_e rate,
input logic req_valid,
input logic [13:0] frame_octets,
output logic rsp_valid,
// Picoseconds, because at 100 Gb/s a minimum frame is 5.12 ns and a
// nanosecond resolution would round it to 5 -- a 2% error on the
// smallest and most numerous frames.
output logic [39:0] serialization_ps,
output logic [39:0] occupancy_ps,
// The two quantities of Section 2, kept separate so a caller cannot
// silently use one for the other.
output logic [39:0] overhead_ps
);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
rsp_valid <= 1'b0;
serialization_ps <= '0;
occupancy_ps <= '0;
overhead_ps <= '0;
end else begin
rsp_valid <= req_valid;
if (req_valid) begin
automatic int unsigned bp = bit_period_ps(rate);
// The frame itself.
serialization_ps <= 40'(frame_octets) * 40'd8 * 40'(bp);
// The wire slot: frame plus the octets that are on the wire and
// in no frame.
occupancy_ps <= (40'(frame_octets) + 40'(WIRE_EXTRA_OCTETS))
* 40'd8 * 40'(bp);
overhead_ps <= 40'(WIRE_EXTRA_OCTETS) * 40'd8 * 40'(bp);
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that the unit has to be picoseconds and not nanoseconds. At 100 Gb/s a minimum-size frame serializes in 5.12 ns, and a nanosecond-resolution result rounds it to 5 — a 2.3% error on the smallest and by far the most numerous frames on a saturated link. A resolution chosen for the rates a design was written at becomes a systematic bias at the rates it is ported to, and the bias always understates the small frames.
Deliberately simplified: a table of bit periods rather than a divider. There are five rates and they are all decimal powers of ten, so the table is exact and a divider would be both larger and less accurate.
Production implication: serialization_ps and occupancy_ps are separate outputs with separate names, and overhead_ps is exposed as a third. Section 2 showed the two differ by 31% on a minimum frame, so a design that exposes one and lets callers add the overhead themselves will have callers who forget — and the error is invisible on the bulk traffic most tests use.
5. RTL 2 — Accumulating What the Wire Actually Carried
// SYNTHESIZABLE INSTRUMENTATION.
//
// Accumulates wire occupancy over a window and reports how much of the
// link's TIME was consumed, separated into the three things consuming it:
//
// payload octets -- what the client actually sent
// frame overhead -- addresses, length/type, check sequence, pad
// wire overhead -- preamble, delimiter, interframe gap
//
// The third is the one no frame contains and no octet counter counts, so
// a design without this module cannot see it at all -- and on
// minimum-size traffic it is 24% of the link's time.
module wire_occupancy_accumulator
import serdelay_pkg::*;
#(
parameter int unsigned CNT_W = 48,
parameter int unsigned WINDOW_BITS = 24
) (
input logic clk,
input logic rst_n,
input logic clear,
input line_rate_e rate,
input logic frame_done,
input logic [13:0] frame_octets,
input logic [13:0] payload_octets, // client data only: pad excluded
output logic [CNT_W-1:0] win_payload_octets,
output logic [CNT_W-1:0] win_frame_octets,
output logic [CNT_W-1:0] win_wire_octets,
output logic [CNT_W-1:0] win_frames,
output logic win_valid,
// Time, not octets. The same octet count is a different amount of link
// time at a different rate, and a design comparing octet counts across
// a rate change is comparing incomparable quantities.
output logic [CNT_W-1:0] win_wire_ps,
// Occupancy that no frame contains, per mille of the total. On
// minimum-size traffic this is about 238 -- nearly a quarter of the
// link's time spent on octets no counter reports.
output logic [CNT_W-1:0] wire_overhead_permille
);
logic [WINDOW_BITS-1:0] win_q;
logic [CNT_W-1:0] pay_q, frm_q, wire_q, cnt_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
win_q <= '0; pay_q <= '0; frm_q <= '0; wire_q <= '0; cnt_q <= '0;
win_payload_octets <= '0; win_frame_octets <= '0;
win_wire_octets <= '0; win_frames <= '0; win_valid <= 1'b0;
win_wire_ps <= '0; wire_overhead_permille <= '0;
end else begin
win_valid <= 1'b0;
if (clear) begin
pay_q <= '0; frm_q <= '0; wire_q <= '0; cnt_q <= '0;
end else if (frame_done) begin
pay_q <= pay_q + CNT_W'(payload_octets);
frm_q <= frm_q + CNT_W'(frame_octets);
// The frame plus the octets that are on the wire and in no frame.
wire_q <= wire_q + CNT_W'(frame_octets) + CNT_W'(WIRE_EXTRA_OCTETS);
cnt_q <= cnt_q + 1'b1;
end
if (&win_q) begin
win_payload_octets <= pay_q;
win_frame_octets <= frm_q;
win_wire_octets <= wire_q;
win_frames <= cnt_q;
win_wire_ps <= wire_q * CNT_W'(8) * CNT_W'(bit_period_ps(rate));
wire_overhead_permille <= (wire_q == '0) ? '0
: (((wire_q - frm_q) * 1000) / wire_q);
win_valid <= 1'b1;
pay_q <= '0; frm_q <= '0; wire_q <= '0; cnt_q <= '0; win_q <= '0;
end else begin
win_q <= win_q + 1'b1;
end
end
end
endmoduleClassification: synthesizable instrumentation.
What it teaches: that win_wire_ps and win_wire_octets are not interchangeable across a rate change. The same octet count is a different amount of link time at a different rate — by a factor of ten at each step — so a design trending octets across an autonegotiation event is comparing incomparable quantities. A management interface reporting octets per interval looks continuous through a rate change while the underlying utilisation moved by 10×.
Deliberately simplified: payload octets are supplied by the caller, which means this module depends on Chapter 5.6's stripper having separated data from pad. Feeding it frame octets instead counts padding as payload and inflates the useful fraction.
Production implication: wire_overhead_permille measures a quantity that no octet counter in the system can see — the preamble, delimiter and gap are on the wire and in no frame. On minimum-size traffic it is about 238 per mille, meaning nearly a quarter of the link's time goes to octets nothing reports. A link that reads as fully utilised by every interface counter is spending a quarter of itself on structure, and this is the only place that shows.
6. RTL 3 — A Budget With Named Terms
// SYNTHESIZABLE.
//
// Holds a latency budget as a set of NAMED terms and reports which term
// is dominant, rather than reporting a total against a limit.
//
// The reason is Section 3: the composition of a latency budget inverts
// across the rate range, so the useful output is not "are we within
// budget" but "which term is the budget". A design that reports only a
// total cannot tell an engineer that the thing they are optimising has
// become the smallest term in the sum.
module latency_budget_monitor
import serdelay_pkg::*;
#(
parameter int unsigned CNT_W = 40
) (
input logic clk,
input logic rst_n,
input logic sample_valid,
// Every term in picoseconds, so they are directly comparable.
input logic [CNT_W-1:0] t_serialization,
input logic [CNT_W-1:0] t_propagation,
input logic [CNT_W-1:0] t_store_forward,
input logic [CNT_W-1:0] t_processing,
input logic [CNT_W-1:0] budget_ps,
output logic result_valid,
output logic [CNT_W-1:0] total_ps,
output logic over_budget,
// WHICH term dominates. The output that changes an engineer's actions,
// and the one a total cannot provide.
output logic [1:0] dominant_term,
output logic [CNT_W-1:0] dominant_ps,
// The dominant term as a fraction, because "dominant" is only useful
// if it is actually large -- four terms of 25% each has a dominant one
// and no lever.
output logic [CNT_W-1:0] dominant_permille,
output logic [CNT_W-1:0] worst_total_ps
);
localparam logic [1:0] T_SER = 2'd0;
localparam logic [1:0] T_PROP = 2'd1;
localparam logic [1:0] T_SF = 2'd2;
localparam logic [1:0] T_PROC = 2'd3;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
result_valid <= 1'b0; total_ps <= '0; over_budget <= 1'b0;
dominant_term <= T_SER; dominant_ps <= '0;
dominant_permille <= '0; worst_total_ps <= '0;
end else begin
result_valid <= sample_valid;
if (sample_valid) begin
automatic logic [CNT_W-1:0] tot =
t_serialization + t_propagation + t_store_forward + t_processing;
automatic logic [CNT_W-1:0] mx = t_serialization;
automatic logic [1:0] ix = T_SER;
if (t_propagation > mx) begin mx = t_propagation; ix = T_PROP; end
if (t_store_forward > mx) begin mx = t_store_forward; ix = T_SF; end
if (t_processing > mx) begin mx = t_processing; ix = T_PROC; end
total_ps <= tot;
over_budget <= (tot > budget_ps);
dominant_term <= ix;
dominant_ps <= mx;
dominant_permille <= (tot == '0) ? '0 : ((mx * 1000) / tot);
if (tot > worst_total_ps) worst_total_ps <= tot;
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that dominant_permille is needed alongside dominant_term. Four terms of roughly equal size have a dominant one, and knowing which it is changes nothing — there is no lever. A dominant term at 700 per mille is a lever; one at 280 is a report. Reporting the identity without the fraction produces an engineer optimising a term that is 28% of the total, for a maximum possible gain smaller than the measurement noise.
Deliberately simplified: four terms with fixed identities. Chapter 8.4 decomposes latency properly and finds more terms — queueing, clock-domain crossings, elastic buffering — and the structure is the same with a wider comparison.
Production implication: every term is in picoseconds, and the uniformity is the point. Terms arriving in mixed units — serialization in nanoseconds because that is convenient, propagation in microseconds because cables are long — must be converted somewhere, and the conversion is where a factor of a thousand goes missing. A budget whose terms are directly comparable can be compared by a comparator; one whose terms need scaling needs a convention, and conventions are what Chapter 6.2 spent a chapter on.
7. RTL 4 — When the Rate Changes Underneath Everything
// SYNTHESIZABLE.
//
// Detects a line-rate change and forces every derived quantity to be
// recomputed, while making the discontinuity visible to anything
// trending the numbers across it.
//
// The failure this prevents is quiet and total. A design that computed
// its serialization times, its budget thresholds and its frame-rate
// ceiling at link-up, and then renegotiated from 1 Gb/s to 100 Mb/s,
// has every one of those numbers wrong by 10x -- and nothing reports an
// error, because nothing is broken. The arithmetic was correct for a
// rate that is no longer the rate.
module rate_change_detector
import serdelay_pkg::*;
#(
parameter int unsigned CNT_W = 16
) (
input logic clk,
input logic rst_n,
input logic rate_valid,
input line_rate_e rate,
output logic rate_changed, // pulse: recompute everything
output line_rate_e rate_current,
output line_rate_e rate_previous,
// Held high for one window after a change, so anything trending a
// rate-derived quantity can mark the sample as a discontinuity rather
// than as a measurement.
output logic trend_break,
output logic [CNT_W-1:0] c_rate_changes,
// Every rate this link has ever run at. A link that has changed rate
// repeatedly is a link with a physical problem, and the set is more
// diagnostic than the count.
output logic [4:0] rates_seen,
output logic ever_downshifted
);
line_rate_e prev_q;
logic have_q;
logic [7:0] hold_q;
assign rate_current = prev_q;
assign rate_previous = prev_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
prev_q <= RATE_1G; have_q <= 1'b0; hold_q <= '0;
rate_changed <= 1'b0; trend_break <= 1'b0;
c_rate_changes <= '0; rates_seen <= '0; ever_downshifted <= 1'b0;
end else begin
rate_changed <= 1'b0;
if (rate_valid) begin
rates_seen[rate] <= 1'b1;
if (!have_q) begin
prev_q <= rate;
have_q <= 1'b1;
end else if (rate != prev_q) begin
rate_changed <= 1'b1;
c_rate_changes <= c_rate_changes + 1'b1;
// A DOWNSHIFT is the interesting direction: a link that
// negotiated down did so because something in the channel
// failed, and it is not a configuration event.
if (rate < prev_q) ever_downshifted <= 1'b1;
prev_q <= rate;
hold_q <= 8'hFF;
end
end
// Mark a window either side of the change, so a trend consumer can
// discard the sample rather than plot a step as a measurement.
if (hold_q != 8'd0) begin
trend_break <= 1'b1;
hold_q <= hold_q - 8'd1;
end else begin
trend_break <= 1'b0;
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that trend_break is what stops a rate change being plotted as a measurement. A management system trending utilisation across a renegotiation sees a step change of 10× and has no way to know whether the traffic changed or the link did. Marking the discontinuity turns an alarming graph into an annotated one, and it is the only signal that carries the distinction.
Deliberately simplified: a fixed hold window. A design whose measurement windows are long needs the hold to cover at least one full window either side, which makes the constant a function of the measurement period rather than a number.
Production implication: ever_downshifted is separated from the change count and is sticky, because the direction matters. A link that negotiated up did so because a peer or a configuration improved. A link that negotiated down did so because something in the channel failed — a marginal pair, a degraded connector, a cable that was disturbed — and that is a physical finding that persists after the link has been restored. It is Chapter 3.3's margin argument surfacing as a discrete event.
8. Store-and-Forward Against Cut-Through, in Time
A store-and-forward device receives the whole frame before it starts transmitting, so it adds one full serialization time per hop. That is Chapter 7.2 §4's policy seen as a latency term rather than as a delivery contract.
A cut-through device begins as soon as it can decide, which is a few octets after the delimiter — the destination address, and whatever else the forwarding decision needs.
So the difference per hop is very nearly the whole serialization time, and it therefore shrinks with the line rate exactly as serialization does:
| Line rate | Store-and-forward penalty, maximum frame | Over five hops |
|---|---|---|
| 100 Mb/s | 121.4 µs | 607 µs |
| 1 Gb/s | 12.14 µs | 60.7 µs |
| 10 Gb/s | 1.214 µs | 6.07 µs |
| 100 Gb/s | 121.4 ns | 607 ns |
Read the last row against Section 3's propagation figures. At 100 Gb/s, five store-and-forward hops cost 607 ns — about the same as 120 metres of cable. In a data centre where the path is a few hundred metres, the forwarding policy and the cabling are comparable terms.
At 100 Mb/s the same five hops cost 607 microseconds, which is a thousand times more than any plausible cable length. The policy was the entire budget.
Which is the chapter's argument arriving at a design decision. Cut-through's complexity — Chapter 7.2's provisional delivery, the abort path, the client obligation — bought an enormous latency saving at 100 Mb/s and buys a modest one at 100 Gb/s. The engineering trade moved, and it moved because of arithmetic that has not changed since 1980.
9. RTL 5 — Measuring the Policy Difference
// SYNTHESIZABLE INSTRUMENTATION.
//
// Measures the per-hop latency of both forwarding policies from the same
// traffic, so the trade of Section 8 is a measurement rather than an
// estimate.
//
// The store-and-forward figure is easy: the whole frame, plus the
// device's own pipeline. The cut-through figure is the interesting one,
// because it depends on how much header the forwarding decision needs --
// which is a property of THIS device rather than of the frame, and which
// is the number a design can actually improve.
module sf_vs_ct_latency_compare
import serdelay_pkg::*;
#(
parameter int unsigned CNT_W = 40,
// Pipeline stages between the last needed header octet and the first
// transmitted octet. The part of cut-through latency a design owns.
parameter int unsigned CT_PIPELINE_CYCLES = 12
) (
input logic clk,
input logic rst_n,
input logic clear,
input line_rate_e rate,
input logic frame_done,
input logic [13:0] frame_octets,
// Octets of the frame the forwarding decision required. Typically the
// destination address and whatever else the lookup needs.
input logic [7:0] decision_octets,
output logic [CNT_W-1:0] sf_latency_ps,
output logic [CNT_W-1:0] ct_latency_ps,
output logic [CNT_W-1:0] saving_ps,
output logic result_valid,
output logic [CNT_W-1:0] avg_saving_ps,
output logic [CNT_W-1:0] worst_sf_ps,
// Saving as a fraction of the store-and-forward figure. The number
// that says whether cut-through is worth its complexity HERE, which is
// not the same question as whether it is worth it in general.
output logic [CNT_W-1:0] saving_permille
);
logic [CNT_W-1:0] sum_q, cnt_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
sf_latency_ps <= '0; ct_latency_ps <= '0; saving_ps <= '0;
result_valid <= 1'b0; avg_saving_ps <= '0; worst_sf_ps <= '0;
saving_permille <= '0; sum_q <= '0; cnt_q <= '0;
end else begin
result_valid <= 1'b0;
if (clear) begin
sum_q <= '0; cnt_q <= '0; avg_saving_ps <= '0;
// worst_sf_ps deliberately survives.
end else if (frame_done) begin
automatic int unsigned bp = bit_period_ps(rate);
automatic logic [CNT_W-1:0] sf =
CNT_W'(frame_octets) * CNT_W'(8) * CNT_W'(bp);
// The decision's header octets, plus the pipeline expressed in
// the same units. The pipeline is in CYCLES and the rest is in
// time -- Section 13's rejected property is exactly this
// conversion done wrongly.
automatic logic [CNT_W-1:0] ct =
CNT_W'(decision_octets) * CNT_W'(8) * CNT_W'(bp)
+ CNT_W'(CT_PIPELINE_CYCLES) * CNT_W'(8) * CNT_W'(bp);
sf_latency_ps <= sf;
ct_latency_ps <= ct;
saving_ps <= (sf > ct) ? (sf - ct) : '0;
result_valid <= 1'b1;
sum_q <= sum_q + ((sf > ct) ? (sf - ct) : '0);
cnt_q <= cnt_q + 1'b1;
if (sf > worst_sf_ps) worst_sf_ps <= sf;
saving_permille <= (sf == '0) ? '0
: ((((sf > ct) ? (sf - ct) : 40'd0) * 1000) / sf);
avg_saving_ps <= (cnt_q == '0) ? '0 : (sum_q / cnt_q);
end
end
end
endmoduleClassification: synthesizable instrumentation.
What it teaches: that saving_permille answers a different question from saving_ps, and it is the one a design should act on. An absolute saving of 12 µs sounds large and is meaningless without the total. A saving of 950 per mille says cut-through removes almost the whole per-hop cost; a saving of 200 per mille says the pipeline has grown until the policy barely matters — and that second case is common in a design that added stages over several revisions without revisiting the decision.
Deliberately simplified: it models the policies rather than measuring both. A device implements one, so the other figure is necessarily computed — and computing it from the same traffic is what makes the comparison honest.
Production implication: decision_octets is an input rather than a constant, and it is the number a design owns. The frame's size is given; the header the forwarding decision needs is a design choice — a device that looks only at the destination address decides in six octets, one that also parses a tag needs ten more, and one that inspects the payload has given up cut-through entirely. The cut-through latency is therefore mostly a statement about how deep this device looks, which is exactly the trade Chapter 2.4's payload opacity was protecting.
10. First Bit, Last Bit, and What a Budget Is Measuring
There is an ambiguity in "how long did the frame take" that Section 8's policy comparison depends on and that latency budgets routinely leave unresolved.
Three instants, and they differ by a whole serialization time.
The first bit arrives after the propagation delay alone. Nothing about the frame's size enters — a minimum frame's first bit and a maximum frame's first bit arrive at exactly the same moment after transmission starts.
The last bit arrives one serialization time later. This is the only interval in the whole budget that depends on the frame's size, which is why frame size is a latency lever at all.
The frame becomes usable after the check sequence has been verified — Chapter 6.3's residue, which by construction is at the end. So a store-and-forward device cannot act before this instant, and Chapter 7.2's cut-through device acts much earlier and accepts an obligation to withdraw.
So "latency" means at least three different numbers, and a budget that does not say which is comparing measurements that differ by 12 µs at gigabit and 1.2 ms at 10 Mb/s.
And the choice is not arbitrary — it follows from what the consumer does. A device that forwards can act on the first bit. A device that must inspect the whole frame cannot. The right instant to measure is the one at which the receiving stage can do its job, and different stages in the same path legitimately have different answers.
11. Assertions — Arithmetic, Units and the Rate That Moves
// ---------------------------------------------------------------------
// P1 -- THE DEFINITION. Serialization is bits divided by rate, expressed
// as bit-times multiplied by the bit period.
// ---------------------------------------------------------------------
property p_serialization_is_bits_over_rate;
@(posedge clk) disable iff (!rst_n)
rsp_valid |-> (serialization_ps ==
40'($past(frame_octets)) * 40'd8 * 40'(bit_period_ps($past(rate))));
endproperty
a_serialization_is_bits_over_rate: assert property (p_serialization_is_bits_over_rate);
// ---------------------------------------------------------------------
// P2 -- Occupancy exceeds serialization by exactly the wire overhead.
// The two quantities of Section 2, kept distinct.
// ---------------------------------------------------------------------
property p_occupancy_exceeds_serialization;
@(posedge clk) disable iff (!rst_n)
rsp_valid |-> ((occupancy_ps - serialization_ps) == overhead_ps);
endproperty
a_occupancy_exceeds_serialization: assert property (p_occupancy_exceeds_serialization);
// ---------------------------------------------------------------------
// P3 -- The overhead is 20 octets of wire time, whatever the frame size.
// ---------------------------------------------------------------------
property p_overhead_is_twenty_octets;
@(posedge clk) disable iff (!rst_n)
rsp_valid |-> (overhead_ps ==
40'(WIRE_EXTRA_OCTETS) * 40'd8 * 40'(bit_period_ps($past(rate))));
endproperty
a_overhead_is_twenty_octets: assert property (p_overhead_is_twenty_octets);
// ---------------------------------------------------------------------
// P4 -- Serialization scales exactly inversely with the rate. Written as
// a relation between two rates rather than as a value, so it holds for
// every frame size.
// ---------------------------------------------------------------------
property p_scales_inversely_with_rate;
@(posedge clk) disable iff (!rst_n)
(rsp_valid && ($past(rate) == RATE_1G))
|-> (serialization_ps == 40'($past(frame_octets)) * 40'd8 * 40'd1000);
endproperty
a_scales_inversely_with_rate: assert property (p_scales_inversely_with_rate);
// ---------------------------------------------------------------------
// P5 -- Resolution: a minimum frame at the highest rate must not round
// to zero or to an integer that loses its fractional nanoseconds.
// 5.12 ns is 5120 ps, and picoseconds are why it survives.
// ---------------------------------------------------------------------
property p_resolution_sufficient;
@(posedge clk) disable iff (!rst_n)
(rsp_valid && ($past(rate) == RATE_100G) && ($past(frame_octets) == 14'd64))
|-> (serialization_ps == 40'd5120);
endproperty
a_resolution_sufficient: assert property (p_resolution_sufficient);
// ---------------------------------------------------------------------
// P6 -- Wire octets always exceed frame octets by the overhead, per
// frame, in the accumulator.
// ---------------------------------------------------------------------
property p_accumulator_overhead;
@(posedge clk) disable iff (!rst_n)
win_valid |-> ((win_wire_octets - win_frame_octets) ==
(win_frames * CNT_W'(WIRE_EXTRA_OCTETS)));
endproperty
a_accumulator_overhead: assert property (p_accumulator_overhead);
// ---------------------------------------------------------------------
// P7 -- Payload never exceeds the frame that carried it.
// ---------------------------------------------------------------------
property p_payload_within_frame;
@(posedge clk) disable iff (!rst_n)
frame_done |-> (payload_octets <= frame_octets);
endproperty
a_payload_within_frame: assert property (p_payload_within_frame);
// ---------------------------------------------------------------------
// P8 -- The overhead fraction is a fraction.
// ---------------------------------------------------------------------
property p_overhead_fraction_bounded;
@(posedge clk) disable iff (!rst_n)
win_valid |-> (wire_overhead_permille <= CNT_W'(1000));
endproperty
a_overhead_fraction_bounded: assert property (p_overhead_fraction_bounded);
// ---------------------------------------------------------------------
// P9 -- A rate change forces recomputation. Nothing derived from the
// rate may survive it.
// ---------------------------------------------------------------------
property p_rate_change_pulses;
@(posedge clk) disable iff (!rst_n)
(rate_valid && (rate != rate_current) && have_q) |=> rate_changed;
endproperty
a_rate_change_pulses: assert property (p_rate_change_pulses);
// ---------------------------------------------------------------------
// P10 -- A rate change marks a trend break, so a discontinuity is not
// plotted as a measurement.
// ---------------------------------------------------------------------
property p_change_marks_trend_break;
@(posedge clk) disable iff (!rst_n)
rate_changed |=> trend_break;
endproperty
a_change_marks_trend_break: assert property (p_change_marks_trend_break);
// ---------------------------------------------------------------------
// P11 -- A downshift is recorded and sticky: it is a physical finding
// that persists after the link is restored.
// ---------------------------------------------------------------------
property p_downshift_sticky;
@(posedge clk) disable iff (!rst_n)
ever_downshifted |=> ever_downshifted;
endproperty
a_downshift_sticky: assert property (p_downshift_sticky);
// ---------------------------------------------------------------------
// P12 -- The budget's dominant term is genuinely the largest.
// ---------------------------------------------------------------------
property p_dominant_is_largest;
@(posedge clk) disable iff (!rst_n)
result_valid |-> (dominant_ps >= $past(t_serialization) &&
dominant_ps >= $past(t_propagation) &&
dominant_ps >= $past(t_store_forward) &&
dominant_ps >= $past(t_processing));
endproperty
a_dominant_is_largest: assert property (p_dominant_is_largest);
// ---------------------------------------------------------------------
// P13 -- The total is the sum of the terms. A budget whose total is
// computed separately can disagree with its own components.
// ---------------------------------------------------------------------
property p_total_is_sum;
@(posedge clk) disable iff (!rst_n)
result_valid |-> (total_ps == ($past(t_serialization) + $past(t_propagation) +
$past(t_store_forward) + $past(t_processing)));
endproperty
a_total_is_sum: assert property (p_total_is_sum);
// ---------------------------------------------------------------------
// P14 -- Cut-through latency never exceeds store-and-forward latency for
// the same frame. If it does, the decision needs more of the frame than
// the frame contains.
// ---------------------------------------------------------------------
property p_cut_through_not_worse;
@(posedge clk) disable iff (!rst_n)
result_valid |-> (ct_latency_ps <= sf_latency_ps);
endproperty
a_cut_through_not_worse: assert property (p_cut_through_not_worse);
// ---------------------------------------------------------------------
// P15 -- The saving is the difference, and the fraction is bounded.
// ---------------------------------------------------------------------
property p_saving_consistent;
@(posedge clk) disable iff (!rst_n)
result_valid |-> ((saving_ps == (sf_latency_ps - ct_latency_ps)) &&
(saving_permille <= CNT_W'(1000)));
endproperty
a_saving_consistent: assert property (p_saving_consistent);
// ---------------------------------------------------------------------
// P16 -- COVERAGE. Every rate exercised, and a rate change observed.
// ---------------------------------------------------------------------
c_rate_10m: cover property (@(posedge clk) disable iff (!rst_n) (rate == RATE_10M));
c_rate_100g: cover property (@(posedge clk) disable iff (!rst_n) (rate == RATE_100G));
c_rate_changed: cover property (@(posedge clk) disable iff (!rst_n) rate_changed);
c_downshift: cover property (@(posedge clk) disable iff (!rst_n) $rose(ever_downshifted));
// ---------------------------------------------------------------------
// P17 -- COVERAGE. Each term dominant at least once, because the whole
// argument of Section 3 is that which one dominates changes.
// ---------------------------------------------------------------------
c_dom_serialization: cover property (@(posedge clk) disable iff (!rst_n) (result_valid && dominant_term == T_SER));
c_dom_propagation: cover property (@(posedge clk) disable iff (!rst_n) (result_valid && dominant_term == T_PROP));12. Verification — Twenty-Four Scenarios and a Link That Changes Rate Underneath You
| # | Scenario | Stimulus | What must be observed |
|---|---|---|---|
| 1 | Minimum frame at 1 Gb/s | 64 octets | serialization_ps = 512 000 (512 ns) |
| 2 | Maximum frame at 1 Gb/s | 1518 octets | 12 144 000 ps (12.144 µs) |
| 3 | Minimum frame at 10 Mb/s | 64 octets | 51 200 000 ps (51.2 µs) |
| 4 | Minimum frame at 100 Gb/s | 64 octets | 5 120 ps — the resolution case (P5) |
| 5 | Maximum frame at 100 Gb/s | 1518 octets | 121 440 ps (121.44 ns) |
| 6 | Occupancy at 1 Gb/s, minimum | 64 octets | 672 000 ps — 31% above the frame's own time |
| 7 | Occupancy at 1 Gb/s, maximum | 1518 octets | 12 304 000 ps — 1.3% above |
| 8 | Overhead is size-independent | any two frame sizes | overhead_ps identical (P3) |
| 9 | Inverse scaling | one frame, five rates | each step is exactly 10× (P4) |
| 10 | Zero-length frame | frame_octets = 0 | serialization zero, occupancy = the overhead |
| 11 | Accumulator overhead | 1000 frames | wire minus frame octets = 1000 × 20 (P6) |
| 12 | Overhead fraction, small frames | 1000 minimum frames | wire_overhead_permille ≈ 238 |
| 13 | Overhead fraction, large frames | 1000 maximum frames | ≈ 13 |
| 14 | Payload excludes pad | short frames with padding | win_payload_octets below win_frame_octets (P7) |
| 15 | Budget: serialization dominant | 100 Mb/s, 100 m link | dominant_term = serialization, high per mille |
| 16 | Budget: propagation dominant | 100 Gb/s, 100 m link | dominant_term = propagation — the inversion |
| 17 | Budget: no clear dominant | four roughly equal terms | dominant reported, dominant_permille near 250 |
| 18 | Total is the sum | any sample | total_ps equals the four terms (P13) |
| 19 | Rate change 1 G to 100 M | renegotiate mid-run | rate_changed; trend_break (P9, P10) |
| 20 | Downshift recorded | the same change | ever_downshifted sets and is sticky (P11) |
| 21 | Upshift not recorded as downshift | 100 M to 1 G | rate_changed; ever_downshifted unchanged |
| 22 | Cut-through never worse | any frame | ct_latency_ps ≤ sf_latency_ps (P14) |
| 23 | Cut-through saving, large frames | 1518 octets, 6 decision octets | saving_permille above 990 |
| 24 | Cut-through saving, small frames | 64 octets, 6 decision octets | saving_permille far lower — the policy barely helps |
13. Debugging — When the Numbers Move and Nothing Broke
Symptom — utilisation jumped by 10× and traffic did not change.
The link renegotiated. rate_changed and trend_break mark it; ever_downshifted says which direction. A management system trending octets across the change sees a step and has no way to know whether the traffic or the link moved — which is exactly what trend_break exists to annotate.
Symptom — a link that keeps renegotiating.
Read c_rate_changes and rates_seen together. The set of rates the link has ever run at is more diagnostic than the count: a link oscillating between two adjacent rates has a channel that is marginal at the higher one, and ever_downshifted confirms the direction. It is Chapter 3.3's margin argument appearing as a discrete event rather than as an eye measurement.
Symptom — latency is worse than the budget predicted, and every term looks right.
Check whether the budget used serialization where it needed occupancy. Section 2's two quantities differ by 20 octets — 31% on a minimum frame — so a budget built from frame times underestimates the interval between frames by that much. The error is a rounding error on bulk traffic and a third on small frames, so it passes every test built from large transfers.
Symptom — optimising frame size for latency produces no measurable improvement.
At high rates it will not. Read dominant_term and dominant_permille: if propagation or per-frame processing dominates, serialization is not the lever and shaving it cannot help. Section 3's inversion means the intuition formed at lower rates points at the smallest term in the sum.
Symptom — cut-through was implemented and the latency barely improved.
Read saving_permille rather than saving_ps. A low fraction means the pipeline has grown until the policy hardly matters — decision_octets plus CT_PIPELINE_CYCLES has crept up over revisions, and the saving is now a small part of the store-and-forward figure. It is a design finding rather than a measurement error, and the fix is in the pipeline rather than in the policy.
Symptom — computed times disagree with measured ones by a few percent, only at the highest rate.
Resolution. At 100 Gb/s a minimum frame is 5.12 ns, and any calculation carrying nanoseconds rounds it to 5 — a 2.3% understatement on the most numerous frames on a saturated link. The bias is systematic and always in the same direction, so it looks like a consistent measurement offset rather than a rounding artefact.
Symptom — a device's own counters and the peer's disagree about link utilisation.
Check whether one of them counts the wire overhead. Preamble, delimiter and gap are on the wire and in no frame, so an octet counter cannot see them — and wire_overhead_permille is about 238 on minimum-size traffic. Two devices measuring different quantities will disagree by nearly a quarter, and both will be correct about what they measured.
14. Common Misconceptions
"Serialization delay is negligible."
The wrong model: a frame goes onto the wire essentially instantly.
What it costs: at low rates it is the entire budget — a maximum frame at 10 Mb/s takes 1.214 milliseconds, which is 2400 times a 100-metre link's propagation delay. A design ignoring it at those rates has ignored everything.
The corrected model: it is bits ÷ rate, and its significance depends entirely on which rate. It dominates at 10 Mb/s and is a rounding error at 100 Gb/s, and neither statement is generally true.
"Faster links make latency budgets simpler."
The wrong model: every term shrinks, so the budget gets easier.
What it costs: the budget's composition inverts, and effort goes to the term that used to matter. An engineer shaving frame sizes for latency at 100 Gb/s is optimising the smallest term in the sum.
The corrected model: serialization scales exactly inversely with the rate; propagation does not scale at all; per-frame processing scales far more slowly. So the terms that were negligible become dominant, and at 100 Gb/s one metre of cable outweighs an entire minimum-size frame.
"Serialization time is how long until the next frame can start."
The wrong model: one quantity serves both questions.
What it costs: a 31% underestimate on minimum-size frames, because the wire also carries eight octets of preamble and delimiter and twelve of interframe gap — 20 octets that are in no frame. On maximum-size frames the same error is 1.3%, so it passes every test built from bulk traffic.
The corrected model: serialization is the frame; occupancy is the wire slot. A latency budget wants the first; a frame-rate calculation wants the second.
"A bound of 12 144 cycles enforces a bound of 12.144 microseconds."
The wrong model: cycles and time are interchangeable.
What it costs: Section 11's rejected property. They are related by the line rate, autonegotiation makes the rate a run-time value, and after a downshift the cycle bound is ten times looser than the requirement — passing, while the requirement is missed by an order of magnitude.
The corrected model: denominate bounds in the unit the requirement uses. A pipeline's own depth really is measured in cycles and should say so; anything derived from the line rate must be recomputed when the rate changes, and a trend across the change must be marked as a discontinuity.
"Cut-through is worth it."
The wrong model: the policy's value is a property of the policy.
What it costs: complexity — Chapter 7.2's provisional delivery, the abort path, the client obligation — bought for a saving that may have become small.
The corrected model: the saving is nearly a whole serialization time, so it shrinks with the rate exactly as serialization does. At 100 Mb/s five store-and-forward hops cost 607 µs; at 100 Gb/s they cost 607 ns, comparable to 120 metres of cable. And saving_permille is the number that says whether it is worth it here — a design whose decision pipeline has grown gets a much smaller fraction than the arithmetic suggests.
15. Interview Reasoning
"What is serialization delay?"
The weak answer gives the formula and stops. The answer that ends the topic gives the formula and then the number that makes it interesting: a maximum-size frame takes 1.214 ms at 10 Mb/s and 121.4 ns at 100 Gb/s — four orders of magnitude — while propagation across a cable, at about 5 ns per metre, does not change with the rate at all. The payoff is that a latency budget's composition inverts across that range, so intuition formed at one rate points at the wrong term at another.
"At 100 Gb/s, what costs more: a minimum-size frame or a metre of cable?"
The cable, and only just — a minimum frame serializes in 5.12 ns and a metre of cable is about 5 ns. The strong answer notes the same comparison at 1 Gb/s, where a minimum frame equals 102 metres of cable, essentially the full length of a twisted-pair link. The same two quantities, and the ratio moved by a factor of a hundred, which is why a rack-scale latency budget at 100 Gb/s is mostly cable and mostly was not at gigabit.
"Why is 'twelve thousand cycles' a bad way to state a frame's transmission deadline?"
Because cycles and time are related by the line rate, and autonegotiation makes the rate a run-time value. After a downshift from 1 Gb/s to 100 Mb/s the same cycle count is ten times longer in seconds — the property passes and the requirement is missed by 10×. The complete answer adds the direction: the bound is loosest at the slowest rate, which is exactly when the absolute times are largest, so it relaxes precisely when it should tighten.
"How much of a link's time is spent on things no frame contains?"
About 24% on minimum-size traffic — eight octets of preamble and delimiter plus twelve of interframe gap, against a 64-octet frame — and about 1.3% on maximum-size traffic. The finishing point is that no octet counter in the system can see it, because those octets are in no frame, so two devices measuring "utilisation" differently will disagree by nearly a quarter and both will be right about what they measured.
16. Understanding Check
Because it scales exactly inversely with the rate and almost nothing else does.
A maximum-size frame is 12 144 bits. At 10 Mb/s that is 1.214 milliseconds; at 100 Gb/s it is 121.4 nanoseconds — a factor of ten thousand.
Propagation does not move at all. It is the medium's length divided by the signal's velocity, about 5 nanoseconds per metre, and the line rate does not appear in that expression. A 100-metre link costs roughly 500 ns at every rate Ethernet has ever run at.
So the comparison inverts. At 10 Mb/s, serialization beats a 100-metre link's propagation by 2400×. At 100 Gb/s, the propagation beats the serialization by 4×.
And the crossover is easy to state in cable length. The length whose propagation equals one frame's serialization is 243 km at 10 Mb/s, 2.4 km at 1 Gb/s, and 24 metres at 100 Gb/s — for a minimum frame at 100 Gb/s it is one metre.
17. What's Next
The claim this chapter defended: serialization is trivial arithmetic whose significance changes by four orders of magnitude, and the change inverts latency budgets.
bits ÷ rate gives 1.214 ms for a maximum frame at 10 Mb/s and 121.4 ns at 100 Gb/s. Propagation, at roughly 5 ns per metre, does not move at all. So the length of cable that costs as much as one frame falls from 243 km to 24 metres — and for a minimum frame at 100 Gb/s, to one metre. What dominated a budget completely has become a rounding error, and something nobody was watching now sets the number.
Along the way two quantities that are easy to conflate: serialization is the frame, occupancy is the wire slot, and they differ by 20 octets — 31% on a minimum frame, and a quarter of a small-frame link's time spent on octets no counter can see. And one unit trap: a bound in cycles stops meaning a bound in seconds the moment autonegotiation changes the rate, passing while the requirement is missed tenfold.
Chapter 8.2 — Propagation Delay owns the term this chapter kept comparing against. Propagation delay is set by the medium's length and its velocity factor, and it is independent of the data rate — which is exactly why it survived four orders of magnitude of progress untouched and now dominates.
8.2 derives the velocity factors for copper and fibre and finds something worth knowing: they are nearly the same, about 5 ns per metre in both, so fibre's advantage is reach and rate rather than speed. It connects the same physics back to Chapter 1.2's slot time — the round trip that sized the minimum frame is this chapter's propagation, measured across the maximum segment — and forward to Chapter 4.4, where a delay that is not the same in both directions becomes a measurable asymmetry.
The full path is on the Ethernet curriculum index.
Continue learning
Related tutorials
- Related topic
One Frame, End to End
A frame's journey down the stack and back up the other side, stage by stage. The transmit path decides and the receive path must discover — at four layers, not one — and that asymmetry is why the receive half of every Ethernet design is the larger, later and buggier one.
- Related topic
Forward Error Correction
FEC converts a gradual degradation into a cliff and hides the gradient behind it. The pre-correction error rate is the link's health metric and gives months of warning; the corrected output reads zero until the moment it collapses, and can be silently wrong when a decoder miscorrects.
- Related topic
The Receive Path
A transmitter assembles from settled quantities; a receiver discovers, and every decision stays provisional until the check sequence at the very end — so the pipeline either waits for the verdict or acquires the ability to withdraw what it has already delivered.
- Related topic
Propagation Delay
Length divided by velocity, about 5 ns per metre in copper and fibre alike, and completely independent of the data rate — which is why it survived four orders of magnitude untouched and now dominates the budgets it used to be invisible in.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.