Why Sub-Microsecond Sync Is a Hardware Problem

Chapter 15.3 closed on a device needing a fact it could not derive locally, and getting it by asking. This module is that problem in its hardest form, because the fact is a number and the act of asking changes it.

A clock cannot measure its own error. It can only measure its difference from another clock, and that difference is contaminated by everything the measurement crossed to arrive — a cable, a PHY, a MAC, a queue, an interrupt controller, a scheduler and a software stack.

The contamination is larger than the quantity being measured, by three orders of magnitude, and that single fact is the whole reason this is hardware.

Rows three and four are the problem and rows five and six are the answer. A mechanism whose measurement noise is a thousand times its target does not need a better algorithm — it needs to be moved to a place where the noise is not.

And that place is startlingly specific: a register that captures a free-running counter at the instant Chapter 5.2's Start Frame Delimiter crosses the MAC/PHY boundary. Everything above that point adds variable delay. Nothing below it does.

1. Scope — What This Chapter Owns

This chapter owns the requirement: what needs synchronised clocks, how far two free-running clocks diverge and how fast, why a software timestamp cannot deliver sub-microsecond accuracy, and where in the datapath the timestamp has to be taken instead.

It does not own the protocol. The Sync, Follow_Up, Delay_Req and Delay_Resp exchange and the offset and path-delay computation are Chapter 16.2. This chapter establishes what that exchange must be built on and does not build it.

It does not own the timestamp unit's integration. One-step against two-step operation, and the detailed placement at the MAC/PHY boundary, are Chapter 16.3. Section 13 here derives where the capture must happen and leaves how to that chapter.

It does not own the servo. How a slave clock is disciplined, and how residence time is corrected in transit, is Chapter 16.4. Section 4's distinction between an offset and a rate is what the servo acts on, and this chapter states it without implementing it.

And it does not own the error sources. Path asymmetry, timestamp granularity and jitter as the limits on achievable accuracy are Chapter 16.5. Section 15 prices granularity because a design must choose a capture clock before any of that chapter's analysis is available.

2. What Actually Needs a Synchronised Clock

The requirement varies by six orders of magnitude across applications that all describe themselves as needing accurate time, and the mechanism differs at every step.

Rows three and below are where this module lives, and the boundary at row two is sharp rather than gradual. Below 1 ms, software timestamping works and NTP is adequate. Above it — meaning tighter than it — every term in Section 9's budget matters and none of them can be argued away.

And the last row is worth stating because it bounds the module honestly: PTP with the best available hardware reaches tens of nanoseconds, and a requirement below that is met by distributing a clock signal rather than a protocol.

What all of rows three to seven have in common is that they need phase, not frequency.

A frequency requirement is that two clocks tick at the same rate. Two independent oscillators trimmed to the same rate satisfy it, and nothing needs to be exchanged at all.

A phase requirement is that two clocks agree on what time it is now. That cannot be established by any local means, and re-establishing it is what the entire module does. A motion controller's two axes running at identical rates but 50 µs apart in phase produce a machined part that is out of tolerance, and every clock in the system reads correct.

3. RTL 1 — A Free-Running Local Clock

Every device in this module holds a counter that represents its idea of the time, and its structure decides what a servo will later be able to do to it.

// -----------------------------------------------------------------------
// syncreq_pkg -- shared types for clock synchronisation requirements.
// -----------------------------------------------------------------------
package syncreq_pkg;

  // Time is carried as seconds plus nanoseconds, which is what PTP
  // uses and what every consumer of a timestamp expects.
  localparam int SEC_W = 48;
  localparam int NS_W  = 32;

  typedef struct packed {
    logic [SEC_W-1:0] sec;
    logic [NS_W-1:0]  ns;    // 0 .. 999_999_999
  } timestamp_t;

  localparam int NS_PER_SEC = 1_000_000_000;

  // The increment is carried in fixed point so the rate can be trimmed
  // in parts per billion. 16 fractional bits gives a resolution of
  // 1/65536 ns per tick, which at 156.25 MHz is 2.4 ppb.
  localparam int INC_INT_W  = 8;
  localparam int INC_FRAC_W = 16;

  typedef logic [INC_INT_W+INC_FRAC_W-1:0] increment_t;

  // What a servo may do to a clock. The distinction is section 4's and
  // it is the most important one in the module.
  typedef enum logic [1:0] {
    ADJ_NONE,
    ADJ_OFFSET,   // a step: jump the clock. Discontinuous.
    ADJ_RATE      // a trim: change the increment. Continuous.
  } adjust_e;

endpackage

// -----------------------------------------------------------------------
// free_running_clock -- the device's own idea of the time.
//
// Two adjustment paths, and they are NOT interchangeable:
//   a STEP fixes an offset instantly and makes time discontinuous;
//   a TRIM fixes a rate and converges slowly without ever going
//   backwards. Section 4 is about why a design needs both and uses
//   the second almost always.
// -----------------------------------------------------------------------
module free_running_clock
  import syncreq_pkg::*;
#(
  parameter int CLK_MHZ = 156           // 6.4 ns nominal period
)(
  input  logic       clk,
  input  logic       rst_n,

  input  adjust_e    adj,
  input  logic signed [NS_W:0] adj_offset_ns,   // for ADJ_OFFSET
  input  increment_t adj_increment,             // for ADJ_RATE

  output timestamp_t now,
  output increment_t increment,
  output logic [31:0] c_steps,
  output logic       stepped_backwards
);

  // Nominal increment: 1e9 / (CLK_MHZ * 1e6) nanoseconds per tick,
  // in INC_FRAC_W fixed point.  At 156 MHz that is 6.4102... ns.
  localparam increment_t NOMINAL_INC =
    increment_t'((NS_PER_SEC << INC_FRAC_W) / (CLK_MHZ * 1_000_000));

  logic [NS_W+INC_FRAC_W-1:0] ns_acc;
  logic [SEC_W-1:0]           sec_q;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ns_acc            <= '0;
      sec_q             <= '0;
      increment         <= NOMINAL_INC;
      c_steps           <= '0;
      stepped_backwards <= 1'b0;
    end else begin
      unique case (adj)
        ADJ_RATE: begin
          // A trim. Time stays monotonic and continuous; only its rate
          // changes. Everything downstream keeps working.
          increment <= adj_increment;
        end

        ADJ_OFFSET: begin
          // A step. Time jumps. Every consumer that assumed monotonic
          // time has just been lied to -- section 11's callout.
          c_steps <= c_steps + 1;
          if (adj_offset_ns < 0) stepped_backwards <= 1'b1;
          ns_acc  <= ns_acc +
                     ((NS_W+INC_FRAC_W)'(adj_offset_ns) << INC_FRAC_W);
        end

        default: begin
          ns_acc <= ns_acc + (NS_W+INC_FRAC_W)'(increment);
        end
      endcase

      // Roll nanoseconds into seconds.
      if (ns_acc[NS_W+INC_FRAC_W-1 -: NS_W] >= NS_W'(NS_PER_SEC)) begin
        ns_acc <= ns_acc -
                  ((NS_W+INC_FRAC_W)'(NS_PER_SEC) << INC_FRAC_W);
        sec_q  <= sec_q + 1'b1;
      end
    end
  end

  assign now.sec = sec_q;
  assign now.ns  = ns_acc[NS_W+INC_FRAC_W-1 -: NS_W];

endmodule

Classification: a fixed-point phase accumulator with two adjustment paths. No datapath, read by everything.

What it teaches: that the fractional increment is what makes a rate correctable at all. An integer nanosecond increment at 156 MHz would be 6 or 7 — the true value is 6.4103 — so the clock would run 6.4% fast or slow with no way to trim it. Sixteen fractional bits give a resolution of 1/65536 ns per tick, which at 156.25 MHz is 2.4 parts per billion — four orders of magnitude finer than the 20 ppm oscillator it is correcting.

And it teaches the distinction the whole module rests on. ADJ_RATE changes how fast time advances and time remains monotonic; ADJ_OFFSET moves time and may move it backwards. A servo uses the first continuously and the second essentially never — because a consumer that recorded an event at T and sees the clock read T − 40 µs afterwards has an ordering violation it cannot detect.

Deliberately simplified: the second rollover is a comparison and a subtract in the same always block as the increment, so a step large enough to cross several seconds rolls over one second per cycle. Production designs handle the seconds field arithmetically. A 40 µs step never triggers it; an initial time-of-day set does, which is the one case where a step is legitimate.

Production implication: stepped_backwards is sticky and it is the flag a systems engineer needs and never gets. A log file whose timestamps go backwards is a log file whose event ordering is wrong, and no amount of analysis downstream recovers it. A device that steps its clock must record that it did, with the magnitude and the direction, because every measurement spanning that instant is invalid.

4. Two Clocks Diverge, and the Rate Is the Problem

Set two clocks to the same time and leave them. What happens next is decided by one number that appears on the oscillator's datasheet.

The first row is what is in most Ethernet devices, because Chapter 3.8's link establishment and Chapter 4.4's elastic buffer both tolerate ±100 ppm comfortably — the PHY does not care, so nothing else in the design asked for better.

And 100 µs per second is the number that makes the whole module necessary. Section 2's requirement is 1 µs. The clock loses that much in ten milliseconds.

Which turns the requirement into a resynchronisation interval:

Read the first column and the mechanism's shape falls out. Holding 1 µs on a commodity crystal means synchronising a hundred times a second, for ever, on every device. That is a protocol running continuously rather than a calibration performed occasionally, and it is why PTP's default Sync interval is measured in fractions of a second rather than in minutes.

And read the last column for the other half of the design space. A ±0.1 ppm oscillator holds 1 µs for ten seconds, so a device with a good oscillator needs the network a hundredth as often — which matters enormously when the network is the thing that fails. The oscillator is not an alternative to synchronisation; it is what determines how gracefully the device behaves when synchronisation stops.

And the same table read the other way explains why cheap devices synchronise constantly and expensive ones do not. A gateway with a 100 ppm crystal that loses its time source is 1 µs out after 10 ms and 8.64 seconds out after a day; one with an OCXO is 8.6 ms out after a day. The first is useless as a time source the moment the network hiccups and the second rides out a maintenance window — which is the same component decision seen from availability rather than from accuracy.

Which also explains why the resync interval is stated as a maximum rather than a target. A device may synchronise more often than Section 6's arithmetic requires and gains nothing by it beyond faster recovery from a lost reference; synchronising less often violates the accuracy budget silently, because nothing in the device measures the error it is accumulating between corrections.

Which is the holdover requirement, and it is usually the reason an OCXO is specified at all. A base station that must stay within 1 µs for four hours after losing its time source needs 1 µs / 14 400 s = 0.07 ppb of stability — far beyond any of the rows above — so real holdover specifications are met by an oscillator plus a model of its ageing, not by an oscillator alone.

5. RTL 2 — Measuring Drift

Drift is measurable locally against nothing, which is the point. This module measures it against a reference that arrives from outside, and its structure is what makes the distinction between offset and rate visible.

// -----------------------------------------------------------------------
// drift_accumulator -- estimates the local clock's rate error from a
// sequence of reference timestamps.
//
// It cannot measure the local clock's ERROR -- section 19's rejected
// property. What it can measure is the RATE at which the difference
// between local and reference is growing, which is a difference of
// differences and is therefore immune to a constant path delay.
// -----------------------------------------------------------------------
module drift_accumulator
  import syncreq_pkg::*;
#(
  parameter int HISTORY = 8
)(
  input  logic       clk,
  input  logic       rst_n,

  // A paired observation: our clock, and the reference's, for the
  // same event. The pairing's quality is section 9's subject.
  input  logic       sample_valid,
  input  timestamp_t local_ts,
  input  timestamp_t ref_ts,

  output logic signed [NS_W:0] offset_ns,      // instantaneous difference
  output logic signed [31:0]   drift_ppb,      // rate error
  output logic                 drift_valid,
  output logic [31:0]          c_samples,
  output logic signed [NS_W:0] offset_spread_ns  // max - min over HISTORY
);

  logic signed [NS_W:0] hist [HISTORY];
  logic [NS_W-1:0]      t_hist [HISTORY];
  logic [$clog2(HISTORY):0] fill;
  logic [$clog2(HISTORY)-1:0] wp;

  logic signed [NS_W:0] diff;
  always_comb begin
    diff = $signed({1'b0, ref_ts.ns}) - $signed({1'b0, local_ts.ns})
         + $signed(NS_W'(NS_PER_SEC)) *
           $signed((NS_W+1)'(ref_ts.sec - local_ts.sec));
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      for (i = 0; i < HISTORY; i++) begin hist[i] <= '0; t_hist[i] <= '0; end
      fill <= '0; wp <= '0;
      offset_ns <= '0; drift_ppb <= '0; drift_valid <= 1'b0;
      c_samples <= '0; offset_spread_ns <= '0;
    end else if (sample_valid) begin
      automatic logic signed [NS_W:0] mx, mn;
      offset_ns <= diff;
      hist[wp]   <= diff;
      t_hist[wp] <= local_ts.ns;
      wp         <= wp + 1'b1;
      c_samples  <= c_samples + 1;
      if (fill != HISTORY) fill <= fill + 1'b1;

      // The DRIFT is how fast the offset is changing, so a constant
      // path delay -- however large -- contributes nothing to it.
      // This is the one quantity a device can estimate honestly.
      if (fill == HISTORY) begin
        automatic logic signed [NS_W+8:0] d_off;
        automatic logic [NS_W-1:0]        d_t;
        d_off = hist[wp] - hist[(wp + 1) % HISTORY];
        d_t   = t_hist[wp] - t_hist[(wp + 1) % HISTORY];
        drift_ppb   <= (d_t == 0) ? 32'sd0
                     : 32'sd1000000000 * 32'(d_off) / 32'(d_t);
        drift_valid <= 1'b1;
      end

      // The SPREAD is the measurement's own noise. It bounds what any
      // servo built on these samples can achieve -- section 11.
      mx = hist[0]; mn = hist[0];
      for (i = 1; i < HISTORY; i++) begin
        if (hist[i] > mx) mx = hist[i];
        if (hist[i] < mn) mn = hist[i];
      end
      offset_spread_ns <= mx - mn;
    end
  end

endmodule

Classification: a difference estimator over a short history, plus a noise estimator. Arithmetic only, no control.

What it teaches: that drift is a difference of differences and is therefore immune to a constant path delay, while offset is not. A 50 µs cable-and-stack delay adds 50 µs to every offset_ns sample and exactly zero to drift_ppb, because it cancels in the subtraction. So a device can estimate its rate error honestly over a path it knows nothing about, and cannot estimate its phase error at all without knowing that path's delay.

And it teaches that offset_spread_ns is the most honest output in the module. It is the observed noise of the measurement, and no servo can converge tighter than it. A path whose samples spread over 200 µs cannot deliver 1 µs of accuracy by any amount of filtering — averaging reduces the noise of the estimate and does nothing about the fact that each individual sample was taken at an unknown instant.

Deliberately simplified: the drift is a two-point difference across the history, which is a crude derivative and noisy. Production servos use a least-squares fit or a PI loop over many more samples — Chapter 16.4's subject — and the improvement is real. What does not improve is offset_spread_ns, which is a property of the path rather than of the estimator.

Production implication: drift_ppb on a device with no reference at all is the number a holdover model needs. A clock that has been observed at −18 ppb for an hour and then loses its reference can keep applying −18 ppb, which extends Section 4's holdover by however long the drift itself is stable. A device that discards its drift estimate when the reference goes away has thrown out the only thing that would have helped.

6. The Drift Arithmetic — ppm Into Seconds

Parts per million is a fraction and every requirement in this module is a duration, so the conversion is done constantly and is worth having as a reflex.

drift × interval = error. That is all of it, and the three forms it takes are:

And the third form is the one that produces the surprising numbers.

Rows three and four are why holdover is a hard requirement rather than a component choice. No oscillator in Section 4's table is stable to a fraction of a part per billion, so a four-hour holdover is met by measuring the oscillator's drift while the reference is available and continuing to apply it — which is exactly what Section 5's drift_ppb is for.

And the same arithmetic run the other way explains the protocol's message rate. PTP's default Sync interval is one per second; at 100 ppm that bounds the accumulated drift between messages at 100 µs, which is far above Section 2's requirement. So a 1 µs deployment on commodity crystals runs Sync at 8, 16 or 64 messages per second — and the message rate is set by the oscillator, not by the network or the application.

Which gives a small design rule that is frequently inverted: choose the oscillator first, and the sync rate follows. A design that fixes the sync rate first has implicitly specified an oscillator, and usually a more expensive one than it needed.

7. RTL 3 — The Software Timestamp Path, Modelled

A model rather than an implementation, because the point is to make the path's variability explicit and then measure it.

// -----------------------------------------------------------------------
// sw_timestamp_model -- models the delay between a frame arriving on
// the wire and software reading a clock for it.
//
// Six stages. Five of them are variable, and the variability is the
// entire subject: a constant delay of any size is removable by a
// servo, and a variable one is not.
// -----------------------------------------------------------------------
module sw_timestamp_model
  import syncreq_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       frame_arrived,      // SFD crossed the boundary
  input  timestamp_t now,                // the true time at that instant

  // Per-stage delays in nanoseconds, driven by the testbench from a
  // distribution rather than a constant -- section 10.
  input  logic [31:0] d_nic_to_irq,
  input  logic [31:0] d_irq_latency,
  input  logic [31:0] d_isr_to_read,
  input  logic [31:0] d_sched_wake,
  input  logic [31:0] d_stack,
  input  logic [31:0] d_clock_read,

  output logic       sw_ts_valid,
  output timestamp_t sw_ts,              // what software records
  output logic [31:0] total_delay_ns,
  output logic [31:0] min_delay_ns,
  output logic [31:0] max_delay_ns,
  output logic [31:0] jitter_ns          // max - min: the irreducible part
);

  logic [31:0] pending;
  logic        armed;
  timestamp_t  true_ts;

  logic [31:0] total;
  assign total = d_nic_to_irq + d_irq_latency + d_isr_to_read
               + d_sched_wake + d_stack + d_clock_read;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending <= '0; armed <= 1'b0; sw_ts_valid <= 1'b0;
      sw_ts <= '0; true_ts <= '0; total_delay_ns <= '0;
      min_delay_ns <= 32'hFFFF_FFFF; max_delay_ns <= '0; jitter_ns <= '0;
    end else begin
      sw_ts_valid <= 1'b0;

      if (frame_arrived && !armed) begin
        armed   <= 1'b1;
        pending <= total;
        true_ts <= now;                 // the value we WANTED to record
      end else if (armed) begin
        if (pending <= 32'd7) begin
          // Software finally reads the clock. It reads NOW, which is
          // the arrival time plus everything above.
          armed          <= 1'b0;
          sw_ts_valid    <= 1'b1;
          sw_ts          <= now;
          total_delay_ns <= total;
          if (total < min_delay_ns) min_delay_ns <= total;
          if (total > max_delay_ns) max_delay_ns <= total;
          jitter_ns      <= ((total > max_delay_ns) ? total : max_delay_ns)
                          - ((total < min_delay_ns) ? total : min_delay_ns);
        end else begin
          pending <= pending - 32'd7;   // ~6.4 ns per tick at 156 MHz
        end
      end
    end
  end

endmodule

Classification: a delay model with a running min/max. It computes nothing the design uses and measures the thing the design must know.

What it teaches: that the timestamp software records is not the arrival time plus a constant — it is the arrival time plus a sample from a distribution, and the sample is not knowable. true_ts is captured in the model purely so a testbench can compare; in a real system nobody has it, which is why the error cannot be corrected after the fact.

And it teaches that jitter_ns is the only output that matters. total_delay_ns can be a hundred microseconds and cost nothing — Section 9's callout: a constant delay is an offset and a servo removes it. A jitter of ten microseconds costs ten microseconds of accuracy and no servo, filter or averaging removes it, because each sample is displaced by an amount nothing observed.

Deliberately simplified: the six stages are summed and applied as one delay, which makes them independent. They are not: a loaded system correlates them — the same interrupt storm that delays the ISR also delays the scheduler — so the real distribution has a heavier tail than six independent draws would produce. The model understates the problem.

Production implication: the six inputs are exactly the six things a real measurement must decompose, and a system that reports only the total has no way to know which one to attack. In practice the ordering is stable: the scheduler wake dominates on a loaded host, interrupt latency dominates on a quiet one, and the clock read is never the problem — which means the common optimisation, a faster clock read via vDSO, addresses the smallest term.

8. Where a Software Timestamp Is Actually Taken

Trace the path a frame's arrival takes to reach a clock_gettime() call, and count the boundaries it crosses.

The third row deserves a paragraph on its own, because it is a feature working exactly as designed and destroying this measurement. Chapter 18.6's interrupt coalescing deliberately delays the interrupt so that several frames are handled together, which is the single most effective throughput optimisation in a NIC driver — and it inserts a delay of up to the coalescing timer, varying with how many frames arrived.

A host tuned for throughput is a host tuned to make this measurement worse, and the two goals are not reconcilable at this layer.

The row that dominates on a real system is the sixth. Waking a user thread requires the scheduler to run, and the scheduler runs when it runs. On an idle system that is microseconds; on a loaded one it is milliseconds; under a real-time scheduler with the thread pinned and priority-boosted it is tens of microseconds and still not deterministic.

And the whole of it is unnecessary, which is the chapter's argument. The event being timed happened at row one. Everything after row one is a transport problem — getting the news of the event to somebody who can write it down — and the news does not need to carry the time if the time was captured where the event was.

9. The Error Budget, Derived

Six terms, each with a typical value and a tail, and the tail is what the requirement has to survive.

And the number that decides everything is the difference between the two columns:

Jitter = 10 519 µs peak to peak. Section 2's requirement is 1 µs.

The measurement's noise exceeds the quantity being measured by four orders of magnitude.

Even the well-behaved case does not rescue it. Strip out the two pathological rows — assume a real-time kernel, a pinned thread, interrupts disabled nowhere for long — and the remaining four terms still span roughly 3 to 28 µs, a jitter of 25 µs against a 1 µs target. Twenty-five times too much, on a system tuned as hard as it can be tuned.

Which is the conclusion the chapter exists to establish, and it is stronger than "software is slow":

No amount of software optimisation reaches 1 µs, because the smallest achievable jitter of a path that crosses an interrupt and a scheduler is larger than 1 µs. The mean can be driven down; the spread cannot, and Section 11 explains why the spread is the only term that matters.

==

10. RTL 4 — Interrupt Latency as a Distribution

Section 9's tail column is where the requirement dies, and a single number cannot represent it. This module measures the shape.

// -----------------------------------------------------------------------
// irq_latency_model -- histograms the interrupt-to-timestamp interval
// so the TAIL is visible rather than averaged away.
//
// A mean is the wrong summary for this quantity: section 11. What a
// sync design needs is a percentile, and a percentile needs a
// histogram.
// -----------------------------------------------------------------------
module irq_latency_model
  import syncreq_pkg::*;
#(
  parameter int BINS     = 16,      // log2 buckets: 1 ns .. 32 us
  parameter int SAMPLES_W = 24
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       sample_valid,
  input  logic [31:0] latency_ns,

  output logic [SAMPLES_W-1:0] bin [BINS],
  output logic [31:0] p50_ns,
  output logic [31:0] p999_ns,
  output logic [31:0] worst_ns,
  output logic [SAMPLES_W-1:0] n_samples
);

  // A log2 bucket index: bin b covers [2^b, 2^(b+1)) nanoseconds.
  function automatic int unsigned log2_bin(input logic [31:0] v);
    int i;
    begin
      log2_bin = 0;
      for (i = 31; i >= 0; i--)
        if (v[i]) begin
          log2_bin = (i >= BINS) ? BINS-1 : i;
          break;
        end
    end
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    int b;
    if (!rst_n) begin
      for (b = 0; b < BINS; b++) bin[b] <= '0;
      p50_ns <= '0; p999_ns <= '0; worst_ns <= '0; n_samples <= '0;
    end else if (sample_valid) begin
      bin[log2_bin(latency_ns)] <= bin[log2_bin(latency_ns)] + 1'b1;
      n_samples <= n_samples + 1'b1;
      if (latency_ns > worst_ns) worst_ns <= latency_ns;
    end else if (n_samples != 0) begin
      // Percentiles, recomputed continuously from the histogram. The
      // 99.9th is the one a sync requirement is written against,
      // because a sync protocol takes many samples per second and
      // WILL hit the tail.
      automatic logic [SAMPLES_W-1:0] cum, half, tail;
      automatic int i50, i999;
      cum = '0; i50 = 0; i999 = 0;
      half = n_samples >> 1;
      tail = n_samples - (n_samples / 1000);
      for (b = 0; b < BINS; b++) begin
        cum = cum + bin[b];
        if ((cum < half) && (b + 1 < BINS)) i50  = b + 1;
        if ((cum < tail) && (b + 1 < BINS)) i999 = b + 1;
      end
      p50_ns  <= 32'd1 << i50;
      p999_ns <= 32'd1 << i999;
    end
  end

endmodule

Classification: a log-bucketed histogram with continuously recomputed percentiles. Sixteen counters.

What it teaches: that log2 bucketing is the right shape for a quantity spanning four orders of magnitude, and it costs sixteen counters. Linear bucketing over the same range would need thousands of bins to resolve the low end, or would put every interesting sample in one bin. A log bucket's relative resolution is constant — each bin is a factor of two — which is exactly right for a distribution described in orders of magnitude.

And it teaches why the 99.9th percentile rather than the 99th, and why a percentile rather than a maximum. A sync protocol at 16 messages per second takes 1.38 million samples per day, so a 99.9th-percentile event occurs 1382 times a day — frequent enough to matter and to be measured. The maximum, by contrast, is one sample and is dominated by whatever happened once.

Deliberately simplified: the percentiles are recomputed on every idle cycle, which is a 16-iteration loop and wasteful. Production designs recompute on a window boundary or export the raw bins and let software do it — the histogram is the valuable part and the percentiles are a convenience.

Production implication: p50_ns against p999_ns is the single measurement that decides whether a software timestamping design can meet its requirement, and the ratio is the finding rather than either value. A path with p50 of 2 µs and p999 of 4 µs is usable at 10 µs accuracy. One with p50 of 2 µs and p999 of 500 µs is not usable at any accuracy below 500 µs, and its mean of 9 µs describes neither.

11. Why the Mean Does Not Help

Averaging is the instinctive response to a noisy measurement and it is the wrong one here, for a reason that is specific rather than general.

Averaging works when the noise is independent of the quantity being measured and zero-mean. Neither holds.

The noise is not zero-mean. Every term in Section 9's budget is a delay, and a delay is non-negative. So the timestamp is always late, never early — the distribution is one-sided, its mean is not zero, and averaging converges on the mean delay rather than on zero. That part is fine: the mean delay is an offset and Section 9's callout removes it.

The noise is not independent of the network's state. The same congestion that delays a PTP message in a switch queue also delays the receiving host's interrupt, because both are consequences of load. So the error is correlated with exactly the conditions under which the measurement is being taken, and averaging over a period of load converges on a biased value.

And the decisive one: averaging reduces the noise of the estimate and does nothing about the individual samples.

Row two is what a synchronisation requirement is actually about. This device's clock is within 1 µs of the reference is a statement about the clock now, not about the average of a hundred past measurements. A servo that has beautifully estimated a mean offset from noisy samples still has a clock whose instantaneous error is whatever the last correction plus the accumulated drift makes it.

And there is a hard floor that no filtering crosses. A servo corrects at some bandwidth; correcting faster tracks the drift better and admits more measurement noise, correcting slower rejects noise and lets drift accumulate. The two trade against each other, and the achievable accuracy is roughly the geometric mean of the measurement noise and the drift over the correction interval — so a 25 µs jitter cannot be filtered into 1 µs by slowing the loop, because slowing it lets Section 4's 100 ppm accumulate.

Which is the final form of the argument: the jitter has to be removed at its source, and its source is above the MAC.

==

12. RTL 5 — The Hardware Timestamp Point

Everything above collapses into a few lines of RTL once the capture is moved to the right place.

// -----------------------------------------------------------------------
// hw_timestamp_point -- captures the free-running clock at the instant
// the SFD crosses the MAC/PHY boundary.
//
// This is the whole fix. Section 9's 10.5 ms of jitter becomes one
// clock period of quantisation, because nothing between the event and
// the capture can vary.
// -----------------------------------------------------------------------
module hw_timestamp_point
  import syncreq_pkg::*;
(
  input  logic       clk,             // the MAC receive clock
  input  logic       rst_n,

  // From 4.2's reconciliation sublayer: the cycle on which the SFD
  // was recognised. 5.2 established that the SFD is the last octet
  // of the preamble and the frame's first defined instant.
  input  logic       sfd_detected,
  input  logic [2:0] sfd_lane,        // which lane within the word

  input  timestamp_t now,             // from the free-running clock

  // A fixed, characterised correction for everything BELOW this point:
  // the PHY's receive pipeline, 4.4's elastic buffer, and the cable.
  // Constant, therefore removable -- section 9's callout.
  input  logic [15:0] cfg_phy_delay_ns,
  input  logic [15:0] cfg_lane_ns,     // ns per lane position

  output logic       ts_valid,
  output timestamp_t ts,
  output logic [31:0] c_timestamps,
  output logic       overrun          // a second SFD before a read
);

  logic captured;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ts_valid <= 1'b0; ts <= '0; captured <= 1'b0;
      c_timestamps <= '0; overrun <= 1'b0;
    end else begin
      ts_valid <= 1'b0;

      if (sfd_detected) begin
        if (captured) overrun <= 1'b1;

        // Capture, then subtract the constant path below us. The
        // subtraction is arithmetic on a value already captured, so
        // it adds no variability -- it may take as many cycles as it
        // likes.
        ts.sec <= now.sec;
        ts.ns  <= now.ns
                - NS_W'(cfg_phy_delay_ns)
                - NS_W'(cfg_lane_ns) * NS_W'(sfd_lane);

        ts_valid     <= 1'b1;
        captured     <= 1'b1;
        c_timestamps <= c_timestamps + 1;
      end
    end
  end

endmodule

Classification: a capture register. One flop's worth of decision and the entire chapter's argument.

What it teaches: that the fix is a location, not an algorithm. There is no filtering here, no estimation, no state machine — the clock is sampled at the instant the event occurs, and everything Section 8's table listed happens afterwards to a value that is already recorded. The interrupt may be late by a millisecond; the timestamp is not.

And it teaches that sfd_lane matters and is the term most often forgotten. Chapter 10.6's XGMII presents four or eight octets per clock, so the SFD may sit in any lane of the word — and a design that timestamps the whole word introduces a quantisation of one word rather than one octet. At 10 Gb/s a 32-bit word is 3.2 ns, so ignoring the lane costs up to 3.2 ns of error on a budget where 10 ns is the target.

Deliberately simplified: cfg_phy_delay_ns is a single constant, and a real PHY's delay depends on its mode, its FEC configuration — Chapter 3.7 — and sometimes on temperature. Production designs carry a table indexed by operating mode, and the values come from the PHY vendor's characterisation rather than from measurement, because measuring them requires an instrument better than the device. Which is Section 19's rejected property in its practical form.

Production implication: overrun catches the case that quietly loses timestamps. A capture register with no queue behind it drops the second of two closely spaced timestamped frames, and a PTP implementation whose Delay_Req timestamp silently belongs to a different frame computes a path delay that is wrong by whatever separated them. Production designs use a small FIFO of (timestamp, frame identifier) pairs, and the identifier is what makes the pairing checkable rather than assumed.

13. Where in the Datapath the Timestamp Must Be Taken

Section 12 asserted a location. This is the argument for it, made by walking down the stack and asking what each layer adds.

The fifth row is the trap and it looks reasonable. Timestamping when the frame is complete is easy — the MAC already knows — and it makes the timestamp depend on the frame's length. A 64-octet frame and a 1518-octet frame differ by 11.6 µs at 1 Gb/s, and PTP messages are short while the traffic around them is not. The dependence is systematic rather than random, which makes it worse: it does not average out.

The sixth and seventh rows are where real designs sit, and the difference between them is the PHY.

Chapter 4.4's elastic buffer is the specific problem. It absorbs the ±100 ppm difference between the recovered receive clock and the local one by inserting or deleting idle symbols, and its occupancy varies. So the delay from the wire to the MAC is not constant — it varies by up to the buffer's depth, which is why row seven's ~10 ns is a variation rather than a delay.

And row eight is the honest limit. The event that actually defines the frame's arrival is a transition on the wire, and the closest a digital design can get to it is the PHY's own clock domain. Everything above that is characterisable and everything below is analog — which is why Chapter 16.5's accuracy analysis begins at the PHY and not at the MAC.

==

The design rule that falls out is short and it decides the whole architecture:

Capture at the earliest point whose delay to the wire is characterisable, and correct for that delay as a constant. Not the fastest point, not the most convenient point — the earliest one whose remaining path does not vary.

Which is the MAC/PHY boundary at the SFD, and it is why Chapter 16.3 is a chapter about a boundary rather than about an algorithm.

14. RTL 6 — Timestamp Granularity

Having captured at the right place, the remaining error is what one clock period costs, and it is worth measuring rather than assuming.

// -----------------------------------------------------------------------
// timestamp_quantizer -- models and measures the error introduced by
// sampling a continuous event with a discrete clock.
//
// The event happens at an arbitrary instant; the capture happens on a
// clock edge. The difference is uniformly distributed over one period,
// so its standard deviation is period / sqrt(12).
// -----------------------------------------------------------------------
module timestamp_quantizer
  import syncreq_pkg::*;
#(
  parameter int CAPTURE_MHZ = 156      // 6.4 ns period
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       event_valid,
  input  logic [15:0] event_frac_ps,   // where within the period, 0..period
  input  timestamp_t captured_ts,

  output logic [15:0] quant_error_ps,
  output logic [15:0] max_error_ps,
  output logic [31:0] sum_error_ps,
  output logic [31:0] n_events,
  output logic [15:0] period_ps,
  output logic [15:0] sigma_ps         // period / sqrt(12), precomputed
);

  localparam int PERIOD_PS = 1_000_000 / CAPTURE_MHZ;
  // sqrt(12) = 3.4641; period / 3.4641, in the same units.
  localparam int SIGMA_PS  = (PERIOD_PS * 1000) / 3464;

  assign period_ps = 16'(PERIOD_PS);
  assign sigma_ps  = 16'(SIGMA_PS);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      quant_error_ps <= '0; max_error_ps <= '0;
      sum_error_ps <= '0; n_events <= '0;
    end else if (event_valid) begin
      // The capture lands on the NEXT edge, so the error is the time
      // remaining in the period. Always positive, never zero-mean --
      // section 11's point, at a much smaller scale.
      quant_error_ps <= 16'(PERIOD_PS) - event_frac_ps;
      if ((16'(PERIOD_PS) - event_frac_ps) > max_error_ps)
        max_error_ps <= 16'(PERIOD_PS) - event_frac_ps;
      sum_error_ps <= sum_error_ps + 32'(16'(PERIOD_PS) - event_frac_ps);
      n_events     <= n_events + 1;
    end
  end

endmodule

Classification: an error model with a running maximum and sum. It measures a quantity the design cannot reduce without changing its clock.

What it teaches: that quantisation error is uniform over one period, so its standard deviation is period / √12 — a constant this design should carry rather than rederive. At 156.25 MHz that is 1.85 ns, which is comfortably inside a 10 ns budget, and at 125 MHz it is 2.31 ns, which is also fine. The number only becomes a problem below about 5 ns of total budget.

And it teaches the same one-sided-noise point as Section 11 at a scale four orders of magnitude smaller. The error is always positive — the capture is always at or after the event — so its mean is half a period, not zero. That mean is an offset and is removable; the 1/√12 spread around it is not.

Deliberately simplified: event_frac_ps is an input, which means the testbench must know where within the period the event fell. A real design cannot know this, which is the entire content of Section 19's rejected property — and the practical consequence is that quantisation error is modelled and bounded, never measured in silicon.

Production implication: the choice of capture clock is a synchronisation decision that is usually made for other reasons. Chapter 10.3's GMII gives 125 MHz and 8 ns; Chapter 10.6's XGMII gives 156.25 MHz and 6.4 ns; a 25G lane clock gives 3.1 ns. So faster interfaces improve timestamp resolution as a side effect — and a design that needs better than the interface's period must run the capture counter on a faster clock than the datapath, which is a real and common technique and costs a clock domain crossing on a value that must not be resampled inaccurately.

15. What One Clock Period Costs

Put the granularity next to the budget it sits in, because the ranking is not what it looks like.

Every row fits inside a 10 ns budget, which is the useful finding: quantisation is not the binding constraint at any interface speed this track has covered. A design agonising over its capture clock is optimising a term that is already small.

And the terms that are binding, for comparison:

The last row is the one that dominates every real deployment and it is not a device problem at all. Chapter 12.6 established that a store-and-forward switch holds a frame for its entire duration before forwarding it, and that duration varies with the frame's length and the queue ahead of it. A PTP message crossing three such switches accumulates tens of microseconds of variable delay, which is worse than the software timestamping this chapter spent nine sections rejecting.

Which is why PTP does not simply run end to end. The switches in the path must measure and report how long they held each message — the residence-time correction, Chapter 16.4's subject — and a switch that does not is a switch that destroys the accuracy of everything behind it.

And it gives the chapter's last piece of architecture: sub-microsecond synchronisation is not a property of the two endpoints. It is a property of every device in the path, each of which must either be transparent or must account for itself. A single ordinary switch between a master and a slave puts the whole system back in the tens of microseconds, whatever the endpoints do.

16. RTL 7 — Sync Telemetry

Six numbers, and the useful ones describe the measurement rather than the clock.

// -----------------------------------------------------------------------
// sync_telemetry -- what an operator needs to decide whether a
// synchronisation deployment can meet its requirement.
//
// The unifying idea: report the achievable accuracy alongside the
// achieved one, so a device that is doing as well as its path allows
// can be distinguished from one that is not.
// -----------------------------------------------------------------------
module sync_telemetry
  import syncreq_pkg::*;
#(
  parameter int TARGET_NS = 1000       // the requirement, 1 us
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic signed [NS_W:0] offset_ns,        // section 5
  input  logic signed [NS_W:0] offset_spread_ns, // section 5's noise
  input  logic signed [31:0]   drift_ppb,
  input  logic [15:0] quant_sigma_ps,            // section 14
  input  logic [31:0] correction_interval_ns,
  input  logic        sample_valid,
  input  logic        window_tick,

  output logic [31:0] achievable_ns,     // the floor this path imposes
  output logic [31:0] achieved_ns,       // what we are managing
  output logic        target_feasible,
  output logic        performing_to_path,
  output logic [31:0] holdover_ns_per_s,
  output logic [31:0] c_out_of_spec
);

  logic signed [NS_W:0] worst_abs;

  function automatic logic [NS_W:0] absval(input logic signed [NS_W:0] v);
    absval = v[NS_W] ? (~v + 1'b1) : v;
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      achievable_ns <= '0; achieved_ns <= '0;
      target_feasible <= 1'b0; performing_to_path <= 1'b0;
      holdover_ns_per_s <= '0; c_out_of_spec <= '0; worst_abs <= '0;
    end else begin
      if (sample_valid) begin
        if (absval(offset_ns) > worst_abs) worst_abs <= absval(offset_ns);
        if (absval(offset_ns) > (NS_W+1)'(TARGET_NS))
          c_out_of_spec <= c_out_of_spec + 1;
      end

      if (window_tick) begin
        // The floor: measurement noise, plus drift accumulated between
        // corrections, plus quantisation. Section 11's trade, evaluated
        // at the interval actually in use.
        achievable_ns <= 32'(offset_spread_ns)
                       + ((32'(drift_ppb) * (correction_interval_ns / 1000))
                          / 1000000)
                       + 32'(quant_sigma_ps / 1000);

        achieved_ns <= 32'(worst_abs);
        worst_abs   <= '0;

        target_feasible <= (32'(offset_spread_ns) < 32'd1000);

        // Are we doing as well as the path permits? Within 2x of the
        // floor is as good as it gets; worse than that is OUR problem.
        performing_to_path <= (32'(worst_abs) <= (2 * achievable_ns));

        // What holdover will cost if the reference goes away.
        holdover_ns_per_s <= (32'(drift_ppb) < 0)
                           ? 32'(-drift_ppb) : 32'(drift_ppb);
      end
    end
  end

endmodule

Classification: an estimator that produces a judgement — performing_to_path — from four measured quantities.

What it teaches: that a synchronisation deployment needs its floor reported alongside its achievement, for the same reason Chapter 15.2 §9's distribution monitor needed an expectation. An offset of 8 µs is a failure on a path whose floor is 200 ns and is the best available on a path whose floor is 7 µs, and no threshold on the offset alone separates them.

And it teaches that target_feasible is answerable before any servo is tuned. The measurement's own noise — Section 5's offset_spread_ns — bounds everything downstream, so a path whose samples spread over 200 µs cannot deliver 1 µs and this is knowable from the first eight samples. Which stops a great deal of servo tuning on paths that could never have worked.

Deliberately simplified: achievable_ns sums three terms linearly where they should add in quadrature, so the estimate is conservative by up to 70%. That is the right direction for a floor — it never claims a path is better than it is — and a production design that wants the tighter figure needs a square root it does not otherwise have.

Production implication: holdover_ns_per_s is the measured drift restated as the number an availability requirement uses. A device reading 18 ns/s holds 1 µs for 55 seconds after losing its reference, and one reading 100 000 ns/s holds it for 10 milliseconds. The two need completely different responses to a network outage — the first rides it out and the second must alarm immediately — and the distinction is invisible until the drift is reported in the units the requirement is written in.

17. RTL 8 — Conformance for a Timestamp Unit

The monitor can check that the timestamp unit behaves. It cannot check that the timestamp is right, and the separation is the chapter's last structural point.

// -----------------------------------------------------------------------
// timestamp_conformance_monitor -- one bit.
//
// It asserts STRUCTURAL properties of the timestamp unit: monotonicity,
// pairing, capture discipline, configuration. It does NOT assert
// accuracy -- section 19's rejected property is exactly that, and the
// reason is that no instrument on this device could evaluate it.
// -----------------------------------------------------------------------
module timestamp_conformance_monitor
  import syncreq_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       ts_valid,
  input  timestamp_t ts,

  input  logic       overrun,           // section 12
  input  logic       stepped_backwards, // section 3
  input  logic       cfg_phy_delay_zero,   // uncharacterised PHY
  input  logic       cfg_lane_ignored,     // section 12's forgotten term
  input  logic       increment_is_integer, // no fractional trim
  input  logic       ts_unpaired,          // a timestamp with no frame

  output logic       conformant,
  output logic [7:0] fault_vector,
  output logic [31:0] c_nonmonotonic,
  output logic [31:0] c_overruns
);

  timestamp_t prev;
  logic       have_prev;
  logic       v_mono, v_over, v_step, v_phy, v_lane, v_inc, v_pair;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      prev <= '0; have_prev <= 1'b0;
      v_mono <= 1'b0; v_over <= 1'b0; v_step <= 1'b0;
      v_phy <= 1'b0; v_lane <= 1'b0; v_inc <= 1'b0; v_pair <= 1'b0;
      c_nonmonotonic <= '0; c_overruns <= '0;
    end else begin
      if (ts_valid) begin
        // Successive timestamps must not go backwards. This is
        // checkable without any external reference, which is exactly
        // why it belongs here and accuracy does not.
        if (have_prev && ((ts.sec < prev.sec) ||
                          ((ts.sec == prev.sec) && (ts.ns < prev.ns)))) begin
          v_mono         <= 1'b1;
          c_nonmonotonic <= c_nonmonotonic + 1;
        end
        prev      <= ts;
        have_prev <= 1'b1;
      end

      if (overrun) begin
        v_over     <= 1'b1;
        c_overruns <= c_overruns + 1;
      end
      if (stepped_backwards) v_step <= 1'b1;
      if (ts_unpaired)       v_pair <= 1'b1;

      // Standing configuration properties -- wrong from elaboration.
      v_phy  <= cfg_phy_delay_zero;
      v_lane <= cfg_lane_ignored;
      v_inc  <= increment_is_integer;
    end
  end

  assign conformant = !(v_mono || v_over || v_step ||
                        v_phy || v_lane || v_inc || v_pair);
  assign fault_vector = {1'b0, v_pair, v_inc, v_lane, v_phy,
                         v_step, v_over, v_mono};

endmodule

Classification: a fault aggregator with three standing configuration terms and four runtime ones. Every check is evaluable from local state alone.

What it teaches: that monotonicity is checkable without a reference and accuracy is not, and that distinction decides what belongs in this module. Successive timestamps going backwards is a structural violation — the counter did something impossible — and requires nothing external to detect. This timestamp is within 10 ns of true time requires knowing true time, which is precisely what the device does not have.

And it teaches that cfg_phy_delay_zero is a fault rather than a default. A timestamp unit with an uncharacterised PHY delay produces timestamps that are systematically early by the PHY's pipeline — 200 ns or so, twenty times the target — and every one of them is perfectly monotonic, perfectly paired and perfectly wrong. The bit exists because nothing else in the module would notice.

Deliberately simplified: ts_unpaired is an input and producing it needs the FIFO of (timestamp, frame identifier) pairs Section 12's simplification described. Without the identifier, a timestamp and a frame are associated by order, and an overrun breaks the ordering silently — every subsequent pairing is off by one, and the resulting path-delay calculations are wrong by whatever separated consecutive frames.

Production implication: conformant here means the timestamp unit is structurally sound and says nothing about the accuracy the system achieves. That is deliberate and it is the honest position: a device with a perfect timestamp unit behind an uncorrected store-and-forward switch delivers tens of microseconds, and Section 16's performing_to_path is where that shows up. Two outputs, two questions — is the unit correct, and is the deployment capable — and no single bit answers both.

18. Three Orders of Magnitude, Three Mechanisms

The whole chapter compresses into one table, and the compression is the point: each row is a different place, not a different algorithm.

Between rows one and four is a factor of a million, and not one step of it comes from making anything faster.

Every step is the removal of a variable stage from between the event and the capture. The scheduler. The interrupt. The frame's length. What remains at row four is the PHY, which varies by nanoseconds and can be characterised no further.

And row five is the module's remaining work. With the endpoints solved, the path becomes the limit — Chapter 12.6's store-and-forward delay, the asymmetry between the two directions of a fibre pair, and the residence time of every switch in between. None of those is an endpoint problem and all of them are larger than the endpoint's residual error.

==

And there is a fifth row the table cannot show, because it is not a capture point at all. Below the PHY's digital domain the event is a transition on a wire, and the closest anything gets to it is the analog slicer's decision instant — inaccessible to any digital design, and about 1 ns from the truth. Everything this module achieves is measured against that unreachable reference.

Which is worth stating because it bounds the ambition honestly. PTP with the best available hardware reaches tens of nanoseconds, and the gap between that and the 1 ns floor is not algorithmic — it is the elastic buffer, the quantisation, and the asymmetry, three terms that Section 15 has already priced and that Chapter 16.5 will price again with the protocol in place.

Which sets up the rest of Module 16 precisely. Chapter 16.2 builds the message exchange that measures the offset and the path delay. Chapter 16.3 places the timestamp unit at the boundary Section 13 derived. Chapter 16.4 closes the loop and corrects the residence time this section just named as the dominant error. And Chapter 16.5 returns to the asymmetry, which is the one term none of the others can remove.

19. Properties Worth Asserting, and One Worth Refusing

The properties divide by what they protect: the clock, the drift estimator, the software model, the capture, the granularity, and the configuration. Every one is evaluable from local state, which Section 17 established is not a small requirement here.

Group 1 — the clock.

// P1. Time is monotonic while no step is applied. Every consumer that
// records an ordering depends on this.
property p_time_is_monotonic;
  @(posedge clk) disable iff (!rst_n)
  (adj != ADJ_OFFSET) |=> ((now.sec > $past(now.sec)) ||
                           ((now.sec == $past(now.sec)) &&
                            (now.ns >= $past(now.ns))));
endproperty

// P2. A rate trim never moves time; it only changes how fast it moves.
property p_trim_does_not_step;
  @(posedge clk) disable iff (!rst_n)
  (adj == ADJ_RATE) |=> (now.ns >= $past(now.ns));
endproperty

// P3. Nanoseconds never reach one second: the rollover always fires.
property p_ns_in_range;
  @(posedge clk) disable iff (!rst_n)
  now.ns < NS_W'(NS_PER_SEC);
endproperty

// P4. A backwards step is always recorded. It invalidates every
// measurement spanning it and nothing downstream can detect it.
property p_backwards_step_recorded;
  @(posedge clk) disable iff (!rst_n)
  ((adj == ADJ_OFFSET) && (adj_offset_ns < 0)) |=> stepped_backwards;
endproperty

// P5. Every step increments the counter -- steps are rare and each one
// matters.
property p_steps_counted;
  @(posedge clk) disable iff (!rst_n)
  (adj == ADJ_OFFSET) |=> $changed(c_steps);
endproperty

// P6. The nominal increment is fractional. An integer increment cannot
// represent 6.4103 ns and the clock runs 6% off with no trim available.
property p_increment_is_fractional;
  @(posedge clk) disable iff (!rst_n)
  (NOMINAL_INC[INC_FRAC_W-1:0] != '0);
endproperty

Group 2 — the drift estimator.

// P7. Drift is immune to a constant path delay: adding a constant to
// every reference timestamp must not change the estimate.
property p_drift_ignores_constant_delay;
  @(posedge clk) disable iff (!rst_n)
  (sample_valid && shifted_by_constant) |=> $stable(drift_ppb);
endproperty

// P8. Offset is NOT immune -- and the asymmetry is the point.
property p_offset_tracks_constant_delay;
  @(posedge clk) disable iff (!rst_n)
  (sample_valid && shifted_by(k)) |=> (offset_ns == $past(offset_ns) + k);
endproperty

// P9. Drift is only published once the history is full.
property p_drift_needs_history;
  @(posedge clk) disable iff (!rst_n)
  drift_valid |-> (fill == HISTORY);
endproperty

// P10. The spread is a max minus a min and is therefore non-negative.
property p_spread_non_negative;
  @(posedge clk) disable iff (!rst_n)
  offset_spread_ns >= 0;
endproperty

// P11. A single sample cannot have a spread.
property p_spread_needs_two;
  @(posedge clk) disable iff (!rst_n)
  (c_samples <= 32'd1) |-> (offset_spread_ns == 0);
endproperty

Group 3 — the software model and the distribution.

// P12. The recorded timestamp is never EARLIER than the event. Every
// term in the path is a delay, so the noise is one-sided.
property p_sw_ts_is_never_early;
  @(posedge clk) disable iff (!rst_n)
  sw_ts_valid |-> (sw_ts.ns >= true_ts.ns);
endproperty

// P13. Jitter is max minus min and is what no servo removes.
property p_jitter_definition;
  @(posedge clk) disable iff (!rst_n)
  sw_ts_valid |=> (jitter_ns == (max_delay_ns - min_delay_ns));
endproperty

// P14. The histogram's bins sum to the sample count -- no sample is
// lost or double-counted.
property p_histogram_is_complete;
  @(posedge clk) disable iff (!rst_n)
  (bin.sum() == n_samples);
endproperty

// P15. Percentiles are ordered.
property p_percentiles_ordered;
  @(posedge clk) disable iff (!rst_n)
  (n_samples > 32'd0) |-> (p50_ns <= p999_ns) && (p999_ns <= worst_ns);
endproperty

// P16. A log bucket contains exactly the range it claims.
property p_bucket_range;
  @(posedge clk) disable iff (!rst_n)
  (sample_valid && (latency_ns inside {[(1<<b) : (1<<(b+1))-1]}))
    |=> $changed(bin[b]);
endproperty

Group 4 — the capture.

// P17. A timestamp is produced exactly when an SFD is detected, and
// never otherwise. This is the whole mechanism.
property p_capture_on_sfd;
  @(posedge clk) disable iff (!rst_n)
  sfd_detected |=> ts_valid;
endproperty

property p_no_spurious_capture;
  @(posedge clk) disable iff (!rst_n)
  ts_valid |-> $past(sfd_detected);
endproperty

// P18. The capture latency is EXACTLY one cycle, always. A variable
// capture latency reintroduces the jitter the whole chapter removed.
property p_capture_latency_is_fixed;
  @(posedge clk) disable iff (!rst_n)
  sfd_detected |-> ##1 ts_valid;
endproperty

// P19. The lane correction is applied. Ignoring it costs up to one
// XGMII word -- 3.2 ns at 10 Gb/s, on a 10 ns budget.
property p_lane_correction_applied;
  @(posedge clk) disable iff (!rst_n)
  (ts_valid && ($past(sfd_lane) != 3'd0)) |->
    (ts.ns != ($past(now.ns) - NS_W'(cfg_phy_delay_ns)));
endproperty

// P20. A second SFD before the first is read sets overrun. Silently
// overwriting breaks every subsequent pairing.
property p_overrun_is_flagged;
  @(posedge clk) disable iff (!rst_n)
  (sfd_detected && captured) |=> overrun;
endproperty

// P21. Successive captures are monotonic -- checkable with no
// external reference, which is why it is here.
property p_captures_monotonic;
  @(posedge clk) disable iff (!rst_n)
  (ts_valid && $past(ts_valid)) |-> (ts.ns >= $past(ts.ns)) ||
                                    (ts.sec > $past(ts.sec));
endproperty

Group 5 — granularity.

// P22. Quantisation error never exceeds one period.
property p_quant_bounded;
  @(posedge clk) disable iff (!rst_n)
  quant_error_ps <= 16'(PERIOD_PS);
endproperty

// P23. And is never negative -- the capture is at or after the event.
property p_quant_non_negative;
  @(posedge clk) disable iff (!rst_n)
  event_valid |=> (quant_error_ps >= 0);
endproperty

// P24. Sigma is period/sqrt(12), a constant of the uniform
// distribution rather than something measured.
property p_sigma_is_uniform_sigma;
  @(posedge clk) disable iff (!rst_n)
  (sigma_ps == 16'((PERIOD_PS * 1000) / 3464));
endproperty

Group 6 — telemetry and configuration.

// P25. The floor is never claimed to be better than the measurement
// noise alone. A conservative floor is the only safe kind.
property p_floor_at_least_noise;
  @(posedge clk) disable iff (!rst_n)
  window_tick |=> (achievable_ns >= 32'(offset_spread_ns));
endproperty

// P26. Feasibility is decided by the measurement's own spread, and is
// answerable before any servo is tuned.
property p_feasible_from_spread;
  @(posedge clk) disable iff (!rst_n)
  target_feasible |-> (offset_spread_ns < (NS_W+1)'(TARGET_NS));
endproperty

// P27. performing_to_path compares the achievement against the FLOOR,
// never against a fixed threshold.
property p_performance_is_relative;
  @(posedge clk) disable iff (!rst_n)
  performing_to_path |-> (achieved_ns <= (2 * achievable_ns));
endproperty

// P28. Standing property: the PHY delay is characterised. An
// uncharacterised PHY produces perfectly monotonic, perfectly paired,
// systematically wrong timestamps.
property p_phy_delay_characterised;
  @(posedge clk) disable iff (!rst_n)
  !cfg_phy_delay_zero;
endproperty

// P29. Standing property: the lane term is not ignored.
property p_lane_not_ignored;
  @(posedge clk) disable iff (!rst_n)
  !cfg_lane_ignored;
endproperty

// P30. Every timestamp is paired with an identified frame. Pairing by
// order alone breaks silently on an overrun.
property p_timestamps_are_paired;
  @(posedge clk) disable iff (!rst_n)
  !ts_unpaired;
endproperty

// P31. conformant is exactly the conjunction it claims.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant <-> (fault_vector == 8'h00);
endproperty

Every property above is a statement about this device's behaviour — monotonicity, latency, bounds, pairing, configuration — and not one of them says the timestamp is correct. That claim is the one this chapter refuses, and the reason is different from every earlier refusal in the series.

20. Verification Scenarios

Seventy scenarios. Several have expected outcomes in which every check passes and the clock is wrong by twenty times the requirement.

The clock

Drift

The software path

Averaging

The capture

Granularity and the path

The directed test random stimulus will not produce

Random stimulus cannot separate a constant path delay from a variable one, because separating them requires driving the same event through the same path twice with the variability switched off — which is a construction, not a distribution. And the finding it produces is the chapter's central claim: that a large constant delay is harmless and a small variable one is fatal. No coverage metric asks for a comparison between two runs.

Setup: one timestamp unit, one free-running clock at 156.25 MHz, one servo model with a 100 ms correction interval, and a 100 ppm local oscillator. A reference clock in the testbench holds true time, which Section 19 established exists only here.

Stimulus, three runs of 10 000 timestamped events each. Run A — constant delay: every event delayed by exactly 9 000 ns. Zero jitter. Run B — constant, large: every event delayed by exactly 10 000 000 ns — 10 ms. Zero jitter. Run C — variable, small: every event delayed by a value drawn uniformly from 9 000 to 34 000 ns — a mean of 21.5 µs and a jitter of 25 µs, which is Section 9's well-tuned case.

Oracle:

Rows 1, 2 and 11 together are the finding. Run B's delay is a thousand times Run C's mean and it meets the requirement while Run C does not — because Run B's delay is the same every time and Run C's is not.

Row 10 is the second finding and it is the reason performing_to_path exists as a separate output from the requirement. All three runs are performing as well as their path permits. The device is not at fault in any of them; in Run C the path simply cannot deliver 1 µs, and row 15's conformant says so by staying high.

And row 12 closes the escape. Filtering improves the estimate of the mean and leaves row 8 exactly where it was, which is Section 11 demonstrated rather than argued.

21. Debugging a Sync Problem

Five questions, in order. The first two are answered before looking at the servo, and they eliminate most investigations.

Step 1 — is the requirement feasible on this path? offset_spread_ns against the target. A path whose samples spread over 200 µs cannot deliver 1 µs by any means, and this is knowable from the first eight samples, before any tuning. target_feasible is the output and it is the cheapest thing in the module.

Step 2 — is the device performing to its path? achieved_ns against achievable_ns. An offset of 8 µs is a failure on a path whose floor is 200 ns and the best available on a path whose floor is 7 µs, and no threshold on the offset separates them. performing_to_path high with the requirement unmet means the path is the problem and the device is not.

Step 3 — where is the timestamp taken? This is a design question asked during an operational investigation, and it is usually the answer. Section 18's table spans a factor of a million and every step is a location. A deployment at ~50 µs is timestamping in the driver; one at ~12 µs is timestamping at frame completion; one at ~10 ms is timestamping in the application.

And a sixth check that belongs before all of them on a new deployment: what is the capture clock, and has the lane term been applied? Section 15 showed quantisation is never the binding constraint — 2.31 ns at 125 MHz, 1.85 ns at 156.25 — so a design worrying about it is optimising a term that is already small. The lane correction is the one to verify instead: an XGMII word is four or eight octets, so a design that timestamps the word rather than the octet quantises to 3.2 ns at 10 Gb/s on a budget where 10 ns is the target, and every check in Section 17 passes.

Step 4 — is the path corrected? A single uncorrected store-and-forward switch puts the system back in the tens of microseconds, whatever the endpoints do — Chapter 12.6's 12.14 µs at 1 Gb/s, varying with frame length and queue depth. Sub-microsecond synchronisation is a property of every device in the path, not of the two endpoints.

Step 5 — is anything systematically wrong? cfg_phy_delay_zero, cfg_lane_ignored, c_nonmonotonic, c_overruns. All four produce timestamps that are perfectly well formed and consistently wrong, and the first is the most common: an uncharacterised PHY delay is a 200 ns systematic error that every downstream check passes.

Those five, in that order, resolve almost every real case — and the ordering matters because steps 1 and 2 are answerable from telemetry alone, step 3 is a design fact, and only steps 4 and 5 require touching anything.

And the finding that ends an investigation without a fault: target_feasible high, performing_to_path high, conformant high, and the achieved error within the requirement. That is a synchronisation deployment working, and any remaining complaint is about the requirement rather than about the system.

22. Common Misconceptions

1 — "Software timestamping is too slow."

The wrong model: the path's latency is the problem.

What it costs: optimisation effort on the wrong term. A constant delay of any size is removable — Section 9's callout: a hypothetical system with a constant 10 ms scheduler delay would deliver perfect synchronisation, because the offset is measured once and subtracted for ever.

The corrected model: the problem is variability, not latency. Interrupt latency contributes 1 µs of delay and 499 µs of jitter; the scheduler contributes 5 µs and 9995 µs. Make the path repeatable, not fast — and since neither of those can be made repeatable, move the capture below them.

2 — "Averaging will fix a noisy timestamp."

The wrong model: noise averages out.

What it costs: a servo tuned indefinitely on a path that could never work. Averaging N samples reduces the uncertainty in the mean offset by 1/√N and leaves the uncertainty in any individual timestamp exactly where it was — and a synchronisation requirement is a statement about the clock now.

The corrected model: filtering trades against drift. At 100 ppm, slowing the loop from 0.1 s to 10 s takes the total error from ~30 µs to ~1002 µs, because the drift term grows linearly while the noise term falls as a square root. Both directions are bounded and their product is still above 1 µs.

3 — "A better crystal will fix it."

The wrong model: drift is the dominant error.

What it costs: an OCXO on a system whose limit is measurement noise. A 0.1 ppm oscillator with a 25 µs measurement jitter still delivers about 3 µs, because the noise term does not care what oscillator produced the clock it is measuring.

The corrected model: the oscillator sets how often you must synchronise and how long you can survive without a reference — Section 6's holdover. It does not set the accuracy achieved while synchronised, which is the measurement path's business.

4 — "Our endpoints do hardware timestamping, so we have sub-microsecond sync."

The wrong model: accuracy is an endpoint property.

What it costs: a deployment that measures tens of microseconds with perfect endpoints. A single uncorrected store-and-forward switch holds each message for 12.14 µs at 1 Gb/s, varying with the frame's length and the queue ahead of it — worse than the software timestamping the endpoints replaced.

The corrected model: sub-microsecond synchronisation is a property of every device in the path. Each intermediate switch must either be transparent or must measure and report its own residence time. One ordinary switch puts the whole system back in the tens of microseconds.

5 — "The timestamp unit passes all its assertions, so the timestamps are right."

The wrong model: conformance implies accuracy.

What it costs: a systematic error nothing detects. An uncharacterised PHY delay makes every timestamp 200 ns early — twenty times a 10 ns target — and every one of them is monotonic, correctly paired, produced in exactly one cycle and within its quantisation bound.

The corrected model: the timestamp unit's properties are structural and accuracy is not among them, because evaluating it needs an instrument better than the device. conformant says the unit is sound; performing_to_path says the deployment is capable; and the systematic terms are caught by standing configuration checks — cfg_phy_delay_zero exists precisely because nothing else would notice.

6 — "Stepping the clock to correct a large offset is fine — it is a one-off."

The wrong model: a step is a fast correction.

What it costs: every measurement spanning the step, silently. A consumer that recorded an event at T and then reads a clock showing T − 40 µs has an ordering violation it cannot detect, and no downstream analysis recovers it. A log whose timestamps go backwards is a log whose event ordering is wrong.

The corrected model: a step is legitimate exactly once, at initial time-of-day set. Everything afterwards is a rate trim, which converges more slowly and keeps time monotonic. And a device that does step must record it — magnitude, direction and instant — because every measurement crossing it is invalid and only the device knows.

23. Interview Reasoning

Q1 — Why can't software timestamping deliver 1 µs?

Because the smallest achievable jitter of a path crossing an interrupt and a scheduler is larger than 1 µs. The typical total is 9 µs and the tail is 10 528 µs, a jitter of 10.5 ms. Even a real-time kernel with a pinned thread leaves roughly 25 µs of spread. And the latency is not the problem — a constant delay of any size is an offset a servo removes. It is the spread, and the spread is irreducible because nothing observes which value each sample took.

Q2 — How far do two clocks drift apart, and what does that imply?

A ±100 ppm crystal — what most Ethernet devices carry, because Chapter 4.4's elastic buffer tolerates it — drifts 100 µs per second: 8.64 seconds a day. Against a 1 µs requirement that is 10 milliseconds of holding time, so synchronisation is a protocol running a hundred times a second rather than a calibration. And a better oscillator does not improve the achieved accuracy — it decides how often you must synchronise and how long you survive without a reference.

Q3 — Where must the timestamp be taken, and why there?

At the MAC/PHY boundary, at Chapter 5.2's Start Frame Delimiter. The rule is: capture at the earliest point whose remaining delay to the wire is characterisable — not the fastest point, the earliest repeatable one. Below that boundary is the PHY, whose pipeline is constant and removable and whose Chapter 4.4 elastic buffer contributes 5 to 20 ns of irreducible variation. Above it, every stage varies. Timestamping at frame completion instead makes the timestamp depend on the frame's length — 11.6 µs between a 64-octet and a 1518-octet frame at 1 Gb/s — and that error is systematic, so it does not average out.

Q4 — Why won't averaging or a slower servo rescue a noisy path?

Because filtering trades against drift, and both bounds are above the requirement. Averaging reduces the uncertainty in the mean as 1/√N and leaves each individual timestamp unchanged — and the requirement is about the clock now. Slowing the servo rejects more noise and lets more drift accumulate: at 100 ppm and 25 µs jitter, 0.1 s gives ~30 µs, 1 s gives ~106 µs, 10 s gives ~1002 µs. The best row is the fastest, and it is thirty times the target.

Q5 — Our endpoints timestamp in hardware and we measure 40 µs. What is wrong?

Almost certainly the path. A single uncorrected store-and-forward switch holds each PTP message for up to 12.14 µs at 1 Gb/s, varying with frame length and queue depth; three of them accumulate tens of microseconds. Sub-microsecond synchronisation is a property of every device in the path, each of which must be transparent or must measure and report its residence time. The second candidate is a systematic endpoint error — an uncharacterised PHY delay is 200 ns and an ignored XGMII lane term is up to 3.2 ns, both invisible to every structural check.

Q6 — Why can't you assert that a timestamp is accurate to 10 ns?

Because evaluating it needs an instrument better than the device under test, and the only clock available is the device under test. True time exists nowhere in the system — not in the local clock, not in a second local oscillator (which measures a difference), not in the reference's timestamps (contaminated by the path), and a GPS receiver on the board is simply a better DUT. What is assertable is the structural precondition: that the delay between the event and the capture is constant, which is checkable locally. Accuracy then follows from a constant delay plus a characterisation performed in a laboratory with an uncertainty attached — and the device's job is to guarantee the constancy, not to verify the result.

24. Understanding Check

25. What's Next

This chapter established a requirement and a location and built neither of the things that use them.

The requirement: 1 µs, against a clock that loses 100 µs per second and a measurement path whose noise is a thousand times the target. The location: a register that captures a free-running counter at the instant the SFD crosses the MAC/PHY boundary, because it is the earliest point whose remaining delay does not vary.

Everything else in Module 16 is built on those two facts.

Chapter 16.2 — The PTP Message Exchange builds the protocol: Sync, Follow_Up, Delay_Req and Delay_Resp, and the arithmetic that turns four timestamps into an offset and a path delay. Section 5's distinction — that drift survives an unknown path and offset does not — is why the exchange has four messages rather than one.

Chapter 16.3 — Hardware Timestamping at the MAC/PHY Boundary places the unit Section 12 sketched, and confronts the question this chapter deliberately left open: a Sync message must carry its own transmit timestamp, and that timestamp is not known until the message is already leaving. One-step and two-step operation are the two answers.

Chapter 16.4 — The Servo closes the loop, using Section 3's rate trim rather than its step, and corrects the residence time Section 15 named as the dominant error in any real path.

And Chapter 16.5 — What Limits Accuracy returns to the three terms nothing removes: path asymmetry, timestamp granularity and jitter.

One thread runs from Module 15 straight through all of it. Chapter 15.3 established that the cheapest information a distributed system can acquire is the information it cannot derive locally at any price — and that the way to get it is to ask. This module is the same problem with a harder object: the thing being asked for is a number, the act of asking changes it, and every chapter that follows is about accounting for the change.