NVMe SSD — One Command, Traced End to End

Module 26 characterised endpoint classes. Module 29 traces one real operation through one of them — and the useful output is not a description but a ledger: how many PCIe transactions a single command costs, and where its microseconds actually go.

1. Sources, Scope, and What This Chapter Refuses to Do

2. What "Trace One Command" Means

A description tells you the parts. A trace tells you the order, the count and the cost.

And the reason this is worth a chapter: the transaction count and the latency budget are the two facts that decide every optimisation argument about a storage path, and neither is visible from an architectural description. A team that cannot say where the microseconds go will optimise the stage that is easiest to change.

3. The Actors

One observation that shapes the whole trace. Software's only outbound PCIe transaction is the doorbell. Everything else — the command fetch, the payload write, the completion write, the interrupt — is initiated by the controller. The host writes four bytes and then waits.

4. The Trace

The ledger — count the PCIe transactions, because the count is the content.

Four transactions plus the payload writes, and one of them is a round trip.

Three readings.

Step 3 is the one that surprises people. The controller reads its own command, and that read is non-posted — so it costs a full round trip to host memory before the media access can even begin. On a small, low-queue-depth command that round trip is a significant fraction of the total (§5).

Steps 8 and 9 are two separate posted writes, and their order matters. The CQ entry must be visible before the MSI-X (26.3 §6 owns why, and 26.1 §7 owns the host-side half). This chapter counts them; those chapters explain them.

And step 12 is off the critical path, which is why batching it is free. Steps 2 and 3 are not, which is why batching them is the design's main lever (§10).

5. The Latency Budget

Sum the critical path from §4.

Four readings, and the last two are the actionable ones.

At this media latency the PCIe path is under 6 % of the command. A faster link cannot help a workload dominated by media — the same structural point 26.6 §6 makes for accelerators, arriving here through a completely different mechanism.

Now change one assumption and the conclusion inverts. Set the media read to 5 µs — a much faster storage class — and the total becomes 11.3 µs, of which PCIe is 3.8 µs, or 34 %, and interrupt delivery is another 18 %. The same architecture, the same trace, and the optimisation target has moved entirely. This is why the budget must be computed per deployment rather than inherited.

The command-fetch round trip is 1.5 of the 3.8 µs — the single largest PCIe item. And it is amortisable: a doorbell carries a tail pointer, so one fetch can retrieve several commands (26.3 §14). At a batch of 8, the per-command fetch cost falls to about 0.19 µs.

And interrupt delivery is 2.0 µs, which is 3 % at 60 µs media and 18 % at 5 µs media. Coalescing it is the second lever, and it is the one whose failure mode 26.4 §6 documents.

6. Where the Budget Comes From

The table in §5 is only worth something if the numbers are measured. That requires timestamps at stage boundaries, and choosing the boundaries is the design decision.

7. Wrong RTL — Timestamps at the Convenient Events

The design follows from picking the events that are easiest to observe.

// WRONG. ILLUSTRATIVE. Per-stage latency accumulators. Every line is
// reasonable-looking, and two event choices make the resulting budget
// systematically wrong in opposite directions.
logic [31:0] cyc_q;
logic [31:0] t_arrive_q [N_CMD];
logic [47:0] lat_fetch_q, lat_media_q, lat_total_q;
logic [31:0] cmd_count_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cyc_q <= '0; lat_fetch_q <= '0; lat_media_q <= '0;
    lat_total_q <= '0; cmd_count_q <= '0;
  end else begin
    cyc_q <= cyc_q + 32'd1;
 
    // BUG 1: T0 is taken when the FETCH is issued, not when the doorbell
    //        arrived. Every command's measured latency therefore excludes the
    //        doorbell-to-fetch interval — which is exactly where queueing
    //        inside the controller shows up.
    if (fetch_issue_fire) t_arrive_q[fetch_cmd_id] <= cyc_q;
 
    // Fetch stage measured from issue to completion — this part is fine.
    if (fetch_cpl_fire)
      lat_fetch_q <= lat_fetch_q + 48'(cyc_q - t_arrive_q[fetch_cpl_id]);
 
    // BUG 2: the total is closed when the MSI-X is EMITTED, not when the host
    //        has consumed the completion record. Interrupt delivery and ISR
    //        entry are excluded — the largest non-media item at low media
    //        latency (§5).
    if (msi_emit_fire) begin
      lat_total_q <= lat_total_q + 48'(cyc_q - t_arrive_q[msi_cmd_id]);
      cmd_count_q <= cmd_count_q + 32'd1;
    end
 
    // BUG 3: media latency accumulated with a separate NBA to the same
    //        register in a different branch. If a media completion and a
    //        fetch completion land in the same cycle, one update is lost.
    if (media_done_fire)
      lat_media_q <= lat_media_q + 48'(media_cycles);
  end
end

Architecture. Three accumulators and a per-command arrival timestamp, feeding a reported average latency.

State. t_arrive_q per command; three accumulators. The missing state is a timestamp at the doorbell, and a retirement event tied to host consumption.

Event. T0 on fetch issue; total closed on MSI emit. Both are the wrong events, and both are the easy ones to wire.

Contract. Whoever reads lat_total_q / cmd_count_q believes it is the command's latency as software experiences it. It is the latency of the middle of the pipeline.

Failure — the timeline. One command, under controller queueing.

First divergence: cycle 100 — the doorbell arrived and nothing recorded it. The reported figure is short by the queueing interval and by interrupt delivery, and both omissions are invisible in the report.

Root cause. The instrument measured the stages that were easy to observe rather than the interval software experiences. BUG 1 hides controller queueing — which is precisely the quantity that grows under load, so the measurement gets more optimistic as the system gets busier. BUG 2 hides interrupt delivery, which is 3 % at 60 µs media and 18 % at 5 µs (§5).

And BUG 3 is a separate, quieter defect. Two if branches writing lat_media_q… in fact only one branch writes it here, but the pattern generalises: accumulators updated from multiple branches lose an update on coincidence (23.3), and a latency total that drifts low is indistinguishable from a fast device.

DV/debug. The symptom is a reported latency that does not match host-side measurement, and the gap widens with queue depth. That divergence is the tell: an instrument whose error is load-dependent is measuring the wrong interval, not measuring imprecisely.

8. Corrected RTL — Measure the Interval Software Experiences

// CORRECT. ILLUSTRATIVE. Three changes: T0 at doorbell arrival, retirement at
// host consumption, and every accumulator updated from a single expression.
logic [31:0] cyc_q;
logic [31:0] t_db_q     [N_CMD];      // T0 — doorbell arrival
logic [31:0] t_fetch_q  [N_CMD];      // stage boundary
logic [31:0] t_media_q  [N_CMD];
logic        cmd_live_q [N_CMD];
logic [3:0]  cmd_gen_q  [N_CMD];      // which use of this slot
 
logic [47:0] acc_queue_q, acc_fetch_q, acc_media_q, acc_post_q, acc_total_q;
logic [31:0] retired_q;
logic [31:0] lat_hw_max_q;            // high-water: the tail, not the mean
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cyc_q <= '0; retired_q <= '0; lat_hw_max_q <= '0;
    acc_queue_q <= '0; acc_fetch_q <= '0; acc_media_q <= '0;
    acc_post_q  <= '0; acc_total_q <= '0;
    for (int i = 0; i < N_CMD; i++) cmd_live_q[i] <= 1'b0;
  end else begin
    cyc_q <= cyc_q + 32'd1;
 
    // T0 at DOORBELL ARRIVAL. This is the fix for BUG 1 — controller queueing
    // now falls inside the measured interval, where software experiences it.
    if (db_arrive_fire) begin
      t_db_q[db_cmd_id]    <= cyc_q;
      cmd_live_q[db_cmd_id] <= 1'b1;
      cmd_gen_q[db_cmd_id] <= cmd_gen_q[db_cmd_id] + 4'd1;
    end
 
    if (fetch_cpl_fire && cmd_live_q[fetch_cpl_id]) begin
      t_fetch_q[fetch_cpl_id] <= cyc_q;
      // Queueing = doorbell to fetch completion, minus the round trip itself.
      acc_queue_q <= acc_queue_q + 48'(fetch_issue_cyc - t_db_q[fetch_cpl_id]);
      acc_fetch_q <= acc_fetch_q + 48'(cyc_q - fetch_issue_cyc);
    end
 
    if (media_done_fire && cmd_live_q[media_cmd_id]) begin
      t_media_q[media_cmd_id] <= cyc_q;
      acc_media_q <= acc_media_q + 48'(cyc_q - t_fetch_q[media_cmd_id]);
    end
 
    // RETIREMENT at host consumption (26.3 §9's cpl_consumed), and gated on
    // generation so a stale event cannot close a command it does not own.
    if (cpl_consumed_fire && cmd_live_q[cons_cmd_id]
        && (cons_gen == cmd_gen_q[cons_cmd_id])) begin
      automatic logic [31:0] total = cyc_q - t_db_q[cons_cmd_id];
      acc_post_q  <= acc_post_q  + 48'(cyc_q - t_media_q[cons_cmd_id]);
      acc_total_q <= acc_total_q + 48'(total);
      retired_q   <= retired_q + 32'd1;
      cmd_live_q[cons_cmd_id] <= 1'b0;
      // The tail matters more than the mean for a storage path — an average
      // hides the outliers that determine a latency service level.
      if (total > lat_hw_max_q) lat_hw_max_q <= total;
    end
  end
end

Architecture. Four stage accumulators whose sum equals the total by construction, a retirement count, and a high-water maximum.

State. Three timestamps and a generation per command slot. The generation is what stops a stale consumption event from closing a command that has since been reissued (26.6 §9's pattern, here protecting a measurement rather than data).

Event. T0 on doorbell arrival; retirement on host consumption of the completion record — the two boundaries §7 got wrong.

Contract. cpl_consumed_fire must genuinely come from the host's CQ head doorbell (§4 step 12) rather than from the controller's own progress. If it is synthesised locally, this is §7 with more registers — the most likely wrong implementation of the fix.

Failure. The residual risk is a queueing term computed with the wrong subtraction sign when a fetch completes in the same cycle it is issued. The identity check in §9 catches it, which is why the accumulators are designed to sum.

DV/debug. lat_hw_max_q is the number a latency service level is written against, and an average cannot substitute for it: a path with a 66 µs mean and a 900 µs tail behaves like a 900 µs path for anything with a deadline.

9. Checks and Assertions

// MANDATORY. English: the four stage accumulators sum to the total. This is the
// property that makes the budget in §5 trustworthy — if it fails, some interval
// is being double-counted or dropped, and every conclusion drawn is void.
a_stages_sum_to_total: assert property (
  @(posedge clk) disable iff (!rst_n)
    (acc_queue_q + acc_fetch_q + acc_media_q + acc_post_q) == acc_total_q
);
 
// MANDATORY. English: a command is retired at most once, and only while live
// and in-generation. Catches a duplicate consumption event inflating the
// retired count and deflating the reported average.
a_retire_once_in_gen: assert property (
  @(posedge clk) disable iff (!rst_n)
    cpl_consumed_fire |-> (cmd_live_q[cons_cmd_id]
                            && (cons_gen == cmd_gen_q[cons_cmd_id]))
);
 
// MANDATORY. English: the reported maximum is never below the most recently
// measured total. Catches a high-water register that was reset by a diagnostic
// clear while accumulation continued — which reports a comfortable tail
// alongside a correct mean.
a_high_water_monotonic: assert property (
  @(posedge clk) disable iff (!rst_n || diag_clear)
    lat_hw_max_q >= $past(lat_hw_max_q)
);

Reading the three.

The first is the load-bearing one and it is unusual: it asserts an arithmetic identity about an instrument. Most assertions protect the design; this one protects the measurement, and without it §5's table is an assumption wearing a number.

|-> is correct in the second because liveness and generation are evaluated in the same cycle as the consumption event. It is a clean formal target — one slot, bounded state.

And the third exists because of a real operational hazard. A diagnostic clear that resets the maximum but not the accumulators produces a plausible mean and a falsely comfortable tail — the disable iff deliberately includes diag_clear so the property does not fire on the legitimate reset.

10. Measured Behaviour — Where Queue Depth Goes

Same command, varying queue depth and fetch batching, at 60 µs media.

Three readings.

The §7 instrument reports essentially the same latency at every queue depth. That is the signature to recognise: a latency measurement that does not move with load is not measuring the interval that grows with load. The actual figure rises from 66.3 to 78.4 µs across the sweep.

The gap widens monotonically with queue depth — 2.7 µs to 14.8 µs — because the omitted interval is the queueing. An instrument whose error scales with the thing being investigated is worse than no instrument, because it produces confident wrong conclusions.

And row 4 shows batching working on the right term. Fetch batching cuts PCIe per command from 3.8 to 2.5 µs. At 60 µs media that is a 2 % end-to-end improvement — real and small. At 5 µs media the same change is worth about 12 %, which is why §5's inversion matters before choosing what to optimise.

11. Executable Counterexamples

Case 2 is the minimum reproduction and it needs load, not a corner case. A single-command test cannot distinguish the two instruments — which is exactly how a wrong latency counter ships.

12. Verification

The reference model is arithmetic, which makes this unusually checkable.

Two readings.

The independent model must be bus-derived, not a mirror of the DUT's own accounting (27.3 §7 states the general independence rule for UCIe; the same principle applies here). A model that timestamps the same events the DUT timestamps will agree with §7's instrument.

And the negative case is the whole verification argument. "Does the reported latency rise when queueing is injected?" If it does not, the instrument is broken — and that is a one-test check that no amount of passing traffic provides.

13. Debugging — Which Stage Owns the Microseconds

Three readings.

Every external measurement exonerates the obvious suspects. The link is idle, the media is nominal, and the command is slow — which means the time is inside the controller, in an interval nobody instrumented.

A protocol analyser cannot resolve this. It sees the doorbell and it sees the fetch, so it can in principle measure the interval — but it cannot say why the gap exists, because the queue depth and scheduling state are internal (25.9 §3's argument). The analyser localises; the internal counter explains.

And the prevention is architectural rather than procedural. An instrument whose stages are required to sum to its total cannot silently omit a stage — the assertion fails instead. That is a stronger guarantee than a review checklist.

14. Misconceptions

"The host sends the command to the drive." §3, §4: software writes the command into host memory and rings a doorbell. The controller fetches it — a non-posted read costing a round trip.

"The command fetch is free." §4, §5: it is 1.5 of 3.8 µs of PCIe cost in the illustrative budget, and it is the largest single PCIe item.

"PCIe is the bottleneck in a storage path." §5: under 6 % at 60 µs media — and 34 % at 5 µs media. The answer depends on the storage class, so the budget must be recomputed.

"A faster link improves storage latency." §5: it improves the 3.8 µs, not the 60 µs. Compute the share before arguing.

"Average latency characterises the path." §8: a 66 µs mean with a 900 µs tail behaves like a 900 µs path under a deadline. The high-water mark is the number a service level is written against.

"Our latency counter is fine — it agrees at low load." §10: an instrument that does not move with queue depth is not measuring queueing. The gap widened from 2.7 to 14.8 µs across the sweep.

"The interrupt is the end of the command." §7 BUG 2: the command ends when software has consumed the completion record, and interrupt delivery is 3 % of the total at 60 µs media and 18 % at 5 µs.

15. Understanding Check

Q1. How many PCIe transactions does one read command cost, and which of them is a round trip?

Four plus the payload writes (§4). The doorbell (posted, host→device), the command fetch — a non-posted read, and the only round trip on the critical path — its Completion, N posted payload writes, the CQ entry write, and the MSI-X. The CQ head doorbell is a fifth but sits after completion and is off the critical path, which is why batching it is free while batching the fetch is the design's main lever. The structural point is that only the doorbell is host-initiated: software writes four bytes and the controller drives every remaining transaction, including the one that announces completion.

Q2. Is PCIe the bottleneck in this path?

It depends on the media, and the same trace gives opposite answers (§5). At 60 µs media the total is ≈66.3 µs of which PCIe is 3.8 µs — 5.7 %, so a faster link cannot help. Set media to 5 µs and the total is 11.3 µs, of which PCIe is 34 % and interrupt delivery another 18 % — the optimisation target has moved entirely with no architectural change. So the honest answer names the storage class, and the actionable version identifies the largest PCIe item: the command-fetch round trip at 1.5 µs, which batching reduces to ≈0.19 µs per command at a batch of 8.

Q3. A latency counter reports the same value at queue depth 1 and 32. What is wrong?

T0 is being captured at the wrong event (§7 BUG 1, §10). If the timestamp is taken when the fetch is issued rather than when the doorbell arrived, the doorbell-to-fetch interval — which is the controller queueing — falls outside the measured window. So the reported figure barely moves while the actual latency rises from 66.3 to 78.4 µs, and the gap widens monotonically from 2.7 to 14.8 µs. The tell is that the error scales with the thing under investigation, which makes the instrument worse than none. The second wrong event is retirement: closing the total on MSI emit rather than on host consumption excludes interrupt delivery — 3 % at 60 µs media, 18 % at 5 µs. And the structural fix is the sum identity (§9): four stage accumulators required to equal the total cannot silently omit a stage.

Q4. Design the verification that would have caught it.

A bus-derived independent model plus one negative test (§12). The model timestamps the doorbell transaction and the CQ head doorbell as observed on the interface — not by mirroring the DUT's counters, because a model that timestamps the same events the DUT does will agree with the broken instrument. The negative test is case 2: inject controller queueing at depth 32 and assert that the reported latency rises. If it does not, the instrument is broken, and no amount of passing traffic reveals that. Add the sum-identity assertion so a dropped or double-counted stage fails immediately, generation-gated retirement so a duplicate consumption event cannot deflate the average, and coverage bins on queue depth including max, fetch batch size, and both media-latency regimes from §5.

16. What Comes Next

This chapter traced one operation and accounted for its cost. The trace's largest PCIe item was a round trip; the next chapter's is a sustained transfer.

29.2 follows a host-to-device transfer across a x16 data path and does the same accounting for a workload where the payload — not the media — is the whole cost. The question changes accordingly: not "where do the microseconds go" but "why does a measured transfer fall short of the link's derived ceiling, and which stage owns the shortfall."