CXL.io ↔ PCIe Transactions

Every chapter of this module has been standing on something it did not explain. 7.2 said a configuration write is non-posted and moved on. 7.3 had software read an event record and assumed the read came back. 7.4 built a probe with a timeout and did not say what was timing out.

This chapter is that foundation. It is the deepest in the module because everything above it is a special case of what is here.

1. The Engineering Problem — Some Requests Need an Answer and Some Do Not

A request travels away from its issuer. Whether the issuer must wait for something to come back is not a performance choice; it is a property of the request type, fixed by the protocol, and it determines almost everything else about the hardware:

A request that needs no answer needs none of that. It is issued and forgotten, and the issuer proceeds immediately.

So the split is not a detail of encoding. It is the line between a transaction that costs a tag, a table entry, a timer and a stall, and one that costs a cycle. Getting a request on the wrong side of that line produces either a hang or silent data corruption, and this chapter builds both.

2. The One-Sentence Model

A transaction is a promise, and the protocol says which promises are collected. A posted request is a statement — issued and complete. A non-posted request is a question — it occupies a tag, a table entry and a deadline until its answer arrives, and every hard problem in the transaction layer is about what happens to that answer: it comes back out of order, it comes back for a tag nobody holds, or it never comes back at all.

Call it the outstanding set. The interesting state of a transaction layer is not what it is sending; it is what it is still waiting for.

3. What This Chapter Owns

Deliberately not repeated: TLP header encoding, completion status codes and the PCIe ordering table, which the PCIe track covers field by field. This chapter is about the hardware consequences of those rules — the tag, the table, the timer, the stall — and what breaks when each is wrong.

4. The Two Paths

The third band is the entire cost of being non-posted, and a posted request skips it completely. That asymmetry is why a write-heavy path can saturate a link while a read-heavy path is limited by something that has nothing to do with bandwidth — a point §11 measures.

5. Teaching-model boundary

6. RTL 1 — Which Requests Need an Answer

  always_comb begin
    case (req_kind)
      3'd0:    is_posted = 1'b1;                 // memory write
      3'd6:    is_posted = 1'b1;                 // message
      3'd2:    is_posted = CFGWR_POSTED;         // configuration write
      default: is_posted = 1'b0;                 // all reads, I/O, cfg read
    endcase
  end
 
  // The two are exact complements: every request either needs a completion
  // or does not, and there is no third state.
  assign needs_completion = !is_posted;

=== EXP1: which requests need a completion ===
  memory write         posted=1 needs completion=0
  message              posted=1 needs completion=0
  memory read          posted=0 needs completion=1
  config read          posted=0 needs completion=1
  config WRITE         posted=0 needs completion=1
  I/O write            posted=0 needs completion=1
  posted=2 non-posted=4
  a configuration write is NOT fire-and-forget       : ok
  the variant retires a config write with no completion: ok

Note that "write" is not the property that decides it. A memory write is posted and a configuration write is not, and both are writes. The property is the type of the request, and the reason is that a configuration write changes device state that software must know has taken effect before it proceeds — so the protocol makes it acknowledge.

The CFGWR_POSTED variant is not a hypothetical. It is what happens when someone reasons "it's a write, so it's posted", and the consequence is a device that reports a configuration write as complete when it has not landed — after which software reads back a register it believes it wrote and gets the old value, intermittently, under load.

The both_err checker asserts that is_posted and needs_completion are never equal. That looks trivial because in this module they are complements by construction, and it is worth keeping precisely because in a larger design they usually are not — they get computed on separate paths, and drifting apart is a real failure.

7. RTL 2 — A Tag Names One Outstanding Request

A completion carries a tag, and the tag is the only thing that says which request it answers. So a tag must be unique among requests in flight — not unique forever, just unique among the live ones.

      if (alloc && tag_valid) begin
        if (busy_q[tag]) reuse_err <= 1'b1;         // still outstanding
        busy_q[tag] <= 1'b1;
        n_alloc_q   <= n_alloc_q + 8'd1;
      end else if (alloc && !tag_valid) begin
        n_denied_q <= n_denied_q + 8'd1;            // back-pressure, not a drop
      end

=== EXP2: a tag names one outstanding request ===
  10 allocations from 8 tags : allocated=8 denied=2 exhausted=1
  exhaustion is back-pressure, not a dropped request: ok
  no tag was handed out twice while outstanding      : ok
  reuse-live variant : allocated=10 reuse flagged=1
  the reusing variant handed out a live tag          : ok

Exhaustion is back-pressure, not failure. Running out of tags means the issuer waits — it does not mean a request is dropped, and it is not an error condition. n_denied_q counts the stalls so the tag count can be sized against real traffic rather than guessed.

The REUSE_LIVE variant is the most dangerous bug in this chapter, and it is worth being precise about why. Suppose tag 3 is outstanding for a read of address X, the requester gives up or the completion is late, and tag 3 is handed out again for a read of address Y. The original completion then arrives, carrying tag 3, and is matched to the request for Y. The requester receives data from address X believing it came from address Y. No error is raised anywhere: the tag was valid, a request was outstanding, and the completion retired it.

That is silent data corruption produced by an allocator, and it is the reason a tag must not be recycled until the protocol guarantees no completion for its previous use can still arrive.

8. RTL 3 — Completions Return in Any Order

Tag 2 was issued last and completed first. Nothing is wrong: completions may return in any order, because they are produced by different targets with different latencies and travel independently.

  // The broken shape retires the OLDEST outstanding request whatever tag the
  // completion carried -- correct only if completions never reorder.
  assign eff_tag = ASSUME_INORDER ? oldest_q : cpl_tag;
  ...
        if (cpl_data != exp_m[eff_tag]) mismatch_err <= 1'b1;

=== EXP3: completions return out of order ===
  3 reads issued, tags 0/1/2, outstanding=3
  completion tag 2 : correct retires tag 2 data c2 | in-order model retires tag 0
  after 3 completions : outstanding=0 issued=3 retired=3
  every completion reached the requester that asked : ok
  in-order model : mismatch flagged=1
  assuming in-order delivered data to the wrong one : ok

The ASSUME_INORDER instance retires the oldest outstanding request regardless of the tag. Look at what it gets right: three completions arrive, three requests retire, outstanding returns to zero, and no counter is out of balance. Every accounting check passes. It just gives tag 2's data to the requester waiting on tag 0.

The only property that catches it is the one comparing the returned data against what that specific requester expected — which is why exp_m exists. A transaction layer that only counts is blind to the entire class of misdelivery bugs, and counting is what most transaction-layer testbenches do.

The outstanding counter uses the exhaustive case for the same reason as the two previous chapters:

=== EXP4: issue and retire in the same cycle ===
  one issued and one retired together : outstanding 1->1
  outstanding held across simultaneous traffic      : ok

9. RTL 4 — A Posted Request Is Done When It Is Issued

  // A posted request needs no completion, so the issuer never blocks on one.
  assign accept = req_valid && (!busy_q || (is_posted && !WAIT_FOR_CPL));

=== EXP5: a posted request is done when it is issued ===
  6 posted writes : issued=6 done=6 stalled=0 busy=0
  every posted write retired at issue, no stall     : ok
  wait-for-completion variant : issued=1 done=0 stalled=5
  waiting for a completion that never comes stalls  : ok

The variant issues one request and then stalls for the rest of the test, waiting for a completion the protocol never sends. This is not a subtle performance loss; it is a hang, and it is what "just wait for the acknowledgement, it's safer" produces when the acknowledgement does not exist.

The second half of the property is the one a mutation caught:

=== EXP5b: a posted write must not queue behind a read ===
  read outstanding, 4 posted writes offered : issued=5 done=4 stalled 0->0
  posted writes went straight past the pending read : ok
  all five requests were accepted                   : ok
  and the read completed independently afterwards    : ok

A read is outstanding — the issuer is waiting for its completion — and four posted writes are offered. All four are accepted immediately. They do not queue behind the read, because they have nothing to wait for.

This case escaped the first mutation run: with only posted traffic in the test, !busy_q was always true and the mutation accept = req_valid && !busy_q was indistinguishable from the correct expression. It took mixed traffic to separate them — and the scenario that separates them is the common one in real systems, where reads and writes are interleaved constantly.

10. RTL 5 — Whose Completion Is Late

Every non-posted request needs a deadline, because a target that never answers must not hold the issuer forever. The design question is whether there is one deadline or one per request.

    if (GLOBAL_TIMER) begin
      // One timer: it can say that SOMETHING is late, and picks arbitrarily.
      if ((|live_q) && (gtimer_q >= LIMIT[7:0])) begin
        to_hit = 1'b1;
        for (k = NTAG-1; k >= 0; k = k - 1) if (live_q[k[2:0]]) to_tag = k[2:0];
      end
    end else begin
      for (k = NTAG-1; k >= 0; k = k - 1)
        if (live_q[k[2:0]] && (age_m[k] >= LIMIT[7:0])) begin
          to_hit = 1'b1; to_tag = k[2:0];
        end
    end

Tag 7 is issued and never answered. Eight cycles later a healthy request is issued on tag 2.

=== EXP6: whose completion is late ===
  tag 7 never answered : correct reported tag=7 count=1
  the per-request timer named the request that hung : ok
  the reported tag was the one that was actually late: ok
  global-timer variant : reported tag=2 wrong-tag flagged=1
  one shared timer blamed a healthy request         : ok

The shared timer blamed tag 2, which had been outstanding for four cycles out of a limit of twelve. It was not late. It was simply the request the arbitrary picker landed on when the one global deadline expired.

The consequence in a real system is that the error log names a healthy path, an engineer investigates it and finds nothing, and the actual hung request is abandoned without ever being identified. A timeout that cannot name the request is barely better than no timeout, because the recovery action — retry, reset, quiesce that path — depends on knowing which path.

wrong_tag_err is what makes this checkable, and it is written about the reported tag's own age rather than restating the picker:

        // The reported tag must be one whose OWN age exceeded the limit.
        if (age_m[to_tag] < LIMIT[7:0]) wrong_tag_err <= 1'b1;

There is a second property that only appeared because a mutation survived:

  a request issued and completed, then 20 idle cycles: timeouts=0
  completing a request cleared it from the timer     : ok

A request that completed must never be declared late afterwards. Obvious, and the first testbench never checked it — because it never retired anything into the timeout module at all.

11. RTL 6 — How Many Tags a Read Path Needs

The outstanding limit caps how many non-posted requests can be in flight, and that cap sets throughput on a path where each request takes a fixed time to answer.

=== EXP8: how many tags a read path needs ===
  4 outstanding, latency 8 : issued=36 stalled=44 over 102 cycles
  unlimited outstanding    : issued=80 stalled=0
  the limit was respected every cycle               : ok
  too few tags makes a read path latency-bound      : ok
  throughput matched outstanding divided by latency : ok

Thirty-six issues in eighty cycles of demand, against eighty when the limit is removed — a factor of about two, and the ratio is not a coincidence. With a completion latency of eight cycles and at most four requests in flight, at most four requests can complete every eight cycles, so throughput is bounded at one every two cycles regardless of how much bandwidth the link has.

This is the bandwidth-delay product, and it is the answer to "how many tags do we need": enough to cover the round-trip latency at the target rate. Fewer, and the path is latency-bound — adding link bandwidth changes nothing, which is a genuinely confusing symptom the first time an engineer meets it.

And the asymmetry from §4 lands here. A posted write path has no outstanding limit because it has no outstanding set, so it is bandwidth-bound. On the same link, at the same clock, a write stream and a read stream can differ in throughput by a large factor, and no amount of link tuning closes the gap — the read path needs more tags, and if the tag space is fixed by the protocol, it needs lower latency instead.

12. RTL 7 — A Read Must Not Pass a Pending Write

The producer-consumer pattern: write the data, write the flag, and a consumer that sees the flag reads the data. It works only if the ordering rules hold.

  // A write lands one cycle after acceptance: that gap is the whole problem.
  assign rd_allowed = READ_PASSES_WRITE ? 1'b1 : (pending_wr_q == 4'd0);

=== EXP7: a read must not pass a pending write ===
  read while a write is pending : correct allowed=0 | bypass allowed=1 data=00
  the read waited for the write to land             : ok
  the bypassing variant returned the OLD value      : ok
  after the write landed : allowed=1 data=ab
  once it landed the consumer saw the new value     : ok

The bypassing variant returned 00 where the value written was ab. It answered faster and it answered wrongly.

The subtlety worth holding onto is that the write was already accepted. It was not rejected, not lost, not in error — it had been taken and had not yet landed, and that gap is where the entire producer-consumer failure lives. A design that treats "accepted" as "visible" has this bug by construction.

This is also why a posted write being fire-and-forget does not mean it is unordered. The issuer does not wait for it, but the fabric must not let a later read of the same path overtake it. No completion and no ordering are different properties, and posted requests have the first without the second.

13. RTL 8 — Nothing In Limbo

  // A completion arriving in the same cycle its timer expires is a real race
  // in silicon, not a testbench artefact. Exactly ONE outcome may be counted:
  // counting both puts the accounting permanently out of balance by one per
  // occurrence. The completion wins, because its data is valid.
  assign race        = retire && timed_out;
  assign eff_retire  = retire;
  assign eff_timeout = timed_out && !retire;

=== EXP9: nothing is in limbo ===
  issued=50 posted=17 non-posted=33 retired=15 timed out=1 outstanding=17
  conservation non-posted == retired + timed out + outstanding : 33 == 33
  conservation held                                 : ok
  every count matched an independent oracle         : ok
  posted writes never entered the outstanding count  : 17 of 50 issued
  completion and timeout collided 1 time(s), counted once each
  the completion won the race and the books balanced: ok

The conservation law is the one that matters: every non-posted request is outstanding, retired, or timed out — there is no fourth state, and a request that falls out of all three has been forgotten while something upstream still waits for it.

The race handling arrived by accident and is the more interesting half. The first version of the testbench asserted retire and timed_out in the same cycle, which broke conservation by exactly one. The obvious response was to call the stimulus illegal and fix the test.

It is not illegal. A completion arriving in the same cycle its deadline expires is a real race, and in silicon it happens whenever a completion is marginally late. The design has to resolve it, and the resolution is that the completion wins because its data is valid and abandoning a request whose answer is in hand loses information for no reason. What must never happen is counting both, which puts the accounting permanently out of balance by one per occurrence — a slow drift that looks like a leak.

So the stimulus stayed, the design gained a race resolution and an n_race_q counter, and the test now asserts the collision occurred exactly once and the books balanced.

14. Assertions

Icarus Verilog 13.0 does not support concurrent SystemVerilog assertions, so every property below is implemented as synthesisable checker logic and verified in simulation. The assert property form states the intent.

The one to notice is "data reaches its requester". Every other property here is satisfied by the in-order matcher that misdelivers data. Counting is not enough; something has to check identity.

15. Mutation Testing

Twenty-two mutations. Twenty-two killed.

The first run scored nineteen of twenty-two, and all three survivors were missing stimulus rather than equivalent mutants. Each one named a scenario worth having.

A posted write blocks on the previous request. With only posted traffic in the test, busy_q was never set, so the mutation was indistinguishable from the correct expression. Separating them required a read outstanding and posted writes offered — which is the ordinary case in a real system, and is now EXP5b. A test that exercises one request type at a time cannot find bugs in the interaction between them.

A retired request stays outstanding. The timeout module was never sent a retire at all: the test issued requests and let them hang, so the retire path was dead code as far as the testbench was concerned. The new check issues a request, completes it, waits twenty idle cycles and asserts no timeout occurred.

That fix needed a second attempt. The first version checked the reported tag, but the tag-7 timeout earlier in the experiment had already latched it, so the check could never fail — a test that passed for a reason unrelated to the property. Running it on a freshly reset instance and asserting the timeout count fixed it. A check that cannot fail is not a check, and the way to find out is to make the mutation and watch.

A completion retires a request that is not outstanding. Nothing in the test ever sent a completion for a tag nobody held. Adding it required a dedicated instance, because a stray completion trips the good instance's own checker and would be scored a failure:

  abuse instance flagged an unexpected completion   : ok
  and it retired nothing: a tag nobody holds is void: ok

The second line is the property that kills the mutation. It is not enough that the stray completion is flagged; it must also retire nothing. A completion carrying a tag with no outstanding request is void, and acting on it corrupts the outstanding set.

That is the third chapter in a row where a checker needed an instance reserved for illegal stimulus. The pattern is now firmly established: any checker whose condition a correct design cannot reach needs its own DUT, or it is decoration.

16. Verification Plan

The coverage model is a cross of request type against completion behaviour: posted and each non-posted kind, crossed with completes-in-order, completes-out-of-order, never-completes, and completes-as-the-timer-expires. The last column is the one that produced the race, and the posted row crossed with a live outstanding request is the one that produced EXP5b.

17. Debug Lab

// Classification.
is_posted = (req_kind == MEM_WRITE) || (req_kind == CFG_WRITE) ||
            (req_kind == MSG);          // "they're all writes"

  config WRITE         posted=0 needs completion=1
  the variant retires a config write with no completion: ok

case (req_kind)
  MEM_WRITE, MSG: is_posted = 1'b1;
  default:        is_posted = 1'b0;    // reads, I/O, and CONFIG WRITES
endcase

// Allocate the next tag.
tag       <= next_tag;
next_tag  <= next_tag + 1;        // wraps, without checking what is still live

  reuse-live variant : allocated=10 reuse flagged=1
  the reusing variant handed out a live tag          : ok

// Allocate only a tag that is not live, and prove it.
if (alloc && tag_valid) begin
  if (busy_q[tag]) reuse_err <= 1'b1;
  busy_q[tag] <= 1'b1;
end
if (free) busy_q[free_tag] <= 1'b0;

// Retire the oldest outstanding request.
if (cpl_valid) begin
  retire_tag <= oldest_q;
  oldest_q   <= oldest_q + 1;
end

  completion tag 2 : correct retires tag 2 data c2 | in-order model retires tag 0
  in-order model : mismatch flagged=1

assign eff_tag = cpl_tag;                       // the tag decides, not the order
assign retire  = cpl_valid && live_q[eff_tag];
if (retire && cpl_data != exp_m[eff_tag]) mismatch_err <= 1'b1;

// Wait for the acknowledgement before accepting another request.
if (req_valid && !busy_q) busy_q <= 1'b1;
if (busy_q && cpl_valid)  busy_q <= 1'b0;

  wait-for-completion variant : issued=1 done=0 stalled=5

assign accept = req_valid && (!busy_q || is_posted);
if (req_valid && accept && is_posted) n_done_q <= n_done_q + 8'd1;   // done now

assign accept = req_valid && !busy_q;    // everything waits for the outstanding read

  read outstanding, 4 posted writes offered : issued=5 done=4 stalled 0->0
  posted writes went straight past the pending read : ok

assign accept = req_valid && (!busy_q || is_posted);

// One timer for all outstanding requests.
if (|live_q) gtimer_q <= gtimer_q + 1;
if (gtimer_q >= LIMIT) begin timeout_valid <= 1'b1; timeout_tag <= any_live; end

  tag 7 never answered : correct reported tag=7 count=1
  global-timer variant : reported tag=2 wrong-tag flagged=1

for (k = 0; k < NTAG; k = k + 1)
  if (live_q[k]) age_m[k] <= age_m[k] + 8'd1;         // one age per request
...
if (live_q[k] && (age_m[k] >= LIMIT)) begin to_hit = 1'b1; to_tag = k; end
if (to_hit && age_m[to_tag] < LIMIT) wrong_tag_err <= 1'b1;

assign rd_allowed = 1'b1;     // reads are served immediately, always

  read while a write is pending : correct allowed=0 | bypass allowed=1 data=00
  after the write landed : allowed=1 data=ab

assign rd_allowed = (pending_wr_q == 4'd0);   // wait for earlier writes to land

if (retire)    n_retire_q <= n_retire_q + 1;
if (timed_out) n_to_q     <= n_to_q     + 1;      // both can be true at once

  completion and timeout collided 1 time(s), counted once each
  conservation non-posted == retired + timed out + outstanding : 33 == 33

assign race        = retire && timed_out;
assign eff_retire  = retire;
assign eff_timeout = timed_out && !retire;   // the completion wins
if (race) n_race_q <= n_race_q + 16'd1;      // and the collision is visible

18. Design Review

1. Which request types are posted? Only memory writes and messages. If configuration writes are in that set, the design is wrong.
2. Can a tag be reused while a completion for its previous use may still arrive? If yes, there is a silent data-corruption path.
3. Does the completion matcher use the tag or the arrival order? Order is correct only if reordering is impossible, which it is not.
4. Is there anything checking that data reached the requester that asked for it? Counters cannot see misdelivery.
5. Does a posted request ever block on an outstanding non-posted one? It must not.
6. Is there one timeout or one per request? One cannot name the late request, and recovery needs identity.
7. What happens when a completion and its timeout land in the same cycle? Exactly one outcome, resolved in the design.
8. How many tags, and how does that compare to the bandwidth-delay product? Too few makes the path latency-bound and link tuning will not help.
9. Can a read overtake an accepted but unlanded write? That breaks every producer-consumer flow above it.

19. How This Appears in Real Engineering

Bring-up. Classification bugs surface first, as a device that appears to accept configuration and does not change behaviour. The read-back-mismatch symptom in lab 1 is one of the most common first-week findings.

Performance work. The tag count against the bandwidth-delay product is the calculation behind "why is our read bandwidth half the link rate". Engineers reach for link tuning; the answer is usually outstanding capacity or latency.

Silicon debug. Tag reuse is the bug people remember for a career, because it produces wrong data with no error indication anywhere and reproduces only under load.

Driver and firmware work. The ordering rule is the contract every producer-consumer ring depends on. When it breaks, the symptom appears in software as a corrupt message, and the investigation starts in the wrong place.

Fleet operation. n_race_q, denied-allocation counts and per-tag timeout identity are what turn intermittent field reports into a number someone can act on.

20. Common Misconceptions

21. Interview Reasoning

22. Exercises

1. Analysis. A device reports issued_np=10000, retired=9998, timed_out=0, outstanding=2, and has been idle for an hour. Nothing violates conservation. Name the two conditions consistent with this and the single measurement that distinguishes them.

2. Design. Specify the tag-free rule for a requester that supports abandoning a request. State exactly when the tag may be reused, what the timeout duration must be at least as long as, and the invariant that makes reuse-after-abandonment safe.

3. RTL task. Extend the completion matcher to support completions split into multiple parts, where one request is answered by several completions. State which properties in §14 change, which new invariant is required, and what a partial completion followed by a timeout must do to the outstanding set.

4. DV task. Write the coverage cross for this layer, then explain why a testbench that only counts issues and retirements can reach full statement coverage while missing the entire misdelivery class.

5. Debug task. A mixed workload shows write throughput collapsing whenever read latency rises, and reads performing at exactly half the expected rate. Both are real. Give your investigation order and name the two independent faults, with the measurement that separates them.

6. Design review. A colleague proposes freeing a tag as soon as the requester abandons a request, arguing that the tag space is a scarce resource and the abandoned request will never be used. Give the strongest version of that argument, then name the specific failure it creates and the smallest change that keeps most of the benefit safely.

23. Summary

A transaction is a promise, and the protocol decides which promises are collected.

• Posted: memory writes and messages. Non-posted: all reads, I/O, and configuration writes. A configuration write is not fire-and-forget, which is the fact every other chapter in this module was standing on.
• Being non-posted costs a tag, a table entry, a deadline and a possible stall. A posted request costs a cycle.
• A tag must be unique among requests in flight, and reusing a live one delivers another request's data with no error logged anywhere — the worst failure in the layer.
• Completions return in any order, so the tag decides who gets the data. An in-order matcher balances every counter and still misdelivers.
• Posted does not mean unordered. No completion and no ordering are different properties, and a read overtaking a pending write breaks every producer-consumer flow above it.
• One timeout per request, not one for all of them — recovery needs to know which path hung, and a shared timer blames a healthy one.
• Tags needed equals the bandwidth-delay product. Too few makes a read path latency-bound, and adding link bandwidth changes nothing.
• A completion and its timeout can land in the same cycle. Resolve it in the design and count one outcome, or the accounting drifts by one per occurrence.
• Verification lesson: counting is not enough. Give every outstanding request distinguishable data and check identity — accounting and timeouts together still pass on a design that corrupts data.

That completes Module 7 — CXL.io. The control-plane spine is now covered end to end: what it carries, how registers behave, how events are reported, how a hierarchy is discovered, and the transaction semantics underneath all of it. What CXL.io deliberately does not provide is coherence — and that is where the rest of CXL begins.