Ordering — What May Pass What, and Why It Is a Correctness Rule

Every chapter so far has treated one transaction at a time. Real fabrics carry hundreds simultaneously, on different paths, through different queues, with different lifetimes — and Chapter 12.5 argued that keeping many in flight is the only way to use a fast link at all.

Concurrency is the point. It is also the thing that can silently break a device driver.

When multiple PCIe transactions are in flight at the same time, which observations are required to preserve order, which may pass each other, and why do attributes such as Relaxed Ordering change correctness rather than merely performance?

1. The Problem Ordering Solves

Start with a device, not a matrix.

A producer wants to hand work to a consumer. It does two things:

1.  write the descriptor — the data describing the work
2.  write the doorbell   — the flag saying the work is ready

The consumer does two things:

1.  observe the doorbell
2.  read the descriptor

The whole pattern rests on one assumption: if the consumer can see the doorbell, the descriptor is already there.

2. Issue Order Is Not Observation Order

A Requester issues transactions in some order. That order is not automatically preserved everywhere.

Two transactions may take different paths, sit in different queues, be subject to different backpressure, and have completely different lifetimes — a posted write finishes when it is delivered; a read is not finished until its Completion returns (Chapter 12.1).

So the question is not "does PCIe preserve order" — it is "which orderings does PCIe require to be preserved, and which does it permit or require to be broken." §4 is the answer.

3. Three Transaction Classes

Ordering rules are stated over classes, not over individual operations (Chapter 11.7 §3).

The classes have genuinely different resource behaviour, and that is why the rules distinguish them rather than treating all TLPs alike. A Non-Posted Request occupies a correlation resource until it is answered; a Posted Request occupies nothing after it is delivered. §5 shows why that asymmetry forces some of the rules to be mandatory passing rather than mandatory ordering.

4. The Verified Rules

5. Mandatory Passing Is Not a Loophole

Rows four and five look like the specification being permissive. They are the specification preventing a deadlock.

6. Relaxed Ordering Changes Correctness, Not Just Speed

Chapter 11.6 §4 introduced Relaxed Ordering as a permission. Here is the permission, precisely: RO allows a later Memory Write, Memory Read or Completion to pass a previously issued Memory Write.

Look at §1's pattern with that in mind. The descriptor write is a Memory Write. The doorbell write is a Memory Write. The rule that makes the pattern work is "Posted must not pass Posted" — and RO is exactly the attribute that relaxes it.

ID-Based Ordering relaxes the same constraint on a different basis. It permits passing a previously issued Memory Write when the Requester identities differ — bus, device, function, and PASID. The intent is to decouple unrelated Requesters that happen to share a path, where the default rules would order one behind another for no reason connected to correctness.

But do not compress that into "same ID is ordered, different ID is free." The relaxation is specific, and the full applicability conditions are beyond what this chapter can source (§4). A design that implements the slogan has written its own ordering model.

7. No Snoop Is Not an Ordering Control

Both live in the same header field (Chapter 11.6 §2), and they solve unrelated problems.

They can be set independently and mean different things. Setting No Snoop does not relax any ordering rule; setting Relaxed Ordering does not change any coherency treatment. A design that treats "the attributes" as one knob will eventually set one when it meant the other, and the two failure modes look similar enough from a distance — wrong data — that the misdiagnosis costs real time.

8. Ordering Does Not Identify Transactions

A recurring confusion, and this chapter is where it must be killed.

Ordering rules constrain the relative movement of transactions. They never say which Request a packet belongs to.

Three different mechanisms, three different questions. The same-Transaction-ID rule is the clearest illustration: it says the Completions of one Request stay in order relative to each other — and Chapter 13.3 §7 showed that this buys a cursor-based reassembly model and nothing about identifying which read a packet belongs to, because the stream a Requester sees is a merge of many reads.

Arrival position is never a substitute for correlation identity, under any ordering rule.

9. Ordered, and Permitted to Pass

The word to stare at is may. The right-hand case is not "the doorbell arrives first." It is "the doorbell is permitted to arrive first" — and on a quiet fabric it usually will not. That gap between permitted and observed is the entire reason RO bugs reach production (§6).

10. Worked Example — Descriptor Then Doorbell

Setup. A producer writes a 64-byte descriptor to 0x8000_2000, then writes a doorbell to 0x8000_3000. Both are posted Memory Writes, same Traffic Class.

And a detail worth extracting: the failure is worse than reading zeros. The descriptor address held a previous, valid descriptor. The consumer processes real work that was already done — a duplicate operation, or work against a buffer that has since been freed. Reading uninitialised memory would at least look wrong.

11. Worked Example — Read After Write

Setup. A producer performs a posted Memory Write, then issues a Memory Read.

The question people want answered: does the read "flush" the write — that is, does a returning Completion prove the write is visible?

12. RTL — The Pass-Permission Function

// SYNTHESIZABLE. Decide whether a younger transaction may pass an older one.
// The rules encoded: NORMATIVE VERIFIED SUBSET (section 4). The enum, the
// ordering-domain abstraction, and the conservative default: ILLUSTRATIVE.
package txn_order_pkg;
 
  typedef enum logic [1:0] {
    TXN_POSTED     = 2'd0,
    TXN_NONPOSTED  = 2'd1,
    TXN_COMPLETION = 2'd2
  } txn_class_e;
 
  // NORMALIZED per-transaction ordering metadata. Not a wire format.
  typedef struct packed {
    txn_class_e  cls;
    logic        relaxed_ordering;   // Attr[1] (Chapter 11.6)
    logic        id_ordering;        // Attr[2]
    logic [2:0]  traffic_class;      // ordering scope (section 4)
    logic [15:0] requester_id;       // ordering identity for IDO
    logic [15:0] txn_id;             // correlation identity for Completions
  } txn_meta_t;
 
endpackage

import txn_order_pkg::*;
 
// COMBINATIONAL. can_pass(older, younger) — may `younger` be issued before
// `older`? Encodes ONLY the verified subset of section 4.
function automatic logic can_pass (input txn_meta_t older,
                                   input txn_meta_t younger);
  logic permitted;
 
  // SCOPE FIRST. The rules of section 4 apply within a single Traffic Class.
  // Transactions in different Traffic Classes are not ordered by these rules,
  // so this teaching model imposes no constraint between them.
  if (older.traffic_class != younger.traffic_class)
    return 1'b1;
 
  // Default: DENY. Anything this subset cannot justify permitting stays
  // ordered, which is the safe direction for a teaching model — it can cost
  // throughput, and it cannot break the producer/consumer pattern.
  permitted = 1'b0;
 
  unique case (older.cls)
 
    // ---- Nothing may pass a previously issued Posted Request ----------
    // ...unless a relaxation applies. This is section 4's first group and
    // section 6's whole subject.
    TXN_POSTED: begin
      if (younger.relaxed_ordering)
        // RO permits a later MWr, MRd or Completion to pass a previous MWr.
        permitted = 1'b1;
      else if (younger.id_ordering
               && (younger.requester_id != older.requester_id))
        // IDO permits it when the Requester identities differ.
        permitted = 1'b1;
      else
        permitted = 1'b0;
    end
 
    // ---- MANDATORY PASSING (section 5) --------------------------------
    // These are obligations, not permissions: a fabric that refused would
    // deadlock. Note there is no attribute term — the rule does not depend
    // on one.
    TXN_NONPOSTED: begin
      permitted = (younger.cls == TXN_POSTED)
               || (younger.cls == TXN_COMPLETION);
      // younger.cls == TXN_NONPOSTED falls through as 1'b0: section 4 does
      // not publish that row, so this model does not permit it.
    end
 
    // ---- Completions ---------------------------------------------------
    TXN_COMPLETION: begin
      if (younger.cls == TXN_COMPLETION)
        // Same Transaction ID must not pass; different adds no requirement.
        permitted = (younger.txn_id != older.txn_id);
      else
        // Completion versus a younger Request: section 4 does not publish
        // this row, so the model stays ordered.
        permitted = 1'b0;
    end
 
  endcase
 
  return permitted;
endfunction

Classification: synthesizable (combinational function; package: compile-time).

Architecture. A pure function of two transactions' normalized metadata. It consumes the descriptor built at acceptance (Chapter 11.6 §8), never live configuration — so a configuration change cannot alter the ordering decision for a transaction already queued.

Why the default is deny. An unsourced row could be permissive or restrictive in the real specification. Guessing permissive risks the §1 failure; guessing restrictive costs throughput and, for the two mandatory-passing rows, is handled explicitly so no deadlock is introduced. The asymmetry of consequences decides the default.

Failure — four, and the first is §16's formal counterexample. can_pass = younger_ready ignores the rules entirely and lets a doorbell pass a descriptor. Omitting the Traffic Class scope check applies the rules across classes they do not govern, over-ordering unnecessarily. Treating RO as requiring a pass rather than permitting one turns an optional relaxation into forced reordering. And omitting the mandatory-passing arm reintroduces §5's deadlock.

Deliberately simplified: the verified subset only; one ordering domain per Traffic Class; no Virtual Channel structure; no element-specific behaviour; PASID not modelled.

13. RTL — Two-Entry Order-Aware Scheduler

// SYNTHESIZABLE. The smallest structure in which ordering is a real decision:
// two queued transactions, oldest normally issued first, younger permitted to
// bypass only when can_pass() allows it.
// The ordering rules: NORMATIVE SUBSET. The two-entry depth, the bypass
// policy and the interface: ILLUSTRATIVE.
import txn_order_pkg::*;
 
module order_aware_scheduler #(
  parameter int PAYLOAD_W = 64
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- Enqueue ---------------------------------------------------------
  input  logic                 in_valid,
  output logic                 in_ready,
  input  txn_meta_t            in_meta,
  input  logic [PAYLOAD_W-1:0] in_payload,
 
  // ---- Issue -----------------------------------------------------------
  output logic                 out_valid,
  input  logic                 out_ready,
  output txn_meta_t            out_meta,
  output logic [PAYLOAD_W-1:0] out_payload,
  output logic                 out_was_bypass,   // telemetry, not protocol
 
  // Per-destination blocking: the oldest cannot make progress right now.
  input  logic                 oldest_blocked
);
 
  // Two slots. Slot 0 is always the older when both are occupied.
  txn_meta_t            meta_q  [2];
  logic [PAYLOAD_W-1:0] pay_q   [2];
  logic [1:0]           occ_q;              // occ_q[0] = older, occ_q[1] = younger
 
  wire have_old   = occ_q[0];
  wire have_young = occ_q[1];
 
  // Accept only into a free slot, and only into slot 1 if slot 0 is taken —
  // so "slot 0 is older" is an invariant rather than a convention.
  assign in_ready = !have_old || !have_young;
 
  wire push = in_valid && in_ready;
  wire push_to_0 = push && !have_old;
  wire push_to_1 = push &&  have_old && !have_young;
 
  // THE ORDERING DECISION. The younger may issue first only when all three
  // hold: the older cannot make progress, the younger exists, and the rules
  // permit the pass. Readiness alone is NEVER sufficient — that is section
  // 16's counterexample.
  wire bypass_ok = have_old && have_young && oldest_blocked
                && can_pass(meta_q[0], meta_q[1]);
 
  wire issue_old = have_old && !oldest_blocked;
 
  assign out_valid      = issue_old || bypass_ok;
  assign out_meta       = issue_old ? meta_q[0] : meta_q[1];
  assign out_payload    = issue_old ? pay_q[0]  : pay_q[1];
  assign out_was_bypass = !issue_old && bypass_ok;
 
  wire pop = out_valid && out_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      occ_q <= 2'b00;
      for (int i = 0; i < 2; i++) begin
        meta_q[i] <= '0; pay_q[i] <= '0;
      end
    end else begin
      // Pop first, in program order below, so a same-cycle pop-and-push is
      // resolved deterministically rather than by if-statement accident.
      if (pop) begin
        if (issue_old) begin
          // Older leaves: the younger, if any, shifts down and becomes old.
          meta_q[0] <= meta_q[1];
          pay_q[0]  <= pay_q[1];
          occ_q     <= {1'b0, have_young};
        end else begin
          // Younger bypassed and left: the older stays put.
          occ_q[1] <= 1'b0;
        end
      end
 
      // Push resolves against the post-pop occupancy. Written as explicit
      // arms so a simultaneous pop and push cannot land in the wrong slot.
      if (push_to_0 && !(pop && issue_old && have_young)) begin
        meta_q[0] <= in_meta; pay_q[0] <= in_payload; occ_q[0] <= 1'b1;
      end else if (push_to_1 || (push && pop && issue_old && have_young)) begin
        meta_q[1] <= in_meta; pay_q[1] <= in_payload; occ_q[1] <= 1'b1;
      end
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. Two slots with a hard invariant — slot 0 is the older whenever both are occupied — so "older" and "younger" are structural rather than a convention a reader has to trust.

State. Two metadata/payload slots and a two-bit occupancy.

Cycle behaviour.

Row four is the chapter. The younger is ready, the older cannot move, and the design still issues nothing — because the rules forbid the pass. A scheduler that issues there is faster and wrong.

Contract. The caller relies on the ordering rules being honoured for every issued pair, and on metadata being stable while a transaction is queued. out_was_bypass is telemetry — it lets a testbench see that a legal bypass happened, and it is not a protocol signal.

Failure — four. bypass_ok = have_young && oldest_blocked without can_pass is §16's counterexample. Dropping oldest_blocked lets the younger jump the queue whenever it is convenient, which turns a permission into a policy. Allowing a push into slot 0 while slot 0 is occupied breaks the age invariant, after which can_pass(older, younger) is called with the arguments swapped. And re-reading in_meta at issue time rather than the stored copy reintroduces Chapter 11.6 §7's metadata-ownership bug.

Deliberately simplified: two entries; one destination; a single blocking input rather than per-destination status; no Virtual Channel structure; no fairness policy beyond oldest-first.

Production implication: a real element has many entries, per-destination and per-class blocking, VC arbitration, and fairness requirements. The can_pass predicate is the part that does not change.

14. RTL — Producer/Consumer Fence Tracker

// SYNTHESIZABLE. ILLUSTRATIVE IMPLEMENTATION POLICY — PCIe does not require
// a device to implement doorbell tracking hardware. This models the local
// contract "do not release the announcing write until the announced writes
// have been released", which is how an ordering requirement becomes a
// scheduler constraint in real RTL.
module producer_fence #(
  parameter int MAX_PENDING = 16
) (
  input  logic clk,
  input  logic rst_n,
 
  // Data writes the device considers "announced by" a later doorbell.
  input  logic  data_wr_issued,
  input  logic  data_wr_released,      // left this element's ordered domain
 
  // The announcing write.
  input  logic  doorbell_valid,
  output logic  doorbell_ready,        // held while protected writes pend
 
  input  logic  doorbell_taken,
 
  output logic [$clog2(MAX_PENDING+1)-1:0] pending,
  output logic  fence_error            // release without a matching issue
);
 
  localparam int CNT_W = $clog2(MAX_PENDING + 1);
 
  logic [CNT_W-1:0] pend_q;
  logic             err_q;
 
  wire underflow = data_wr_released && !data_wr_issued && (pend_q == '0);
  wire overflow  = data_wr_issued  && !data_wr_released
                                   && (pend_q == CNT_W'(MAX_PENDING));
 
  // THE CONTRACT, in one line: the doorbell is offered downstream only when
  // nothing it announces is still pending.
  assign doorbell_ready = (pend_q == '0);
  assign pending        = pend_q;
  assign fence_error    = err_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pend_q <= '0; err_q <= 1'b0;
    end else begin
      // Four explicit arms. Simultaneous issue and release leaves the count
      // unchanged — a live gauge, never derived from cumulative totals
      // (Chapter 12.5 section 13).
      unique case ({data_wr_issued, data_wr_released})
        2'b10:   pend_q <= overflow  ? pend_q : (pend_q + CNT_W'(1));
        2'b01:   pend_q <= underflow ? pend_q : (pend_q - CNT_W'(1));
        default: pend_q <= pend_q;
      endcase
      if (underflow || overflow) err_q <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable, illustrative implementation policy.

Architecture. A live pending gauge that gates the announcing write's readiness. It does not reorder anything — it withholds an offer, which is the cheapest possible way to enforce a local ordering contract.

Why this exists in the chapter. §4's rules are stated as constraints on a fabric. This is what such a constraint looks like when a device takes responsibility for it, and the shape is worth recognising: an ordering requirement becomes a ready signal that is low while something else is outstanding.

Contract. The caller defines what "released" means for its element — it is a local ownership boundary, not a PCIe event.

Failure — three. Deriving pending by subtracting cumulative issue and release counters breaks when either saturates (Chapter 12.5 §20). Gating on data_wr_issued rather than on the pending count releases the doorbell as soon as the last data write is offered rather than released. And an if/else if instead of the four-arm case drops the release on a simultaneous event, so the gauge climbs and the doorbell never goes out.

15. Assertions

// SVA over can_pass, order_aware_scheduler and producer_fence. These assert
// the VERIFIED SUBSET of section 4 and the LOCAL scheduler contract. They do
// NOT assert the full PCIe ordering matrix, and they do not assert anything
// about rows section 4 declined to publish.
 
// ---- SAFETY ----------------------------------------------------------
 
// P1: THE CENTRAL PROPERTY. A younger transaction is never issued before an
// older one unless the rules permit it.
property p_no_illegal_bypass;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_was_bypass) |-> can_pass(meta_q[0], meta_q[1]);
endproperty
a_no_illegal_bypass : assert property (p_no_illegal_bypass);
 
// P2: baseline ordering, stated positively over the producer/consumer case.
// Two Posted transactions in the same Traffic Class with default attributes
// must never be reordered.
property p_posted_not_passed_by_posted;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_was_bypass
             && (meta_q[0].cls == TXN_POSTED)
             && (meta_q[1].cls == TXN_POSTED)
             && (meta_q[0].traffic_class == meta_q[1].traffic_class)
             && !meta_q[1].relaxed_ordering
             && !meta_q[1].id_ordering) |-> 1'b0;
endproperty
a_posted_ordered : assert property (p_posted_not_passed_by_posted);
 
// P3: MANDATORY PASSING is available. A Posted transaction behind a blocked
// Non-Posted one must be issuable — the deadlock-avoidance rule of section 5,
// asserted as an obligation rather than a permission.
property p_posted_can_pass_nonposted;
  @(posedge clk) disable iff (!rst_n)
  (have_old && have_young && oldest_blocked
            && (meta_q[0].cls == TXN_NONPOSTED)
            && (meta_q[1].cls == TXN_POSTED)
            && (meta_q[0].traffic_class == meta_q[1].traffic_class))
    |-> out_valid;
endproperty
a_mandatory_pass : assert property (p_posted_can_pass_nonposted);
 
// P4: same-Transaction-ID Completions never pass each other.
property p_same_txn_id_completions_ordered;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_was_bypass
             && (meta_q[0].cls == TXN_COMPLETION)
             && (meta_q[1].cls == TXN_COMPLETION)
             && (meta_q[0].txn_id == meta_q[1].txn_id)) |-> 1'b0;
endproperty
a_same_id_ordered : assert property (p_same_txn_id_completions_ordered);
 
// P5: RO IS A PERMISSION, NOT A REQUIREMENT. Setting it must not force a
// bypass — the younger may still wait. Catches a design that treats the
// attribute as an instruction (section 6).
property p_ro_does_not_force_passing;
  @(posedge clk) disable iff (!rst_n)
  (have_old && have_young && meta_q[1].relaxed_ordering && !oldest_blocked)
    |-> (out_valid && !out_was_bypass);
endproperty
a_ro_permits_only : assert property (p_ro_does_not_force_passing);
 
// P6: OWNERSHIP. A queued transaction's ordering metadata is stable while it
// waits. Configuration may change underneath it; the transaction does not.
property p_meta_stable_while_queued;
  @(posedge clk) disable iff (!rst_n)
  (occ_q[0] && !(out_valid && out_ready && issue_old)) |=> $stable(meta_q[0]);
endproperty
a_meta_stable : assert property (p_meta_stable_while_queued);
 
// P7: CONSERVATION. Every accepted transaction is issued exactly once.
// (in_count / out_count are testbench counters.)
property p_no_drop_no_duplicate;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_ready) |-> (out_count + 1 <= in_count);
endproperty
a_conserved : assert property (p_no_drop_no_duplicate);
 
// P8: the age invariant. Slot 0 is the older whenever both are occupied — so
// can_pass is never called with its arguments swapped.
property p_slot0_is_older;
  @(posedge clk) disable iff (!rst_n)
  push_to_1 |-> have_old;
endproperty
a_age_invariant : assert property (p_slot0_is_older);
 
// P9: correctly stalled. When the older is blocked and the pass is not
// permitted, NOTHING issues. The property that fails on an
// "issue whatever is ready" scheduler.
property p_stall_when_pass_forbidden;
  @(posedge clk) disable iff (!rst_n)
  (have_old && have_young && oldest_blocked && !can_pass(meta_q[0], meta_q[1]))
    |-> !out_valid;
endproperty
a_correct_stall : assert property (p_stall_when_pass_forbidden);
 
// P10: the fence contract. The announcing write is not offered while
// anything it announces is pending.
property p_doorbell_held_while_pending;
  @(posedge clk) disable iff (!rst_n)
  (pending != '0) |-> !doorbell_ready;
endproperty
a_fence : assert property (p_doorbell_held_while_pending);
 
// P11: the fence gauge is a live gauge — bounded, no underflow, and
// simultaneous issue/release conserves it.
property p_fence_gauge_sane;
  @(posedge clk) disable iff (!rst_n)
  (data_wr_issued && data_wr_released) |=> (pending == $past(pending));
endproperty
a_fence_gauge : assert property (p_fence_gauge_sane);
 
// ---- LIVENESS, with its assumptions stated separately ----------------
 
// A1: whatever blocks the oldest eventually clears. PCIe does not guarantee
// this; it depends on the destination, the fabric and flow control.
assume property (@(posedge clk) disable iff (!rst_n)
  oldest_blocked |-> s_eventually !oldest_blocked);
// A2: the downstream consumer eventually accepts.
assume property (@(posedge clk) disable iff (!rst_n)
  out_valid |-> s_eventually out_ready);
 
// L1: under A1-A2, the oldest transaction eventually issues. Without A1 it
// is simply false — a permanently blocked destination stalls forever, and
// that is not a scheduler bug.
property p_oldest_eventually_issues;
  @(posedge clk) disable iff (!rst_n)
  occ_q[0] |-> s_eventually (out_valid && out_ready && issue_old);
endproperty
a_liveness : assert property (p_oldest_eventually_issues);

P2 and P5 are a pair, and they fail on opposite mistakes. P2 catches a scheduler that reorders when it must not. P5 catches one that reorders when it merely may — treating Relaxed Ordering as an instruction rather than a permission, which produces reordering the Requester did not ask for and which no rule required.

P3 is stated as an obligation, which is unusual and deliberate. Most ordering properties forbid something. This one requires that a bypass be available — because §5's mandatory-passing rules exist to prevent deadlock, and a scheduler that is merely conservative everywhere satisfies P1, P2, P4 and P9 while being unusable.

P9 is the "correctly stalled" property and it is where the value is. A scheduler that always issues whatever is ready passes P7 (nothing dropped), passes P8 (age invariant intact), and fails only P1 and P9 — the two that encode the rules. Without them the design looks correct and fast.

The liveness assumptions are load-bearing. L1 is false without A1: a permanently blocked destination stalls the oldest forever, and that is a system condition, not a scheduler defect. Stating the assumption is what separates "this design does not add a deadlock" from "this system makes progress", and only the first is a property of the RTL.

16. A Formal Counterexample

Mutate one line — the mistake a scheduler written for throughput naturally makes:

// MUTATED — ordering ignored entirely.
wire bypass_ok = have_old && have_young && oldest_blocked;
//                                                        ^ can_pass() dropped

Run P1 against it with the producer/consumer stimulus. Here is the counterexample, cycle by cycle.

At cycle 4 the mutated bypass_ok is true — the older is blocked, the younger exists — and can_pass(A, B) would have returned false, because both are Posted in the same Traffic Class with default attributes and §4's first row forbids it.

P1 fires at cycle 4 with a two-transaction trace. P2 fires with it, naming the specific rule.

17. Verification

Monitors observe: the enqueue interface with full ordering metadata; the issue interface with out_was_bypass; the blocking input; and the fence tracker's gauge and readiness.

The scoreboard must model a partial order

This is the chapter's most important DV lesson, and it is one that a naive scoreboard gets exactly backwards.

Rule coverage

For each published row of §4, test both directions:

And test permitted-is-not-required for every permitted row: with the older not blocked, verify the older issues first anyway (P5).

Attribute and scope

• IDO set with identical Requester IDs. Verify passing is not permitted — the identity term is load-bearing.
• IDO set with different Requester IDs. Verify permitted.
• RO and IDO both set. Verify permitted, and that the design does not depend on evaluation order.
• Different Traffic Classes with every class pair. Verify the rules are not applied across them.
• Attribute change on a control register while a transaction is queued. Verify the queued transaction's metadata does not move (P6) — Chapter 11.6's ownership rule, tested here in a scheduler.

Structural

• Simultaneous enqueue and issue, in each of: older issues and younger shifts down; younger bypasses and older stays. Verify the new transaction lands in the correct slot (P8).
• Continuous back-to-back traffic with independent blocking, long run, partial-order scoreboard active.
• Blocking asserted and released repeatedly while both slots are occupied.
• Reset with both slots occupied.

Which mutation which check kills

18. Debugging

The doorbell was observed but the descriptor is stale

§1's failure, in the field. Four things to check, in order.

The RO attribute on both writes. If either carries Relaxed Ordering, the guarantee was relinquished — and the bug is in whatever set it, not in the fabric. This is the first check because it is the most common cause and the cheapest to test.

The transaction classes. The pattern relies on both being Posted. If the descriptor update was performed by something that is not a posted write, the rule invoked does not apply.

The scheduler's pass logic, if the element is yours. §16's mutation is the shape to look for: a bypass condition that consults readiness and blocking but not the rules.

The producer's synchronisation assumption. Did software or the device assume something stronger than §4 provides — a read-as-a-fence assumption (§11), or an ordering guarantee across Traffic Classes that §4 does not extend?

And note the check that is not useful: examining the packets for malformation. Both writes are perfectly legal, which is why the analyser will show nothing wrong.

Throughput is lower than expected although Relaxed Ordering is set

RO is a permission. Nothing is required to reorder (§6).

Three candidates. There is no alternative work to promote — a bypass needs something younger and ready, and if the queue is empty the permission is unused. The scheduler's policy does not exercise bypass even when permitted, which is legal and may be deliberate. Or the bottleneck is elsewhere entirely: flow control, the destination, the return path (Chapter 12.5 §18).

The observation: count out_was_bypass events. Zero bypasses with RO set and a blocked oldest means the scheduler is not using the permission. Non-zero bypasses with unchanged throughput means the bottleneck is not ordering.

Verification flags reordered packets and the hardware is legal

The scoreboard is asserting a total order that the protocol does not define (§17).

The signature is unmistakable once you know it: the failures correlate with load, and every flagged pair turns out to be a permitted pass on inspection. The design is exercising exactly the freedom the rules grant.

The fix is structural, not a tolerance adjustment. Replace the sequence comparison with a per-pair constraint check. Loosening a total-order check until it stops firing produces a scoreboard that no longer detects illegal reordering either — and that is a worse outcome than the false failures.

The test fails only when two Requester contexts interleave

Ordering identity and scope.

Check the IDO path first. IDO's relaxation depends on the Requester identities differing; a design that drops the identity term permits passing between two transactions from the same Requester, which the rule does not.

Then check the Traffic Class scope. If the two contexts use different Traffic Classes, §4's rules do not order them relative to each other — and a scoreboard that assumed they did will flag legal behaviour. If they use the same class and the design treated them as different domains, the reverse.

The distinguishing observation: for a flagged pair, print both Requester IDs, both Traffic Classes and both attribute sets. The rule that should have applied is then a table lookup, not an investigation.

19. Common Misconceptions

• "PCIe preserves issue order globally." It preserves specific relationships. Some passing is permitted; some is required (§5).
• "No transactions may reorder." A fabric that refused all reordering would deadlock — mandatory passing exists for exactly that reason (§5).
• "Relaxed Ordering means anything may pass anything." It permits a later Memory Write, Memory Read or Completion to pass a previously issued Memory Write. That is one relaxation, not a suspension (§6).
• "Relaxed Ordering guarantees better performance." It grants a permission. Whether anything is reordered depends on the fabric, the traffic and the moment (§6, §18).
• "No Snoop is an ordering control." It concerns coherency treatment. Different mechanism, different failure mode (§7).
• "The Tag determines ordering." The correlation fields identify transactions. Ordering is stated over classes and attributes (§8).
• "Legal reordering means data corruption is acceptable." Legal reordering means the fabric did nothing wrong. The corruption came from a Requester relying on a guarantee it had relinquished (§6).
• "Packet arrival order can replace correlation." Never, under any ordering rule (§8).
• "A scoreboard should compare an exact packet sequence." It should check per-pair constraints. A total-order comparison flags legal behaviour (§17).
• "Posted writes always remain globally ordered." They must not pass each other within a Traffic Class and under default attributes — every clause of which is a scope (§4).
• "A non-posted transaction acts as a universal fence." §4's rule is specific; general visibility claims need system-level justification this chapter deliberately does not provide (§11).
• "The Data Link Layer's acknowledgement controls Transaction Layer ordering." Different layer, different scope. Module 14 opens that boundary.
• "Ordering is only a software concern." It becomes a can_pass predicate and a ready signal in RTL (§12, §14).
• "Ordering only matters for writes." Completions with the same Transaction ID must not pass each other, and that is what makes Chapter 13.3's reassembly model correct (§4).

20. Understanding Check

21. Module 13 Complete

Four chapters have taken the Completion apart and put the transaction layer's concurrency rules around it.

The learner now has the whole Transaction Layer story: what a packet is (Module 11), what memory traffic does with it (Module 12), and how responses and concurrency behave (Module 13).

Module 14 changes layer. Everything so far assumed a packet handed downward arrives intact at the other end of the link — an assumption nothing has justified. Chapter 14.1 — Reliability Goals asks what makes that assumption safe, and what the Data Link Layer does and does not promise in exchange.

The idea to carry forward: ordering is not about keeping things in order — it is about which specific observations a consumer is entitled to rely on, and which permissions the fabric needs in order to move at all.