Reliability Goals — The Contract Under the Transaction Layer

Eleven modules of Transaction Layer material rest on an assumption none of them examined.

Chapter 12.1 launched a Request and waited for its Completion. Chapter 13.3 accounted for returned bytes. Chapter 13.4 reasoned about which packets may pass which. Every one of them assumed that a packet handed downward arrives at the other end of the Link intact.

A serial link running at multi-gigatransfer rates does not provide that for free.

What reliability problem remains after the Transaction Layer has built a correct TLP, and what contract does the Data Link Layer provide across one PCIe Link?

1. What This Chapter Adds

Chapter 3.2 already introduced this layer. It established that reliability is hop-local, derived why a transmitter must retain a packet it has sent, and built retry-bookkeeping RTL for the general problem of tracking packets that may need resending.

This chapter does not repeat any of that. Read 3.2 first if you have not; the hop-local argument in particular is developed there and is assumed here.

What Module 14 opens with instead is the contract, stated precisely enough to be relied on — and the one question 3.2 could not ask because Modules 10 through 13 had not happened yet:

2. The Contract

State it as a contract, because that is what a layer boundary is.

3. What Can Go Wrong on a Link

At the level this chapter needs — what problem must be solved, not how.

No probabilities appear here, and none should. Error rates depend on the physical link, the generation, the channel and the environment. What the architecture needs is that errors are possible, not how likely.

And notice the third row is created by the second row's solution. Retransmission solves non-acceptance and immediately introduces the risk that a packet is delivered upward twice. A reliability mechanism that only retransmitted would trade one failure for another — which is why identity (§6) is part of the problem statement and not an implementation detail.

4. Completion Is Not Acknowledgement

The comparison Chapter 3.2 could only gesture at. Module 13 now makes it sharp.

5. One Packet, Several Transmissions

The identity distinction, and it is this chapter's central new idea.

Transaction Layer:   ONE packet instance, handed down once
Data Link Layer:     ONE OR MORE transmissions of that instance
Receiving TL:        ONE packet delivered upward

All three are simultaneously true, and a design or a testbench that collapses any two of them is wrong.

6. The Layers, and the Scope of Each

The dashed edge is the one to read carefully. The Switch does not pass Link 1's reliability state to Link 2. It terminates one relationship and originates another — which is why a packet retransmitted on Link 1 may cross Link 2 exactly once, and why §11's cross-layer debugging has to ask which Link before it asks anything else.

7. Where the Mechanisms Come From

Every named mechanism in Module 14 is a consequence of §2's contract. Here is the derivation, and nothing more — each mechanism's actual behaviour belongs to its own chapter.

8. RTL — Single-Packet Reliable Transmit Slot

// SYNTHESIZABLE. An abstract Data Link transmit ownership model: a packet
// accepted from the Transaction Layer is RETAINED across transmission until
// an abstract reliability event retires it, and may be sent again unchanged.
// The ownership boundary is a NORMATIVE consequence of the contract
// (section 2). The single slot, the state names, and the abstract ack/retry
// events are ILLUSTRATIVE — see the label above.
module dll_tx_retry_slot #(
  parameter int PKT_W = 128            // opaque TLP payload/metadata
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- From the Transaction Layer --------------------------------------
  input  logic              tl_valid,
  output logic              tl_ready,
  input  logic [PKT_W-1:0]  tl_packet,
 
  // ---- To the Physical Layer -------------------------------------------
  output logic              send_valid,
  input  logic              send_ready,
  output logic [PKT_W-1:0]  send_packet,
  // Telemetry: this is a retransmission, not a first send. NOT a wire
  // signal — it exists so a testbench can distinguish transmissions from
  // packets (section 5).
  output logic              send_is_replay,
 
  // ---- Abstract reliability events -------------------------------------
  // Results of Data Link machinery this module does not implement.
  input  logic              link_ack,
  input  logic              link_retry,
 
  output logic              slot_busy,
  // An acknowledgement arrived with nothing outstanding to acknowledge.
  output logic              spurious_ack_error
);
 
  typedef enum logic [1:0] {
    S_EMPTY    = 2'd0,   // no packet owned
    S_TO_SEND  = 2'd1,   // owned, not yet transmitted
    S_WAIT_ACK = 2'd2    // owned and transmitted, awaiting resolution
  } slot_e;
 
  slot_e             state_q;
  logic [PKT_W-1:0]  pkt_q;
  logic              replay_q;
  logic              spur_q;
 
  // A new packet is accepted ONLY when nothing is owned. This is the whole
  // contract in one expression: an unretired packet can never be overwritten
  // (section 2), which is why a finite slot must be able to refuse work.
  assign tl_ready = (state_q == S_EMPTY);
 
  assign send_valid     = (state_q == S_TO_SEND);
  assign send_packet    = pkt_q;              // ALWAYS from storage
  assign send_is_replay = replay_q;
  assign slot_busy      = (state_q != S_EMPTY);
  assign spurious_ack_error = spur_q;
 
  wire tl_accept  = tl_valid && tl_ready;
  wire sent       = send_valid && send_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state_q <= S_EMPTY; pkt_q <= '0; replay_q <= 1'b0; spur_q <= 1'b0;
    end else begin
      // Reported: an ack with nothing outstanding cannot free anything, and
      // must not be allowed to look like it did.
      if (link_ack && (state_q != S_WAIT_ACK)) spur_q <= 1'b1;
 
      unique case (state_q)
 
        S_EMPTY: begin
          if (tl_accept) begin
            // The packet is COPIED. Everything downstream reads pkt_q, so
            // the Transaction Layer may reuse its bus immediately — and a
            // replay is guaranteed to send the same bits.
            pkt_q    <= tl_packet;
            replay_q <= 1'b0;
            state_q  <= S_TO_SEND;
          end
        end
 
        S_TO_SEND: begin
          if (sent) state_q <= S_WAIT_ACK;
        end
 
        S_WAIT_ACK: begin
          // SAME-CYCLE CONTRACT, stated by priority rather than left to
          // if-ordering: a retry event takes precedence over an ack. If both
          // arrive, the packet is resent — the conservative direction, since
          // resending a delivered packet is a duplicate the receiver can
          // reject, while retiring an undelivered one loses it forever.
          if (link_retry) begin
            replay_q <= 1'b1;       // the packet itself is NOT touched
            state_q  <= S_TO_SEND;
          end else if (link_ack) begin
            state_q  <= S_EMPTY;
            replay_q <= 1'b0;
          end
        end
 
        default: state_q <= S_EMPTY;
      endcase
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. A three-state ownership model around one retained packet. The packet register is written in exactly one place — on acceptance — and never on a retry, which is what makes "a replay sends the same bits" structural rather than a property to be hoped for.

State. The slot state, the packet, a replay flag, a sticky spurious-acknowledgement flag.

Cycle behaviour.

Why retry wins the same-cycle race. The two outcomes are not symmetric. Resending a packet that was in fact delivered produces a duplicate, which the receiver's machinery is built to reject. Retiring a packet that was not delivered loses it permanently, and the layer above — which was promised it need not implement hop-local retry — has no mechanism to recover. The asymmetry of consequences chooses the priority, and encoding it as an explicit if/else if rather than leaving it to statement order is the difference between a decision and an accident.

Contract. The Transaction Layer relies on tl_ready being low while a packet is owned, and may reuse its bus the cycle after acceptance. The layer below relies on send_packet being byte-identical across every transmission of one packet.

Failure — five. Freeing on sent rather than on link_ack discards a packet the moment it is transmitted, which is the whole failure this layer exists to prevent. Driving send_packet from tl_packet rather than pkt_q means a replay sends whatever the Transaction Layer's bus now holds. Accepting a new packet while S_WAIT_ACK overwrites an unretired one. Allowing link_ack to free from S_TO_SEND retires a packet that was never sent. And treating a retry as a new Transaction Layer acceptance creates a second transaction from one (§5).

Deliberately simplified: one entry, so no oldest-unacknowledged tracking, no replay walk, no wrap, no sizing — Chapter 14.4 owns all of it; abstract acknowledgement events rather than the protocol that produces them (14.2, 14.3); no identity scheme (14.5); no integrity generation; no retry limit or give-up behaviour.

9. RTL — Receive Acceptance Filter

// SYNTHESIZABLE. The receive-side boundary: a packet reaches the Transaction
// Layer only when Link-local acceptance conditions pass.
// That a corrupted packet must not be delivered upward as a normal accepted
// TLP is a NORMATIVE consequence of the contract (section 2). The abstract
// integrity input, the error outputs and the interface are ILLUSTRATIVE —
// this module does NOT implement LCRC or any integrity computation.
module dll_rx_accept #(
  parameter int PKT_W = 128
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- From the Physical Layer -----------------------------------------
  input  logic              phy_valid,
  input  logic [PKT_W-1:0]  phy_packet,
  // ABSTRACT integrity verdict, produced by machinery this module does not
  // implement. Not an LCRC and not a wire signal.
  input  logic              phy_integrity_ok,
  // ABSTRACT duplicate verdict from the identity machinery (Chapter 14.5).
  input  logic              phy_is_duplicate,
 
  // ---- To the Transaction Layer ----------------------------------------
  output logic              tl_rx_valid,
  input  logic              tl_rx_ready,
  output logic [PKT_W-1:0]  tl_rx_packet,
 
  // ---- Reported conditions, and the outcome the transmitter needs -------
  output logic              integrity_error,
  output logic              duplicate_dropped,
  // A retry is warranted for this Link. NOT a NAK DLLP — Chapter 14.3 owns
  // the actual mechanism and its exact conditions.
  output logic              request_retry
);
 
  // ACCEPTANCE, and the order of the terms is the contract. Integrity is
  // checked FIRST: a corrupted packet's identity fields cannot be trusted,
  // so asking whether it is a duplicate is meaningless until integrity
  // passes.
  wire integrity_bad = phy_valid && !phy_integrity_ok;
  wire is_dup        = phy_valid &&  phy_integrity_ok && phy_is_duplicate;
  wire deliver       = phy_valid &&  phy_integrity_ok && !phy_is_duplicate;
 
  // Registered single-entry stage, so a stalled Transaction Layer cannot
  // cause a packet to be dropped silently.
  logic             v_q;
  logic [PKT_W-1:0] pkt_q;
  logic             ie_q, dd_q;
 
  wire consume = v_q && tl_rx_ready;
  wire can_take = !v_q || tl_rx_ready;
 
  assign tl_rx_valid       = v_q;
  assign tl_rx_packet      = pkt_q;
  assign integrity_error   = ie_q;
  assign duplicate_dropped = dd_q;
  // A failed-integrity packet warrants a retry; a duplicate does not — it
  // means an earlier transmission already succeeded from this receiver's
  // point of view.
  assign request_retry     = integrity_bad;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_q <= 1'b0; pkt_q <= '0; ie_q <= 1'b0; dd_q <= 1'b0;
    end else begin
      if (deliver && can_take) begin
        v_q   <= 1'b1;
        pkt_q <= phy_packet;
      end else if (consume) begin
        v_q <= 1'b0;
      end
 
      // Reported. NEITHER path writes pkt_q, so neither can reach the
      // Transaction Layer — the acceptance boundary is structural, not a
      // downstream filter someone might forget to apply.
      if (integrity_bad) ie_q <= 1'b1;
      if (is_dup)        dd_q <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. A gate with three outcomes — deliver, drop as duplicate, drop and request retry — and only the first writes the packet register. A corrupted packet has no path upward.

Why the term order matters. Integrity is evaluated before duplicate. A corrupted packet's identity fields are themselves suspect, so asking "is this a duplicate" of a packet that failed its integrity check is asking a question about data you have just established is unreliable.

Why a duplicate does not request a retry. A duplicate means an earlier transmission of that packet was already accepted here. Asking for it again would be asking for a third copy — the transmitter's view and the receiver's have simply not converged yet, and that is what the identity machinery (14.5) is for.

Contract. The Transaction Layer relies on every packet it receives having passed the acceptance conditions, and on the packet being stable while it stalls. It receives each logical packet exactly once regardless of how many times it crossed the Link.

Failure — four. Delivering upward and reporting the integrity error in parallel is the worst version: the Transaction Layer processes a corrupted packet and an error is logged, so the design looks instrumented while being broken. Checking duplicate before integrity trusts fields from a corrupted packet. Making tl_rx_valid combinational on phy_valid drops packets whenever the Transaction Layer stalls. And requesting a retry for a duplicate causes an unnecessary retransmission loop.

Deliberately simplified: abstract integrity and duplicate verdicts rather than the mechanisms that compute them; one packet at a time; no ordering or gap handling (14.5); no acknowledgement generation (14.2).

10. Assertions

// SVA over dll_tx_retry_slot and dll_rx_accept. These assert the LOCAL
// ownership and acceptance contracts of sections 8 and 9 plus the contract
// of section 2. They assert NOTHING about ACK/NAK protocol behaviour,
// sequence numbers, LCRC, or replay-buffer management — Chapters 14.2
// through 14.5 own those.
 
// ---- SAFETY ----------------------------------------------------------
 
// P1: THE CENTRAL PROPERTY. A packet accepted from the Transaction Layer is
// retained until an abstract reliability retirement. Transmission alone
// never frees it.
property p_retained_until_retirement;
  @(posedge clk) disable iff (!rst_n)
  (sent && !link_retry && !link_ack) |=> (state_q == S_WAIT_ACK) && slot_busy;
endproperty
a_retained : assert property (p_retained_until_retirement);
 
// P2: the packet is BIT-IDENTICAL across every transmission. The property
// that makes "a replay sends the same packet" checkable rather than assumed.
property p_packet_stable_while_owned;
  @(posedge clk) disable iff (!rst_n)
  (slot_busy && !(link_ack && !link_retry)) |=> $stable(pkt_q);
endproperty
a_pkt_stable : assert property (p_packet_stable_while_owned);
 
// P3: a retry resends, and resends the SAME packet.
property p_retry_resends_same;
  @(posedge clk) disable iff (!rst_n)
  (link_retry && (state_q == S_WAIT_ACK))
    |=> (send_valid && (send_packet == $past(pkt_q)) && send_is_replay);
endproperty
a_retry_same : assert property (p_retry_resends_same);
 
// P4: an acknowledgement frees ONLY an outstanding transmitted packet. One
// arriving in any other state frees nothing and is reported.
property p_ack_frees_only_outstanding;
  @(posedge clk) disable iff (!rst_n)
  (link_ack && (state_q != S_WAIT_ACK))
    |=> ($stable(state_q) || (state_q != S_EMPTY)) && spurious_ack_error;
endproperty
a_ack_scope : assert property (p_ack_frees_only_outstanding);
 
// P5: NO OVERWRITE. A new Transaction Layer packet can never displace an
// unretired one — which is why a finite slot must be able to refuse work.
property p_no_overwrite;
  @(posedge clk) disable iff (!rst_n)
  slot_busy |-> !tl_ready;
endproperty
a_no_overwrite : assert property (p_no_overwrite);
 
// P6: retirement happens at most once per accepted packet.
// (accept_count / retire_count are testbench counters.)
property p_retire_once;
  @(posedge clk) disable iff (!rst_n)
  ((state_q == S_WAIT_ACK) && link_ack && !link_retry)
    |-> (retire_count + 1 <= accept_count);
endproperty
a_retire_once : assert property (p_retire_once);
 
// P7: A REPLAY IS NOT A NEW ACCEPTANCE. The identity property of section 5,
// stated where it is checkable: a retransmission must never coincide with
// taking a new packet from the Transaction Layer.
property p_replay_is_not_new_acceptance;
  @(posedge clk) disable iff (!rst_n)
  send_is_replay |-> !tl_accept;
endproperty
a_replay_identity : assert property (p_replay_is_not_new_acceptance);
 
// P8: SAME-CYCLE ack and retry resolves to RESEND, not retire (section 8).
property p_retry_wins_race;
  @(posedge clk) disable iff (!rst_n)
  ((state_q == S_WAIT_ACK) && link_ack && link_retry)
    |=> (state_q == S_TO_SEND) && slot_busy;
endproperty
a_retry_priority : assert property (p_retry_wins_race);
 
// P9: THE RECEIVE BOUNDARY. A packet failing its integrity check is NEVER
// delivered upward. Stated over the delivery event, so a design that
// delivers and logs in parallel fails.
property p_corrupt_never_delivered;
  @(posedge clk) disable iff (!rst_n)
  (phy_valid && !phy_integrity_ok) |=> !$rose(tl_rx_valid);
endproperty
a_no_corrupt_up : assert property (p_corrupt_never_delivered);
 
// P10: a duplicate is dropped, not delivered.
property p_duplicate_not_delivered;
  @(posedge clk) disable iff (!rst_n)
  (phy_valid && phy_integrity_ok && phy_is_duplicate) |=> !$rose(tl_rx_valid);
endproperty
a_no_dup_up : assert property (p_duplicate_not_delivered);
 
// P11: a duplicate does not trigger a retry request.
property p_duplicate_no_retry;
  @(posedge clk) disable iff (!rst_n)
  (phy_valid && phy_integrity_ok && phy_is_duplicate) |-> !request_retry;
endproperty
a_dup_no_retry : assert property (p_duplicate_no_retry);
 
// P12: OWNERSHIP on receive. A delivered packet is stable while the
// Transaction Layer stalls.
property p_rx_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
  (tl_rx_valid && !tl_rx_ready) |=> (tl_rx_valid && $stable(tl_rx_packet));
endproperty
a_rx_stable : assert property (p_rx_stable_under_stall);
 
// P13: reset clears the LOCAL teaching state of these two blocks. It says
// nothing about PCIe's system reset behaviour, which later chapters own.
property p_reset_clears_local;
  @(posedge clk)
  !rst_n |=> ((state_q == S_EMPTY) && !slot_busy && !tl_rx_valid);
endproperty
a_reset : assert property (p_reset_clears_local);
 
// ---- LIVENESS, with assumptions stated ------------------------------
 
// A1: an abstract reliability event eventually resolves a transmitted
// packet. PCIe does NOT guarantee an acknowledgement arrives — that is what
// retry limits and Link recovery exist for — so this is an assumption about
// a well-behaved environment, not a protocol fact.
assume property (@(posedge clk) disable iff (!rst_n)
  (state_q == S_WAIT_ACK) |-> s_eventually (link_ack || link_retry));
// A2: the Physical Layer eventually accepts a transmission.
assume property (@(posedge clk) disable iff (!rst_n)
  send_valid |-> s_eventually send_ready);
// A3: retries do not continue forever.
assume property (@(posedge clk) disable iff (!rst_n)
  link_retry |-> s_eventually link_ack);
 
// L1: under A1-A3, the slot eventually becomes reusable.
property p_slot_eventually_free;
  @(posedge clk) disable iff (!rst_n)
  tl_accept |-> s_eventually tl_ready;
endproperty
a_liveness : assert property (p_slot_eventually_free);

P1 and P2 are the ownership pair. P1 says transmission does not free the packet; P2 says the packet does not change while owned — and P2 is the one that catches a design driving send_packet from the Transaction Layer's live bus, which passes P1 perfectly and replays garbage.

P7 is §5's identity distinction, made checkable. It is a small property with an outsized job: a retransmission must never coincide with taking a new packet from above. A design that treated a retry as a fresh acceptance would allocate new state for it, and P7 fires the first time it happens.

P9 is stated over $rose(tl_rx_valid) deliberately. The failure it targets is delivering upward and logging the error in parallel — a design that looks well-instrumented and is broken. Asserting only that integrity_error sets would pass that design.

A3 is worth reading twice. PCIe does not guarantee that retries eventually succeed; a link that keeps failing is handled by retry limits and recovery mechanisms this chapter does not teach. L1 is therefore a statement that this design does not add a deadlock, not that the system makes progress — and the assumption is where a reviewer learns which one they are getting.

11. Verification

Monitors observe: the Transaction Layer interface with a shadow packet identity; the transmit interface with send_is_replay; the abstract reliability events; the physical receive interface with its verdicts; and the Transaction Layer receive interface.

The scoreboard tracks logical packets, not transmissions

This is the chapter's central DV lesson and it inverts a natural instinct.

// VERIFICATION-ONLY. One entry per LOGICAL Transaction Layer packet.
// Transmissions are counted as an attribute of that entry, never as
// separate objects — because a replay is not a new packet (section 5).
typedef struct {
  int          id;              // testbench shadow identity, NOT a PCIe field
  bit [127:0]  packet;          // as accepted from the TL
  int          transmissions;   // how many times it crossed the Link
  bit          retired;
} sb_tlp_t;
 
// On TL acceptance : create ONE entry
// On each send     : increment transmissions; assert packet is UNCHANGED
// On retirement    : assert retired exactly once
// On TL delivery   : assert this logical packet is delivered EXACTLY ONCE,
//                    regardless of transmissions

transmissions may be any number ≥ 1 and that is not an error. A scoreboard that treated each send as a new packet would report a protocol violation on correct hardware — §5's third failure mode, and the one that gets a working design rejected.

And the assertion on each send is the valuable one: the packet bits must be identical to what was accepted. That is what proves a replay replayed rather than re-derived.

Transmit

• Normal send followed by acknowledgement. One transmission, one retirement, slot free.
• send_ready stalled for a long run. Verify send_valid holds and the packet is stable.
• Acknowledgement delayed for many cycles. Verify tl_ready stays low throughout (P5).
• One retry, then acknowledgement. Verify two transmissions, identical bits (P3), one retirement.
• Several retries, then acknowledgement. Verify N transmissions, one logical packet, one retirement.
• A retry while send_ready is low. Verify the state moves to S_TO_SEND and waits.
• Acknowledgement with nothing outstanding. Verify spurious_ack_error and that nothing is freed (P4).
• Acknowledgement in S_TO_SEND — before transmission. Verify it does not retire an unsent packet.
• A new Transaction Layer packet offered while the slot is occupied. Verify tl_ready is low and the retained packet is untouched (P5).
• Acknowledgement and retry in the same cycle. Verify retry wins (P8).
• Acknowledgement and a new Transaction Layer offer in the same cycle. Verify the retirement completes and the new packet is accepted.
• Reset in each of the three states.

Receive

• A clean packet. Delivered once.
• phy_integrity_ok low. Verify nothing is delivered (P9), integrity_error sets, request_retry asserts.
• A duplicate. Verify nothing is delivered (P10) and no retry is requested (P11).
• A packet that is both corrupt and flagged duplicate. Verify the integrity path wins — the duplicate verdict on a corrupt packet is not trustworthy (§9).
• The Transaction Layer stalled while packets arrive. Verify the held packet is stable (P12) and nothing is dropped silently.
• A corrupt packet immediately followed by a clean one. Verify the clean one is delivered and the corrupt one left no residue.

Fault injection

12. Cross-Layer Debugging

These are the scenarios where knowing the layer boundary is the entire skill.

The Transaction Layer shows one Memory Write; the analyser shows it twice

This may be completely correct. §5: one Transaction Layer packet, several Link transmissions.

Do not diagnose a duplicated transaction until you have checked the layer. The order of questions:

1. Does the receiving Transaction Layer see it once or twice? Once → the Data Link Layer did its job, and there is no bug. Twice → now there is one.
2. Was send_is_replay set on the second transmission? Set → it was a retransmission of one packet. Clear → two separate acceptances, which is a transmit-side identity bug (P7).
3. Which Link? A packet retransmitted on Link 1 may cross Link 2 once (§6). Analyser placement determines what you are counting, and comparing counts from different Links proves nothing.

The instinct to fix is counting transmissions and calling the total a transaction count. They are different quantities measured at different boundaries.

The Transaction Layer receives a corrupted packet

An acceptance-boundary failure — the contract in §2 was broken.

The packet should have been stopped at §9's filter. Two possibilities.

The integrity verdict said the packet was fine. Then the fault is in whatever computes it, not in the acceptance logic — and that is a different investigation entirely.

The verdict said it was bad and the packet was delivered anyway. That is §9's first failure: delivered and logged in parallel. Check whether integrity_error is also set for that packet — if both the error flag and the delivery are present, the design is instrumented and broken, which is the version that survives longest.

The transmitter stops accepting new TLPs after one error

The slot never retired, and there are exactly three ways.

The reliability event never arrived. state_q sits in S_WAIT_ACK forever. That is an environment condition, not necessarily a bug — assumption A1 in §10 says so explicitly, and a link that has genuinely failed is handled by mechanisms this chapter does not teach.

The acknowledgement arrived and was rejected. Check spurious_ack_error: if it is set, the acknowledgement came while the state was not S_WAIT_ACK — a race between the transmission handshake and the event.

A retry loop. send_is_replay toggling continuously with no retirement means retries keep arriving. If the receiver is requesting retries for duplicates (§9's fourth failure), the two sides have built a loop: the transmitter resends, the receiver recognises a duplicate and asks again.

The distinguishing observation is one signal: does send_valid pulse repeatedly, or is it flat? Repeated → a retry loop. Flat → nothing is resolving.

The same transaction side effect happens twice at the far end

The most valuable scenario in the chapter, because two very different faults produce it.

Fault A — the receiving Data Link Layer delivered a retransmission upward twice. The Link carried one packet twice, legitimately; the acceptance boundary failed to recognise the second as a duplicate, so the Transaction Layer performed the operation twice (§5).

Fault B — the Transaction Layer genuinely issued the operation twice. Two separate packets, two separate acceptances, both legitimately delivered. The Data Link Layer did exactly what it was told.

They are distinguished at the transmitting Transaction Layer's interface, not at the receiver:

And note that the receiver alone cannot tell them apart. It sees two packets. The information that separates them exists only at the other end of the Link, which is why cross-layer debugging means instrumenting both sides before forming a hypothesis — and why send_is_replay exists in §8 as telemetry despite not being a wire signal.

13. Common Misconceptions

• "A Data Link acknowledgement is the same as a Completion." Different layers, different scopes, different meanings. One says the neighbour got the packet; the other says the operation was serviced (§4).
• "The Data Link Layer decides whether a Memory Read or Write is valid." It does not interpret the transaction at all. It moves packets (§2).
• "The Transaction Layer should resend a TLP when a bit error is detected." That is precisely what this layer exists to prevent (§2).
• "Every retransmission is a new transaction." It is one packet instance sent again. The receiving Transaction Layer sees it once (§5).
• "Data Link reliability is end-to-end across the fabric." It is per-Link. A Switch terminates one relationship and originates another (Chapter 3.2 §2, §6).
• "A Switch forwards one Link's acknowledgement to the next Link." The relationships share no state. Each Link's reliability is independent (§6).
• "Replay means software sees duplicate requests." Only if the receive-side acceptance boundary is broken. Correct hardware delivers each logical packet once (§9).
• "A packet may be discarded immediately after the Physical Layer transmits it." It must be retained until an abstract reliability event retires it — the single requirement from which everything in Module 14 follows (§7, §8).
• "An acknowledgement means the remote application consumed the transaction." It means the neighbouring component received the packet. Nothing more (§2, §4).
• "A negative indication is a Completion error." Different layer. Completion status is Chapter 13.2's; Link-local indications are 14.3's.
• "A Sequence Number is a Requester Tag." A Tag correlates a Completion to a Request end to end (Chapter 11.3 §6). A Link-local identity scheme is a different mechanism at a different scope (14.5).
• "LCRC and ECRC are the same mechanism." They have different scopes — one Link-local, one end-to-end. This chapter teaches neither, and conflating them is a scope error rather than a detail.

14. Understanding Check

15. What's Next

Module 14 opens with a contract rather than a mechanism, and that is deliberate. Everything the rest of the module builds — retained storage, a shared identity, positive and negative indications, the packets that carry them — follows from one requirement: a transmitter that has promised to deliver a packet cannot discard it until it knows the promise was kept.

Chapter 14.2 takes positive acknowledgement: what it actually says, when it is sent, and what a transmitter may conclude from it. Chapter 14.3 takes the negative indication and the retransmission it triggers. Chapter 14.4 replaces §8's single slot with a real replay buffer — many entries, oldest-unacknowledged tracking, the replay walk, and how to size it. Chapter 14.5 supplies the identity scheme that §5's whole argument depends on and that §9's duplicate verdict was an abstraction of.

Module 15 then covers the packets that carry these indications, which are not TLPs at all.

The idea to carry forward: the layer below does not make errors impossible — it makes them invisible to the layer above, at the cost of a packet count that no longer matches.