Data Link Layer — Reliability Across One Hop

Chapter 3.1 ended on an assumption. The Transaction Layer decides what operation is being performed, hands the resulting packet down, and proceeds as though it arrives.

That assumption is not free. Something has to make it true, and this chapter is about what:

Why does PCIe need a layer dedicated to reliable movement across one Link, when the Transaction Layer has already created the transaction?

The answer turns on a scope distinction that is worth getting exactly right, because almost every misconception about this layer comes from getting it wrong.

1. Three Questions, One Packet

The cleanest way to hold the layer separation is as three questions asked about the same packet, each by a different layer, each answerable without the others.

Take the operation threaded through this module: an Endpoint has assembled data and needs it written into host memory.

The Transaction Layer asks: what operation is this? A write, directed at a particular address in the system address map, carrying this much data, requiring no data returned. Answering that requires understanding the operation and nothing about wires.

The Data Link Layer asks: did this packet get across this Link intact? Not whether the write eventually happened. Not whether the host memory was updated. Just: the thing handed to me — did the component at the other end of this connection receive it correctly, and if not, what do I do? Answering that requires no understanding of the operation whatsoever.

The Physical Layer asks: how does the information cross the connection? Answering that requires no understanding of either the operation or the delivery bookkeeping.

Each question is complete on its own. That is what makes the layering real rather than decorative — and it is why a packet can be forwarded by a Switch that has no interest in what the packet means.

2. Hop-Local, Not End-to-End

This is the section that matters most, and it is where intuition most reliably misleads.

Consider a transaction travelling from an Endpoint to the Root Complex through one Switch. Chapter 2.8 established the vocabulary precisely: that is one path made of two Links.

The Data Link Layer does not operate on the path. It operates on each Link separately.

Three consequences follow, and each is load-bearing.

There are two independent reliability relationships, not one. The Endpoint and the Switch have one, covering Link 1. The Switch and the Root Complex have another, covering Link 2. They share no state.

A Switch is a full participant in both. It is not a passive relay that lets a packet flow through untouched. On Link 1 it is the receiving partner — checking what arrived, responding to its neighbour as that relationship requires. On Link 2 it is the transmitting partner, with its own obligations toward a different neighbour.

Success on one hop implies nothing about the next. A packet can cross Link 1 perfectly and encounter trouble on Link 2. The Endpoint's Data Link Layer has no visibility into that and no responsibility for it — its relationship ended when its neighbour accepted the packet.

3. What Reliability Requires the Transmitter to Keep

Now derive the bookkeeping rather than asserting it, because the derivation is the engineering content.

Suppose a transmitter hands a packet to the layer below and it does not arrive correctly at the neighbour. Something must happen, and the options are limited: give up, or send it again.

Giving up would push the problem upward — every layer above would need to handle arbitrary packet loss, and the Transaction Layer's clean assumption from Chapter 3.1 would collapse. So: send it again.

But sending it again requires still having it. The moment a transmitter accepts responsibility for delivering a packet across a Link, it cannot discard that packet until it knows delivery succeeded. That single requirement generates everything else this layer does on the transmit side:

• Storage. The packet must be retained after transmission.
• Identity. Transmitter and receiver need a shared way to refer to a specific packet, so a "this one arrived" or "this one did not" indication is unambiguous.
• Pending state. The transmitter must know which retained packets are still unresolved.
• A release rule. Storage is finite, so something must indicate when a packet may be discarded.
• A retry path. When delivery failed, a retained packet must be selectable for resending.
• Backpressure. When storage fills, the transmitter must stop accepting new packets rather than overwrite unresolved ones.

That last point is worth pausing on. Finite retry storage means this layer must be able to refuse work from the layer above — which is why the Transaction Layer's downward interface has a ready signal at all, and why Chapter 3.1's intake stage had to handle sustained stalls.

PCIe defines specific normative mechanisms for the identity scheme, the integrity check, the indications exchanged between neighbours, and the rules governing retry. Modules 14 through 16 cover them. This chapter deliberately stays at the level of what problem each mechanism solves, because the architecture is comprehensible without the encodings and misleading if you learn the encodings first.

4. An Illustrative Microarchitecture

Given those requirements, one reasonable structure follows. As always: illustrative, not mandated.

The structural observation worth extracting: the transmit and receive paths are coupled through the control path, not through the data path. A packet does not flow from receive to transmit — but the outcome of receiving drives what is reported to the neighbour, and indications arriving from the neighbour drive what happens to stored entries. That coupling is where the interesting bugs live, because it is the part that cannot be verified by testing either direction alone.

5. Retry Bookkeeping in RTL

This is the most instructive hardware in the batch, because the bookkeeping problem is real, general, and easy to get subtly wrong.

// Illustrative retry-bookkeeping RTL — NOT a complete PCIe Data Link Layer.
// Models the general problem: retain a transmitted packet until its delivery
// is resolved, allow re-selection if it must be sent again, and refuse new
// work when storage is exhausted.
module retry_bookkeeping #(
  parameter int unsigned ENTRIES  = 8,
  parameter int unsigned META_W   = 64,          // opaque per-packet metadata
  localparam int unsigned IDX_W   = $clog2(ENTRIES)
) (
  input  logic                clk,
  input  logic                rst_n,
 
  // From the layer above: a packet offered for transmission
  input  logic                tx_valid,
  output logic                tx_ready,
  input  logic [META_W-1:0]   tx_meta,
 
  // Identity assigned to the accepted packet (local bookkeeping handle)
  output logic [IDX_W-1:0]    tx_id,
  output logic                tx_accept,
 
  // Resolution indication from the neighbour relationship
  input  logic                ack_valid,         // entries up to ack_id delivered
  input  logic [IDX_W-1:0]    ack_id,
 
  // Retry indication: re-send starting from retry_id
  input  logic                retry_valid,
  input  logic [IDX_W-1:0]    retry_id,
 
  // Re-selected entry presented for resending
  output logic                replay_valid,
  output logic [META_W-1:0]   replay_meta,
  output logic [IDX_W-1:0]    replay_id
);
 
  logic [META_W-1:0] meta_q   [ENTRIES];
  logic              pending_q[ENTRIES];
 
  logic [IDX_W-1:0]  alloc_ptr;   // next entry to allocate
  logic [IDX_W-1:0]  replay_ptr;  // entry currently being re-sent
  logic              replaying_q;
 
  // Occupancy is tracked explicitly rather than inferred from pointers, so a
  // full table is distinguishable from an empty one.
  logic [IDX_W:0]    occupancy;
 
  wire table_full = (occupancy == ENTRIES[IDX_W:0]);
 
  // Refuse new packets while storage is exhausted, or while re-sending: a
  // real design may allow interleaving, but forbidding it here keeps the
  // ordering invariant simple and the example honest.
  assign tx_ready  = !table_full && !replaying_q;
  assign tx_accept = tx_valid && tx_ready;
  assign tx_id     = alloc_ptr;
 
  assign replay_valid = replaying_q;
  assign replay_meta  = meta_q[replay_ptr];
  assign replay_id    = replay_ptr;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      alloc_ptr   <= '0;
      replay_ptr  <= '0;
      replaying_q <= 1'b0;
      occupancy   <= '0;
      for (int i = 0; i < ENTRIES; i++) pending_q[i] <= 1'b0;
    end else begin
      // ---- allocation ------------------------------------------------
      if (tx_accept) begin
        meta_q[alloc_ptr]    <= tx_meta;
        pending_q[alloc_ptr] <= 1'b1;
        alloc_ptr            <= alloc_ptr + 1'b1;
      end
 
      // ---- resolution: clear the acknowledged entry -------------------
      // Guarded on pending_q so a stale or duplicate indication naming an
      // entry that is not outstanding cannot corrupt occupancy.
      if (ack_valid && pending_q[ack_id]) begin
        pending_q[ack_id] <= 1'b0;
      end
 
      // ---- occupancy -------------------------------------------------
      case ({tx_accept, (ack_valid && pending_q[ack_id])})
        2'b10:   occupancy <= occupancy + 1'b1;
        2'b01:   occupancy <= occupancy - 1'b1;
        default: occupancy <= occupancy;
      endcase
 
      // ---- retry -----------------------------------------------------
      // Only an entry that is still pending may be re-selected: re-sending
      // an already-resolved entry would duplicate a delivered packet.
      if (retry_valid && pending_q[retry_id]) begin
        replay_ptr  <= retry_id;
        replaying_q <= 1'b1;
      end else if (replaying_q) begin
        replaying_q <= 1'b0;   // one entry per retry request in this model
      end
    end
  end
endmodule

What this models: the retain-until-resolved bookkeeping that any retry-capable transmitter needs, plus the backpressure that finite storage forces.

What is deliberately simplified: it resends a single entry per retry request rather than a run of packets; it carries opaque metadata instead of a real packet; there is no integrity generation or checking; ordering interactions with the layer above are reduced to "do not accept while replaying"; and the resolution model clears exactly one entry rather than a range. PCIe's actual rules differ and are normative — Modules 14–16.

What to notice — three things, and they are the whole point:

1. Both ack and retry are guarded on pending_q. An indication naming an entry that is not outstanding is ignored. Without those guards, a duplicated or stale indication corrupts occupancy or re-sends a packet that already arrived — producing a duplicate transaction at the far end. This is the most common bug in this class of hardware.
2. Occupancy is tracked explicitly, not inferred from pointer comparison, so full and empty are distinguishable.
3. tx_ready falls when storage is exhausted. The layer above will be stalled, which is exactly why Chapter 3.1's intake had to hold metadata stable under sustained backpressure. The two chapters' interfaces meet here.

6. Asserting the Bookkeeping Invariants

The properties worth checking are about state integrity, not about protocol encodings.

// SVA over the illustrative bookkeeping module above.
// These check the internal invariants of THIS model. They are not PCIe
// protocol requirements and should not be presented as such.
 
// P1 — a packet is only taken when both sides agree.
property p_accept_requires_handshake;
  @(posedge clk) disable iff (!rst_n)
  tx_accept |-> (tx_valid && tx_ready);
endproperty
a_accept_handshake : assert property (p_accept_requires_handshake);
 
// P2 — allocation never silently overwrites an unresolved entry.
// If the entry about to be allocated is still pending, we must not be
// accepting into it.
property p_no_silent_overwrite;
  @(posedge clk) disable iff (!rst_n)
  (tx_accept) |-> !pending_q[alloc_ptr];
endproperty
a_no_overwrite : assert property (p_no_silent_overwrite);
 
// P3 — re-selection only ever references an entry that is still outstanding.
// Re-sending a resolved entry would duplicate a packet the neighbour already
// has, which surfaces far away as a duplicated transaction.
property p_retry_only_pending;
  @(posedge clk) disable iff (!rst_n)
  (replay_valid) |-> pending_q[replay_id];
endproperty
a_retry_pending : assert property (p_retry_only_pending);
 
// P4 — storage exhaustion must produce backpressure, never overflow.
property p_backpressure_on_full;
  @(posedge clk) disable iff (!rst_n)
  (occupancy == ENTRIES) |-> !tx_ready;
endproperty
a_backpressure : assert property (p_backpressure_on_full);

Each of these encodes an engineering fear rather than a specification clause.

P2 exists because overwriting an unresolved entry loses a packet that may still need resending — and the loss is silent, surfacing much later as a transaction that never completed.

P3 exists because the opposite failure is equally bad and more confusing: re-sending an already-resolved packet delivers it twice. At the far end that is a duplicated transaction, which for a write may corrupt data. Tracing it back to a retry bookkeeping bug several hops away is genuinely difficult, which is why asserting it locally is worth the effort.

P4 exists because the alternative to backpressure is overflow, and overflow in retry storage means unresolved packets are discarded — reintroducing exactly the arbitrary loss this layer was created to prevent.

7. Verifying Hop-Local Reliability

What to assert. The bookkeeping invariants above, plus: no entry is resolved twice; every accepted packet is eventually either resolved or re-selected (a liveness property requiring an explicit assumption that indications eventually arrive); and occupancy is consistent with the number of pending entries.

What to score-board. Independently track which packets were handed down and which the receiving side accepted upward, then confirm the sets correspond — allowing for retries. This is the scoreboard that catches duplication: if the receiver's Transaction Layer accepted a packet twice, the retry path resent something already delivered. That check cannot be done from the transmit side alone, which is why a hop-level environment needs visibility on both sides of the Link.

What to generate.

• Clean delivery with no errors, as the baseline.
• Corruption indications at varying rates, including bursts, since a single injected error exercises far less than a burst does.
• Repeated failure of the same packet — does the design keep retrying, and does it eventually escalate rather than looping silently forever?
• Back-to-back packets with no gaps, to stress allocation.
• Retry storage driven to exhaustion, confirming backpressure rather than overwrite.
• Stale and duplicate indications — an indication naming an already-resolved entry, or the same indication twice. This is where P2 and P3 earn their keep, and it is a case a naive testbench never produces.
• Reset asserted while packets are outstanding, confirming state clears coherently and nothing is left half-resolved.

Coverage worth defining. Retry storage occupancy including the full case; retries triggered at different occupancy levels; retry occurring while the layer above is offering new work; and the interaction of reset with a non-empty table. Occupancy extremes are where the interesting bugs cluster, and a purely random test spends very little time there.

8. Debugging: Three Layers, Three Signatures

The practical value of this layer's scope is that it partitions the suspect list. Six signatures:

The Transaction Layer created a packet but it never advances. Look at the downward ready. If this layer is not accepting, the likely cause is exhausted retry storage — which means resolution indications are not arriving or are not clearing entries. Look at occupancy: if it is pinned at full, the resolution path is the problem, not the transmit path.

A packet is retried repeatedly. Something is genuinely failing on this hop. That points at physical transport for this specific Link — not at the transaction, whose contents are irrelevant to whether it arrived intact.

An entry is never released. A resolution indication was lost, was ignored because of a guard condition, or referenced the wrong identity. This eventually manifests as permanent backpressure upward, which looks like a hang far from its cause.

The receiver rejects traffic while the connection is operational. Suggests a check failing consistently rather than an intermittent physical fault. Consistent rejection with a healthy physical connection points at an integrity mechanism disagreement or a bookkeeping mismatch, not at the wire.

A duplicated transaction appears at the destination. Almost always a retry bookkeeping bug — an entry resent after it was already delivered. P3 is the local assertion that catches this at its source rather than several hops away.

Wrong identity association. A resolution indication clearing the wrong entry, releasing one packet while another stays pending forever. Symptoms appear on the packet that was not released, far from the actual fault.

9. Common Misconceptions

10. Understanding Check

11. Summary

The Data Link Layer exists to make the Transaction Layer's assumption true: that a packet handed down arrives. Its scope is hop-local — two components and the Link between them — and it has no view of the operation's meaning or of the path beyond its own connection.

A transaction crossing two Links involves two independent relationships sharing no state, with a Switch as a full participant in both. Success on one hop implies nothing about the next, and a hop-local delivery indication is not confirmation that an operation completed.

The transmit-side bookkeeping follows from one requirement: if a packet may need resending, it must be retained until delivery is resolved. That produces storage, identity, pending state, a release rule, a retry path, and — because storage is finite — backpressure to the layer above.

The failure modes worth guarding against are symmetric and both severe. Overwriting an unresolved entry loses a packet silently. Re-selecting a resolved entry duplicates one, appearing at the destination as a duplicated transaction that is very hard to trace back. Guarding both on pending state, and asserting those guards, is the local defence against faults that otherwise surface far away.

Hold the model: this layer answers whether a packet crossed one Link intact — not what it meant, and not whether it reached the end.

12. What Comes Next

This chapter assumed a working connection. Packets were handed down and either arrived or did not, with the mechanism of arrival left unexamined.

Chapter 3.3 — Physical Layer takes that on: what remains once the upper layers have decided what information must cross, how it is turned into something a connection can carry and reconstructed at the far end, and — of practical importance to an RTL engineer — which parts of "the PHY" are digital logic you might own and which are not.

Chapters 3.4 and 3.5 then make the responsibility boundaries rigorous and trace a packet's complete journey down and back up the stack.

Revisit Transaction Layer for the layer whose assumption this one satisfies, or Point-to-Point Links for the Link-versus-path distinction this chapter's scope depends on. Browse the full path on the PCIe tutorials index.