PCIe vs Ethernet — Where the Cost of Overload Lands

Chapter 28.1 compared PCIe with a fabric that shares its model — load/store transactions between a requester and a completer — and disagrees about mechanism.

Ethernet does not share the model. It does not address memory, it returns no Completion, and it is permitted to discard what it cannot carry. Comparing the two is therefore more useful than comparing PCIe with AXI, because it isolates what a load/store fabric actually requires.

1. Sources, Scope, and What Is Being Compared

2. Two Jobs That Look Similar and Are Not

Both move data between two devices over a serial link with a physical layer, a framing layer and error detection. The resemblance ends there.

The row that generates the others is the third. PCIe has a request/response model: a Memory Read is a question, and the fabric is obliged to deliver an answer to a specific requester. Ethernet has a delivery model: a frame is handed over, and whether anything comes back is entirely the business of layers above.

That difference makes "which is faster" an unanswerable question and "which guarantees what" the useful one.

3. Delivery Contract: The Choice Everything Follows From

§11 measured the consequence with identical traffic. The same 59,405 units of overload appear as sender stalls in one fabric and discarded frames in the other. Delivered counts are identical.

Four things follow, and each is a real design consequence.

PCIe's Transaction Layer needs no retransmission. There is a replay mechanism at the Data Link Layer (14.4), and it exists for a different reason — a TLP corrupted in transit — not for congestion. Nothing above it re-sends anything, because nothing above it is ever dropped for lack of space.

Ethernet needs a layer that notices loss. TCP exists because the fabric beneath it may discard. That layer is not free: it costs sequence numbers, acknowledgements, timers, retransmission buffers and congestion control — an entire protocol whose job is to repair a contract the fabric declined to make.

PCIe's cost lands on the sender, and that creates failure modes Ethernet does not have. A stalled sender can stall the agent behind it, and if two agents stall waiting for each other, the link stops permanently with no error anywhere — 25.8 is an entire chapter about a fault that only exists because the fabric refuses to drop.

And Ethernet's cost lands on the endpoints, which must detect and recover. Neither contract is better. They are different placements of the same unavoidable cost, and the right one depends on whether the sender can usefully wait.

4. Addressing: A Location vs A Recipient

A PCIe memory transaction carries an address that means something to the receiver's decode logic. The address selects a region a device claimed during enumeration (25.5 §3), and the fabric routes on it (11.5).

An Ethernet frame carries a destination address that identifies a device, not a location within it. What the frame means is decided entirely by the receiving device's software.

The practical difference:

And there is a consequence for debugging. A PCIe transaction to an unclaimed address produces a defined response — a UR that names the failure (13.2). An Ethernet frame to a nonexistent destination produces nothing at all. The absence of a reply is the only signal, and it is indistinguishable from a dozen other causes.

5. The Response Model

This is the difference most often overlooked, and it drives a large fraction of PCIe's complexity.

A PCIe Memory Read is a question the fabric must answer. The requester allocates a Tag, holds context for the outstanding request, and waits. The answer may arrive as several Completions (13.3), and matching them to the request is the requester's job (23.5).

Ethernet has no read. A device that wants data from another sends a frame asking for it, and the answer comes back as an entirely separate frame, correlated by whatever the application layer chose. The fabric has no notion that the two are related.

What PCIe needs because it has reads:

• Tags and an outstanding-request table;
• completion buffer space, reserved before issue (23.6 §4);
• a completion timeout, because a question can go unanswered (25.7);
• ordering rules relating Completions to other traffic (13.4);
• and split-completion accounting by byte count.

None of that exists in Ethernet, and none of it is optional in PCIe. It is the price of the request/response model, and it is worth recognising that a designer who wants only writes could avoid nearly all of it.

6. Where Flow Control Lives

PCIe's flow control is between adjacent link partners and is always present. Credits are exchanged during initialisation and returned continuously (15.2, 16.6). Every hop applies it independently — it is link-local, not end-to-end.

Ethernet's base service has no flow control. Optional link-level mechanisms exist in some deployments, and end-to-end congestion control lives in TCP, several layers up. Whether either is present is a property of the deployment, not of the fabric.

The structural consequence:

That last row is the honest summary of the trade. A fabric that can always drop can never deadlock; a fabric that never drops can. PCIe chose the second and pays for it with an entire class of failure that has no error bit.

7. The Comparison, Drawn

Read the two lower boxes together. The overload is the same quantity in both fabrics. One design turns it into latency at the sender; the other turns it into work for a layer that does not exist yet.

8. RTL — The Two Receivers

Block 1 — a lossless receiver. Its correctness argument is that overrun is impossible.

// Credit-based: the receiver ADVERTISES space and the sender may not exceed
// it. This module's job is to return credits accurately as it drains.
module lossless_rx #(
  parameter int unsigned DEPTH = 64
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                             tlp_valid,     // a TLP the sender was PERMITTED to send
  input  logic [63:0]                      tlp_data,
  input  logic                             consumer_take,
  output logic                             cred_return_valid,
  output logic [$clog2(DEPTH+1)-1:0]       level,
  output logic                             overrun        // must be PROVABLY dead
);
 
  logic [63:0] mem [DEPTH];
  logic [$clog2(DEPTH)-1:0] wr, rd;
 
  // The receiver does not gate the sender here — it already did, by granting
  // a finite number of credits. `overrun` exists only as an instrument: if it
  // ever asserts, the credit accounting upstream is broken (28.1 §11 P1).
  assign overrun = tlp_valid && (level == DEPTH);
 
  // A credit is returned when an entry is FREED, not when it is written.
  // Returning on arrival would grant permission for space still occupied —
  // a credit leak in the dangerous direction (mutation 3).
  assign cred_return_valid = consumer_take && (level != 0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= '0; rd <= '0; level <= '0;
    end else begin
      if (tlp_valid && (level != DEPTH)) begin
        mem[wr] <= tlp_data;
        wr <= (wr == DEPTH-1) ? '0 : wr + 1'b1;
      end
      if (consumer_take && (level != 0))
        rd <= (rd == DEPTH-1) ? '0 : rd + 1'b1;
 
      // Arrival and drain are NOT exclusive — the four-way case again.
      case ({tlp_valid && (level != DEPTH), consumer_take && (level != 0)})
        2'b10: level <= level + 1'b1;
        2'b01: level <= level - 1'b1;
        default: ;
      endcase
    end
  end
 
endmodule

Block 2 — a best-effort receiver. Its correctness argument is that dropping is legal and must be counted.

// Best-effort: the sender was never restricted, so arrival at a full buffer
// is an expected event. The obligation is to drop deliberately and record it.
module besteffort_rx #(
  parameter int unsigned DEPTH = 64
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                             frame_valid,
  input  logic [63:0]                      frame_data,
  input  logic                             consumer_take,
  output logic [$clog2(DEPTH+1)-1:0]       level,
  output logic                             dropped,
  output logic [31:0]                      drop_count
);
 
  logic [63:0] mem [DEPTH];
  logic [$clog2(DEPTH)-1:0] wr, rd;
 
  // Dropping is a DEFINED OUTCOME here, not a failure. What would be a bug is
  // dropping silently: a discard nobody counted is indistinguishable from a
  // frame that never arrived, and the two have completely different causes.
  assign dropped = frame_valid && (level == DEPTH);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= '0; rd <= '0; level <= '0; drop_count <= '0;
    end else begin
      if (frame_valid && (level != DEPTH)) begin
        mem[wr] <= frame_data;
        wr <= (wr == DEPTH-1) ? '0 : wr + 1'b1;
      end
      if (consumer_take && (level != 0))
        rd <= (rd == DEPTH-1) ? '0 : rd + 1'b1;
      if (dropped && drop_count != 32'hFFFF_FFFF)
        drop_count <= drop_count + 32'd1;
 
      case ({frame_valid && (level != DEPTH), consumer_take && (level != 0)})
        2'b10: level <= level + 1'b1;
        2'b01: level <= level - 1'b1;
        default: ;
      endcase
    end
  end
 
endmodule

The two modules are nearly identical and their specifications are opposites. In the first, overrun asserting is a bug upstream. In the second, dropped asserting is normal operation. Same logic, different contract — and that is the chapter.

9. Same-Cycle Audit

10. Invariants

// LOSSLESS RECEIVER — the invariant is that dropping is impossible.
 
// L1 — the receiver never sees an arrival it has no room for. This is the
// property that makes the fabric lossless, and it is enforced UPSTREAM by
// credit accounting; here it is asserted as a check on that accounting.
// Assumption: the sender obeys the credit protocol.
property p_no_overrun_possible;
  @(posedge clk) disable iff (!rst_n) !overrun;
endproperty
 
// L2 — a credit is returned exactly once per entry freed. Returning on
// arrival instead grants permission for space that is still occupied.
property p_credit_per_free;
  @(posedge clk) disable iff (!rst_n)
    cred_return_valid |-> (consumer_take && (level != 0));
endproperty
 
// L3 — occupancy is conserved across simultaneous arrival and drain.
property p_level_conserved;
  @(posedge clk) disable iff (!rst_n)
    (tlp_valid && (level != DEPTH) && consumer_take && (level != 0))
      |=> (level == $past(level));
endproperty
 
// BEST-EFFORT RECEIVER — the invariant is that dropping is COUNTED.
 
// B1 — every arrival is either stored or counted as dropped. A discard that
// nobody counted is indistinguishable from a frame that never arrived, and
// those have entirely different causes (§13 case 1).
property p_every_arrival_accounted;
  @(posedge clk) disable iff (!rst_n)
    frame_valid |-> ((level != DEPTH) ^ dropped);
endproperty
 
// B2 — the drop counter advances whenever a drop occurs.
property p_drops_counted;
  @(posedge clk) disable iff (!rst_n)
    dropped |=> (drop_count > $past(drop_count));
endproperty
 
// SHARED — the same four-way case both receivers depend on.
property p_no_underflow;
  @(posedge clk) disable iff (!rst_n)
    (level == 0) |-> !(consumer_take && $past(level == 0));
endproperty

The contrast in the assertions is the whole point. p_no_overrun_possible says an event must never happen; p_every_arrival_accounted says the same event is fine as long as it is recorded. Two receivers, two specifications, one piece of logic.

11. Measured — The Same Overload, Two Contracts

Three readings.

Delivered is identical. Neither contract makes the consumer faster. The fabric's drain rate is what it is, and 120,373 units get through either way.

The 59,405 units of overload appear in exactly one column each. In the lossless fabric they are sender stalls — latency, applied to the source. In the best-effort fabric they are discarded frames — work destroyed, to be redone by someone else or not at all.

And neither column is free. Sender stalls propagate: the agent behind the sender waits too, and if two such agents wait on each other the link stops permanently (25.8). Discarded frames require a layer that notices — sequence numbers, timers, retransmission state, congestion control. The comparison is not lossless-versus-lossy; it is which cost your system can absorb.

12. Bug Bank

13. Debugging

14. Misconceptions

"PCIe and Ethernet are both packet protocols, so they work the same way." Why it sounds reasonable: both serialise data into framed units with error detection over a serial link. What actually happens: PCIe transactions name memory addresses and reads receive Completions; Ethernet frames name a device and there is no read (§2, §5). What it causes: engineers looking for a PCIe equivalent of a socket, or an Ethernet equivalent of a Completion. Neither exists.

"A PCIe packet can be dropped when the receiver is busy." Why it sounds reasonable: every other packet fabric they have used may drop. What actually happens: a transmitter without a credit does not send (§3). Overload becomes a stalled sender, and §11 measured zero drops. What it causes: recovery paths that never execute, and — worse — designs that hold a resource expecting a drop that will never come (§13 case 5).

"Lossless is strictly better." Why it sounds reasonable: losing data sounds worse than waiting. What actually happens: a fabric that cannot drop can deadlock, and one that can drop never will (25.8, §6). What it causes: underestimating a whole failure class with no error bit, on the assumption that losslessness removed the risk rather than relocating it.

"PCIe has flow control, so it can't be congested." Why it sounds reasonable: flow control is what prevents overload. What actually happens: flow control prevents buffer overrun, not congestion. A credit-starved link is congested and idle at the same time (22.3). What it causes: interpreting a stalled link as broken rather than as backpressured, and looking for an error that does not exist.

"Ethernet's error detection means frames are repaired." Why it sounds reasonable: error detection sounds adjacent to error correction, and PCIe's Data Link Layer really does replay. What actually happens: detection at this layer means a damaged frame is discarded, and recovery is somebody else's job (§3). What it causes: expecting a link-level retransmission that is not there, and misattributing loss to congestion when it was corruption or vice versa.

"PCIe needs no reliability layer because it's lossless." Why it sounds reasonable: if nothing is dropped, nothing needs resending. What actually happens: PCIe does have a reliability layer — sequence numbers, CRC, ACK/NAK and a replay buffer (14.4) — and it exists for corruption in transit, which is a different problem from congestion (§3). What it causes: conflating the two, and then being surprised that replay does not help when the real problem is a full receiver.

15. Understanding Check

Q1. The same overload is applied to a PCIe link and an Ethernet link. Both deliver the same amount of data. Where did the difference go, and what does each system now owe?

Into a stalled sender on PCIe and discarded frames on Ethernet — §11 measured 59,405 units in exactly one column each, with delivered counts identical at 120,373. PCIe now owes latency at the source, and that stall propagates to whatever is behind the sender; if two agents stall on each other the link stops permanently with no error (25.8). Ethernet now owes a recovery layer — something above it must notice the loss and resend, which costs sequence numbers, timers, retransmission buffers and congestion control. Neither is free, and neither made the consumer faster.

Q2. Why does PCIe's Transaction Layer have nothing resembling TCP, while PCIe still has a replay buffer?

Because the two solve different problems (§3). There is no end-to-end retransmission because nothing is ever dropped for lack of space — credits stop the sender first, so there is nothing for a TCP-like layer to recover. The replay buffer at the Data Link Layer exists for corruption in transit (14.4): a TLP damaged by the physical link is NAK'd and resent between adjacent link partners. Congestion loss and transit corruption are separate concerns, and conflating them leads engineers to expect replay to help with a full receiver, which it does not.

Q3. A NIC is dropping received frames and every PCIe counter reads clean. Explain how both facts can be true.

The two contracts meet inside the device (§13 case 3). The NIC's DMA engine is on the PCIe side, where the lossless contract applies: if the host path is slow, the engine is backpressured, and nothing is dropped or errored on PCIe. But the NIC cannot backpressure the wire — the network keeps delivering frames — so the packet buffer fills and the excess is discarded on the Ethernet side. 26.4 §5 measured 23,495 drops with the MAC and offered load unchanged. The bottleneck is on the lossless side and the symptom appears on the lossy side, which is why the PCIe counters are clean and correct.

Q4. A colleague proposes that PCIe would be simpler if it could drop packets under congestion. What would it gain and what would it lose?

It would gain the elimination of credit deadlock — a fabric that can always drop can never have a wait-for cycle, which removes the failure class 25.8 exists to diagnose, along with credit accounting, pool sizing and the bandwidth-delay product question (28.1 §12). It would lose the guarantee that a Memory Write, once accepted, is delivered — which means every write would need an acknowledgement and a retransmission path, and Posted writes could no longer be Posted. The request/response machinery would grow rather than shrink, because a dropped Completion would now be possible and the completion timeout would stop being an exceptional case.

16. What Comes Next

This chapter isolated the delivery contract and found that the choice is not about quality but about placement. Overload becomes latency or it becomes loss; there is no third option, and the layers above are shaped entirely by which one the fabric picked.

28.3 changes a different assumption. Both PCIe and Ethernet let a device transmit when it has something to send. USB does not — the host schedules every transfer, and a device with data waits to be asked. That single rule reshapes the endpoint, the latency distribution and what the device hardware must contain.