PCIe vs AXI — What Distance Does to a Bus

Module 27 asked how to reason inside PCIe. This module asks what PCIe is for, by comparing it with the interconnects it is not.

AXI is the right place to start, because AXI and PCIe solve the same problem — move read and write transactions between a requester and a completer — and disagree about almost every mechanism. Every disagreement is a consequence of one physical fact: AXI runs across a chip, PCIe runs across a connector.

1. Sources, Scope, and What Is Being Compared

2. The Same Job, Two Contracts

Both protocols move reads and writes from a requester to a completer, and both must answer the same five questions.

The last row is the one that generates most of the others. An AXI transfer that is accepted is accepted; there is no notion of it arriving damaged, because a handshake on a synchronous bus either happened or did not. A PCIe TLP crosses a serial link where corruption is expected, so there is a whole layer — sequence numbers, CRC, ACK/NAK, a replay buffer (14.4) — whose only job is to make the link behave like the wires AXI already has.

That is a useful way to read the PCIe stack: much of the Data Link Layer exists to reconstruct guarantees that an on-chip bus gets for free.

3. What Distance Breaks

Four mechanisms work at zero distance and stop working as it grows.

Observed backpressure. READY is a live signal: the sender sees the receiver's state this cycle. Insert delay and the sender is acting on the past. §12 measured the consequence — 45,907 overruns at a 64-cycle round trip.

Combinational dependency. AXI forbids VALID from waiting on READY precisely so that a handshake can close in one cycle (handshake-deadlock-rules). Across a link there is no cycle in which both ends are visible to each other at all.

Implicit reliability. Wires do not drop transfers. Serial links do, so PCIe adds sequence numbers and replay.

Cheap wide parallelism. AXI can have separate address, data and response channels because wires are cheap on a die. PCIe serialises everything onto the same lanes, which is why a TLP has a header describing what it is — the channel no longer identifies it.

Each PCIe mechanism in the following sections is a replacement for one of these four.

4. Backpressure: Observation vs Permission

Three consequences follow, and all three are measurable.

Credits never overrun, at any distance (§12). The permission was granted from real buffer space; spending it cannot exceed that space regardless of how long the grant took to arrive.

Credits must be sized for the round trip. §12 measured a pool of 2 delivering 12% of the zero-distance rate at a 32-cycle round trip, and a pool of 32 delivering 99%. The pool must cover the bandwidth-delay product, or the sender runs out of permission while the receiver sits empty. 22.3 owns this as a throughput question.

And credits introduce a failure mode AXI does not have. A sender can hold permission it cannot use, or wait for permission that will never come, and the result is a link that is idle with no error anywhere — 25.8 measured a healthy link entering 818 zero-credit episodes and resolving every one, which is exactly why a zero-credit reading proves nothing.

AXI's equivalent failure is different in kind. An AXI deadlock comes from a dependency cycle between channels — a master that will not accept a response until it can issue another request, facing a slave that will not accept a request until its response is taken. That is why handshake-deadlock-rules exists, and the rule it enforces (VALID must not depend on READY) has no PCIe counterpart because PCIe has no combinational handshake to break.

5. Ordering: Identity vs Class

Both protocols allow transactions to complete out of order, and they express the guarantee differently.

AXI orders by ID. Transactions carrying the same AxID must return in order; transactions with different IDs may return in any order (same-id-ordering, different-id-ordering). The requester chooses its ordering by choosing its IDs, which is a remarkably direct control.

PCIe orders by transaction class. The rules are about relationships between Posted, Non-Posted and Completion traffic (13.4), not about a requester-chosen tag. A Tag identifies a request so its Completion can be matched; it does not express an ordering intent.

The practical difference for a designer:

This is the single most common source of confusion for engineers moving between the two, and it produces a specific bug: assuming that reusing a PCIe Tag creates ordering. It does not. A Tag makes a Completion matchable; ordering between two outstanding non-posted requests is not implied by their Tags at all.

And the producer-consumer pattern is where PCIe's class-based rules earn their keep. A device writes data and then writes a flag; the rules are what make the flag's arrival meaningful to a reader of the data. AXI expresses the same intent with IDs and response ordering. Same goal, different vocabulary, and translating between them mechanically is where mistakes happen.

6. Identity and Splitting

AXI bursts are described by the request. AxLEN, AxSIZE and AxBURST say how many transfers of what size in what pattern, and the data channel delivers exactly that (burst-fundamentals). The requester knows the shape of the answer before it arrives.

PCIe reads may be answered by several Completions, and the requester does not choose how many (13.3). The completer splits according to its own policy and the Read Completion Boundary; the requester tracks bytes and retires on the byte count.

That single difference produces a whole class of PCIe logic that has no AXI analogue:

• a byte counter per outstanding request rather than a beat counter;
• retirement on byte count satisfied rather than on a known final beat;
• the Byte Count field as a running check against the requester's own total;
• and a partial-timeout state — some data arrived and the rest did not — that a burst-shaped protocol simply cannot be in.

The 4 KiB rule is a useful contrast. AXI forbids a burst from crossing a 4 KiB boundary (4kb-boundary-rule) so that a burst never spans two pages or two slaves. PCIe handles the same concern by letting the completer split, which moves the complexity from the requester's address arithmetic into the requester's completion tracking.

7. What Each Protocol Does Not Have

Naming the absences is often more clarifying than naming the features.

PCIe has no READY. There is no signal by which a receiver tells a sender to stop right now. Everything is credits, granted in advance.

PCIe has no burst descriptor. A read requests a length; the answer's shape is the completer's choice.

AXI has no packets. There is no header, because the channel a signal arrives on says what it is.

AXI has no replay. There is no sequence number and no retransmission, because there is nothing to retransmit across.

AXI has no enumeration. Address decode is fixed at design time by the interconnect; there is no discovery phase, no configuration space, no BARs.

AXI has no link training. There is no LTSSM because there is no link to bring up — the wires are connected or the chip does not work.

And PCIe has no combinational path between the two ends at all, which is the fact that makes all the other differences necessary.

8. The Comparison, Drawn

The asymmetry in the figure is the point. The AXI path is two channels and four signals. The PCIe path is three layers, a packet format, a credit loop and an identity-matching structure — all of it doing the job those four signals do on-chip.

9. RTL — The Two Backpressure Mechanisms, Side by Side

Block 1 — a ready/valid sink. The correctness argument is local and immediate.

// AXI-style: the receiver's state is visible to the sender THIS CYCLE.
// No accounting, because nothing is predicted.
module rv_sink #(
  parameter int unsigned DEPTH = 16
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                              in_valid,
  output logic                              in_ready,
  input  logic [63:0]                       in_data,
  input  logic                              drain,
  output logic [$clog2(DEPTH+1)-1:0]        level,
  output logic                              overrun          // must never assert
);
 
  logic [63:0] mem [DEPTH];
  logic [$clog2(DEPTH)-1:0] wr, rd;
 
  // READY is a function of CURRENT occupancy. The sender sees it this cycle,
  // so it cannot send into a full buffer — provided there is no delay
  // between the two, which is exactly the assumption §12 removes.
  assign in_ready = (level != DEPTH);
  assign overrun  = in_valid && !in_ready && accepted_anyway;  // instrumentation
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= '0; rd <= '0; level <= '0;
    end else begin
      if (in_valid && in_ready) begin
        mem[wr] <= in_data;
        wr <= (wr == DEPTH-1) ? '0 : wr + 1'b1;
      end
      if (drain && (level != 0))
        rd <= (rd == DEPTH-1) ? '0 : rd + 1'b1;
 
      // The same four-way case as 27.4 §8: push and pop are NOT exclusive.
      case ({in_valid && in_ready, drain && (level != 0)})
        2'b10: level <= level + 1'b1;
        2'b01: level <= level - 1'b1;
        default: ;                              // both or neither: unchanged
      endcase
    end
  end
 
endmodule

Block 2 — a credit-based sender. The correctness argument is an accounting invariant.

// PCIe-style: the receiver is never observed. The sender holds permission
// and spends it. Correctness is an INVARIANT on the counters, not a
// property of a wire.
module credit_sender #(
  parameter int unsigned CNT_W = 12
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                  init_valid,        // flow-control initialisation
  input  logic [CNT_W-1:0]      init_credits,
  input  logic                  want_send,
  output logic                  do_send,
  input  logic                  cred_return_valid, // an FC DLLP arrived
  input  logic [CNT_W-1:0]      cred_return_amt,
  output logic [CNT_W-1:0]      limit,
  output logic [CNT_W-1:0]      consumed,
  output logic [CNT_W-1:0]      available,
  output logic                  underflow_attempt
);
 
  // Consumed and limit are tracked SEPARATELY, and `available` is derived.
  // A single up/down counter cannot distinguish "nothing was returned" from
  // "too much was consumed", and both are real bugs (25.8 §8).
  always_comb begin
    available         = limit - consumed;
    // The guard is a COMPARISON BEFORE the subtraction. These counters are
    // unsigned: a decrement below zero wraps to a huge value and the sender
    // concludes it has abundant permission. There is no sign bit to test.
    underflow_attempt = want_send && (available == '0);
    do_send           = want_send && !underflow_attempt;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      limit <= '0; consumed <= '0;
    end else begin
      if (init_valid) limit <= init_credits;
      // Credit return and credit consumption in the SAME cycle: both must
      // land. Making them exclusive loses one, and the direction of the
      // drift depends on which branch wins.
      if (cred_return_valid) limit <= limit + cred_return_amt;
      if (do_send)           consumed <= consumed + 1'b1;
    end
  end
 
endmodule

The two correctness arguments are worth stating side by side, because this is the transferable idea:

Ready/valid is correct if the wire is right. Credits are correct if the arithmetic is right.

One is verified by checking a handshake; the other by asserting a conservation invariant. Neither is harder — they fail differently, and §11 is that comparison.

10. Same-Cycle Audit

11. Invariants — and How They Differ Between the Two

The invariant worth asserting is different in each protocol, and that difference is the chapter in one page.

// AXI-side: the argument is about the HANDSHAKE.
 
// A1 — VALID must not wait for READY. This is what makes a one-cycle
// handshake possible and is the rule that prevents combinational deadlock.
property p_valid_not_gated_by_ready;
  @(posedge clk) disable iff (!rst_n)
    (in_valid && !in_ready) |=> in_valid;
endproperty
 
// A2 — payload is stable while a transfer is offered and not accepted.
property p_payload_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (in_valid && !in_ready) |=> $stable(in_data);
endproperty
 
// A3 — the receiver never accepts into a full buffer. On-chip this is
// guaranteed by READY being combinational from level; §12 measured what
// happens when the sender's view of it is delayed.
property p_no_overrun;
  @(posedge clk) disable iff (!rst_n)
    (in_valid && in_ready) |-> (level < DEPTH);
endproperty

// PCIe-side: the argument is about the ACCOUNTING.
 
// P1 — consumed never exceeds the granted limit. This is the invariant that
// replaces the READY wire, and violating it means the receiver is overrun
// with no signal anywhere to say so.
property p_consumed_bounded;
  @(posedge clk) disable iff (!rst_n) (consumed <= limit);
endproperty
 
// P2 — available is DERIVED, never cached. A stale cached value permits a
// send the accounting does not.
property p_available_derived;
  @(posedge clk) disable iff (!rst_n) (available == (limit - consumed));
endproperty
 
// P3 — the comparison precedes the subtraction. These counters are unsigned,
// so a decrement below zero WRAPS UPWARD and the sender floods a receiver
// with no space. There is no sign bit to test afterwards.
property p_no_send_without_credit;
  @(posedge clk) disable iff (!rst_n) do_send |-> (available != '0);
endproperty
 
// P4 — a credit return raises the limit rather than lowering consumed.
// Both arithmetic forms "work" until an underflow, and only one is checkable.
property p_return_raises_limit;
  @(posedge clk) disable iff (!rst_n)
    (cred_return_valid && (cred_return_amt != 0)) |=> (limit > $past(limit));
endproperty

The comparison to carry away:

PCIe's failure mode is quieter, and that asymmetry is why 25.8 exists as a whole chapter and AXI needs no equivalent.

12. Measured — What Distance Does

Three readings.

At zero distance the two schemes are identical — 99,924 transfers, no overruns, either way. On-chip, credits buy nothing, which is why AXI does not have them.

Ready/valid appears to send more as distance grows. It does: it sends into a receiver that has no room. The extra transfers are the overruns. A throughput measurement alone would rank the broken configuration higher, which is a trap this module has now hit in four separate chapters.

And credits pay a different price. At a 64-cycle round trip the 16-credit pool delivered 47,922 — under half — because the sender ran out of permission waiting for returns. That is not a failure; it is the sizing rule:

The pool must cover the round trip's worth of traffic. Below that it throttles a receiver with room to spare; above it, nothing improves. This is a design decision AXI never has to make, and it is the clearest single example of a cost PCIe pays for working across a connector.

13. Performance Reasoning

The relationship both protocols obey:

throughput ≤ outstanding_bytes / round_trip_time

Define the terms. outstanding_bytes is the total bytes committed and not yet answered; round_trip_time is the interval from issue to answer. Assumption: the requester always has work, and no stage other than this one is the limit.

Worked example, ILLUSTRATIVE. A requester with 32 outstanding requests of 512 bytes each and a 300-cycle round trip:

outstanding_bytes = 32 × 512 = 16,384
throughput ≤ 16,384 / 300 = 54.6 bytes/cycle

§12's credit table is the same equation seen from the other side. A pool of N credits at a round trip of R cycles permits at most N/R transfers per cycle sustained — so a pool of 8 at a 32-cycle round trip permits 0.25, which is the 46% figure measured.

What the equation cannot predict. It says nothing about whether the requester is the limit. 27.4 §14 measured six causes of low throughput and only two of them are in this formula. An engineer who reaches for this equation first will be right about a third of the time, and the occupancy reading is what tells you whether you are in that third.

The AXI counterpart is the same arithmetic with different names — outstanding transactions per ID, burst length, and the interconnect's latency (outstanding-depth-tuning, latency-analysis). The formula transfers; the mechanisms that limit each term do not.

14. Debugging — What the Symptom Tells You

15. Misconceptions

"AXI is just PCIe on-chip." Why it sounds reasonable: both carry reads and writes between a requester and a completer, with out-of-order completion and identifiers. What actually happens: the backpressure mechanism, the ordering model, the identity semantics and the reliability layer are all different (§2). AXI observes; PCIe is granted permission. What it causes: engineers assume a READY-equivalent exists and look for the signal that is stopping the link. There is not one (§14 case 2).

"Credits are just a slower READY." Why it sounds reasonable: both stop a sender from overrunning a receiver. What actually happens: READY reports current state; a credit is permission granted in advance from real buffer space. §12 measured READY overrunning 45,907 times across distance and credits overrunning zero times. What it causes: a credit pool sized as though it were a status flag — and §12 measured a pool of 2 delivering 12% of the achievable rate.

"A PCIe Tag orders transactions the way an AXI ID does." Why it sounds reasonable: both are small identifiers attached to outstanding transactions, and AXI's genuinely does order. What actually happens: a Tag is a matching key only (§5). PCIe ordering is expressed between transaction classes (13.4). What it causes: code that assumes two reads with the same Tag return in order — which cannot even be issued, since a Tag must be unique while outstanding.

"A read returns one response." Why it sounds reasonable: on AXI the burst shape is in the request, so the answer's shape is known. What actually happens: PCIe completers split according to their own policy (13.3). What it causes: a requester that retires on the first Completion, freeing its Tag with data still in flight — and the remaining Completions arrive as orphans.

"More transfers means better throughput." Why it sounds reasonable: transfers are the thing being counted. What actually happens: §12's delayed-READY rows sent 45% more transfers than the correct configuration by sending into a full receiver. What it causes: a broken configuration ranked as the fastest one.

"PCIe's Data Link Layer is overhead." Why it sounds reasonable: it adds sequence numbers, CRC and a replay buffer to every packet, none of which AXI needs. What actually happens: it reconstructs the reliability that on-chip wires provide for free (§2). AXI does not lack a replay layer because it is better designed; it lacks one because it has nothing to replay across. What it causes: attempts to "optimise away" a mechanism whose absence would make the link unusable.

"Zero credits means the link is stuck." Why it sounds reasonable: zero permission means nothing can be sent, and that is what a stuck link looks like. What actually happens: 25.8 §14 measured a healthy link entering 818 zero-credit episodes and leaving every one of them. What it causes: investigations that begin at the credit pool and never reach the wait-for graph, which is the only structural test.

16. Understanding Check

Q1. A design converts an AXI master to PCIe by mapping READY to a credit-available signal. What breaks, and at what point?

Nothing breaks at zero distance and everything breaks as the round trip grows (§4, §12). READY reports the receiver's state now; the mapped signal reports permission the sender holds. Those coincide only when the two ends are in the same clock domain with no delay — §12 measured them identical at 0 cycles (99,924 transfers, zero overruns each). At a 64-cycle round trip the observed-state scheme overran the receiver 45,907 times while the credit scheme overran zero. The deeper problem is that the mapping loses the sizing question entirely: a credit pool must cover the bandwidth-delay product, and §12 measured a pool of 2 at a 32-cycle round trip delivering 12% of the achievable rate.

Q2. Two PCIe reads are outstanding with different Tags. What ordering is guaranteed between their Completions, and how would you get ordering if you needed it?

None between them (§5). Tags are matching keys, not ordering domains — the AXI intuition that a shared identifier creates ordering does not transfer, and two outstanding requests cannot share a Tag anyway. To order two reads you must wait for the first to complete before issuing the second, which converts a pipelined pair into a serialised pair and costs a full round trip. That cost is precisely why PCIe expresses ordering between classes (13.4) rather than between individual requests: the common case that actually needs ordering is producer-consumer — data then flag — and the class rules cover it without serialising every read.

Q3. Your credit-based sender's throughput is 46% of what the link should carry, the receiver's buffer is never full, and no error is set. Diagnose.

The credit pool is undersized for the round trip (§12). The sender is running out of permission while the receiver has room — exactly the signature §12 measured for a pool of 8 at a 32-cycle round trip, which delivered 46%. The confirming reads are credit availability at the sender hitting zero while receiver occupancy stays below capacity; that pair distinguishes it from a receiver that genuinely cannot drain. The fix is pool depth, not link speed — and note that a pool larger than the bandwidth-delay product buys nothing, since §12's rows for 32 and 64 credits are both 99%.

Q4. Why does AXI need a rule that VALID must not wait for READY, and why does PCIe need no equivalent?

Because AXI's handshake is combinational and PCIe's is not (§3, §11 A1). If both ends may gate their output on the other's input in the same cycle, a combinational loop forms and neither side can ever assert — a deadlock created by the interface itself rather than by any traffic pattern. AXI forbids it explicitly (handshake-deadlock-rules). PCIe has no combinational path between its two ends at all, so the failure is structurally impossible. PCIe's analogous hazard is a credit deadlock — a wait-for cycle between agents — which is a different mechanism with a different test (25.8 §4).

Q5. §12 shows the delayed-READY configuration sending 45% more transfers than the correct one. Explain why that is not an improvement, and name the general lesson.

The extra transfers are overruns — data written into a receiver that had no room, because the sender was acting on a READY that belonged to several cycles earlier (§12). The throughput counter rises precisely because correctness has been abandoned. The general lesson is that a throughput measurement cannot distinguish work done from work destroyed, and this module has now measured that same shape four times: a broken ring-full test posting 11% more work (25.6 §6), a completion queue reporting six times the throughput by clobbering entries (26.3 §17), and here. A performance gain with no corresponding design change deserves a correctness check, not a celebration.

17. What Comes Next

This chapter compared PCIe with the interconnect closest to it. The remaining three move further away, and each one isolates a different assumption.

AXI and PCIe share a load/store model and disagree about mechanism. 28.2 is the first chapter where the model itself differs: Ethernet does not address memory, does not return a Completion, and is permitted to discard. Every mechanism that survives from this chapter to that one is a mechanism that load/store fabrics genuinely require.