Endpoints in Topology — Why Placement Changes Behaviour

Chapter 2.3 defined what an Endpoint is: a device-side participant that terminates a hierarchy path, defined by position rather than packaging or workload. Chapter 4.1 established that the host side is a set of anchors and that placement is a design decision.

This chapter joins those two:

How does an Endpoint attach to a real PCIe topology, and what does its location imply for behaviour, performance, verification, and debugging?

The practical form of that question is one engineers hit constantly: why does this device behave differently in this slot than it did in that one, when nothing about the device changed?

1. What Attachment Actually Decides

An Endpoint's own Link is genuinely dedicated (Chapter 2.8): exactly two components, nothing else attached, capability provisioned for that connection. Nothing in this chapter contradicts that.

What attachment decides is everything beyond that Link:

• How many hops separate it from the Root Complex.
• Which segments its traffic traverses on the way.
• Who else uses those segments.
• Which failures can take it out, and which take out its neighbours too.

None of that is visible from the device, its datasheet, or its own Link status. All of it is visible from the topology.

2. Three Attachment Cases

Direct-attached (A). One Link between the device and a Root Port. Its traffic shares no segment with any other device until it reaches the Root Complex. There is no intermediate forwarding component in the path.

Behind one Switch (B, C). Two Links: the device's own, then the Switch's upstream. B and C each have a dedicated Link, and they share the upstream segment — which means their traffic meets before reaching the root.

Behind cascaded Switches (D). Three Links, two forwarding components, two convergence points. Everything true of B and C is true of D and then some. Chapter 4.4 develops deep hierarchies properly; the point here is only that depth is a variable and it does not stop at one.

3. Local Link Capability Is Not End-to-End Bandwidth

This is the single most consequential consequence of placement, and it deserves stating as a rule.

An Endpoint's Link capability describes one hop. What the device achieves depends on every segment along its path, and those segments are shared with everything behind them.

Consider Endpoints B and C, both moving data toward host memory:

• Each has a dedicated Link to Switch 1. Neither contends with the other there.
• Both traverse Switch 1's upstream Link. Their demands sum on that segment.
• Both then converge at the Root Complex and into system memory, along with everything else in the system.

So B's achievable throughput is a function of C's activity, and neither device is at fault when it falls short. This is Chapter 2.7's convergence reasoning applied at the point where an engineer actually meets it: a device that benchmarked well in isolation underperforming in a populated system.

A dedicated Link guarantees something about one hop. It guarantees nothing about the path.

4. Isolation: What Placement Protects and What It Does Not

Placement determines failure domains as much as performance, and the distinction is sharp enough to be diagnostic.

A fault on an Endpoint's own Link affects that Endpoint only. Its siblings' Links are separate connections. B's Link failing does not disturb C.

A fault on a Switch's upstream Link affects everything below that Switch. B, C, D, and Switch 2 all lose their path to the root simultaneously, because they all traverse that one segment.

A fault in the Switch itself affects its whole subtree, for the same reason.

That produces a genuinely useful inference, and it works in reverse:

The set of affected devices identifies the topology element they share. One device affected implicates something local to it. A group affected implicates the smallest topology element containing all of them.

Chapter 4.4 turns that into a full debugging method for deep fabrics. For a single-Switch topology it is already immediately usable.

5. Siblings: Separate Links, Shared Fate

The nuance most often missed is that sibling Endpoints are partly independent, and knowing which part is which is the whole skill.

Independent: their own Links. Signalling conditions, Link capability, local Link state, and Link-local delivery are per-connection (Chapter 2.8). A retry on B's Link is invisible to C.

Shared: the upstream segment, the Switch's internal resources, the Root Complex, and system memory. Congestion on any of these affects both.

The practical consequence for verification is significant: testing an Endpoint with its siblings idle does not exercise the conditions it will actually operate under. A design can pass every isolated test and misbehave when a neighbour becomes active — not because either device is wrong, but because the shared segment behaves differently under combined load.

6. The Endpoint's Integration Boundary

Topology meets implementation at the point where a device's own logic connects to its PCIe interface. An Endpoint controller integration typically involves:

• A Link-facing interface toward the physical connection.
• The layer stack of Module 3 producing and consuming transactions.
• A configuration and resource interface through which software discovers and configures the function.
• The device's functional logic — whatever it actually does.
• Request and response tracking for operations awaiting completion.
• An event or interrupt source for signalling the host.
• A data-mover interface where the device originates bulk traffic.

The topology-relevant part is narrower than that list suggests, and it is where this chapter's RTL sits: the device's local logic must not be allowed to issue work the path cannot currently carry. That admission decision is where placement becomes hardware.

7. Endpoint Admission Control in RTL

// Illustrative internal representation — NOT PCIe-defined registers.
// Status an Endpoint integration might expose to its local logic.
typedef struct packed {
  logic       path_ready;        // abstract: connection usable
  logic       outstanding_full;  // no room to track another operation
  logic       error_latched;     // a fault requiring intervention
  logic [7:0] outstanding_count; // operations awaiting completion
} endpoint_status_t;

// Illustrative synthesizable RTL — Endpoint admission control.
// NOT a complete PCIe endpoint controller. Models the decision to accept or
// refuse local work based on path availability, tracking capacity, and error
// state, with a completion path that releases tracking entries.
module endpoint_admission #(
  parameter int unsigned MAX_OUTSTANDING = 32,
  localparam int unsigned CNT_W = $clog2(MAX_OUTSTANDING + 1)
) (
  input  logic clk,
  input  logic rst_n,
 
  // Abstract path availability from the layer stack below
  input  logic link_ready,
 
  // Fault indication requiring software intervention before new work
  input  logic fault_set,
  input  logic fault_clear,     // software-driven acknowledgement
 
  // From the device's own logic
  input  logic req_valid,
  input  logic req_needs_response,  // reads occupy a tracking entry; writes may not
  output logic req_ready,
 
  // Toward the layer stack
  output logic tx_valid,
  input  logic tx_ready,
 
  // Completion of a previously issued operation releases its entry
  input  logic cpl_valid,
 
  // Status
  output endpoint_status_t status
);
 
  logic [CNT_W-1:0] outstanding_q;
  logic             fault_q;
 
  // One registered holding slot. Its presence is what lets req_ready depend
  // on local state rather than on tx_ready — see the note below.
  logic             hold_valid_q;
  logic             hold_needs_rsp_q;
 
  wire full = (outstanding_q == MAX_OUTSTANDING[CNT_W-1:0]);
 
  // Admission depends only on LOCAL state: path usable, no latched fault,
  // tracking room for response-requiring work, and a free holding slot.
  // It deliberately does NOT depend on tx_ready.
  assign req_ready = link_ready
                   && !fault_q
                   && (!req_needs_response || !full)
                   && (!hold_valid_q || tx_ready);
 
  wire accept = req_valid && req_ready;
 
  // valid is driven from the registered slot, so it is asserted because we
  // hold work — never because the downstream happens to be ready.
  assign tx_valid = hold_valid_q;
 
  assign status.path_ready        = link_ready;
  assign status.outstanding_full  = full;
  assign status.error_latched     = fault_q;
  assign status.outstanding_count = outstanding_q[7:0];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      outstanding_q    <= '0;
      fault_q          <= 1'b0;
      hold_valid_q     <= 1'b0;
      hold_needs_rsp_q <= 1'b0;
    end else begin
      // Fault is sticky: set by hardware, cleared only by software. A
      // self-clearing fault would let work resume before the cause is known.
      if (fault_set)        fault_q <= 1'b1;
      else if (fault_clear) fault_q <= 1'b0;
 
      // Holding slot: fills on accept, empties on downstream acceptance.
      if (accept) begin
        hold_valid_q     <= 1'b1;
        hold_needs_rsp_q <= req_needs_response;
      end else if (tx_valid && tx_ready) begin
        hold_valid_q     <= 1'b0;
      end
 
      // A tracking entry is consumed when the work is actually issued
      // downstream, not merely when it was accepted locally.
      case ({(tx_valid && tx_ready && hold_needs_rsp_q), cpl_valid})
        2'b10:   outstanding_q <= outstanding_q + 1'b1;
        2'b01:   outstanding_q <= (outstanding_q == '0) ? '0 : outstanding_q - 1'b1;
        default: outstanding_q <= outstanding_q;
      endcase
    end
  end
endmodule

Classification: synthesizable.

What it models: the admission decision at an Endpoint's integration boundary — refusing local work when the path is unusable, when tracking capacity is exhausted, or when a fault is latched.

Deliberately simplified: one undifferentiated request stream (real designs separate operation classes with independent resources and ordering rules); completion release modelled as a bare pulse without matching a specific entry; no payload path; link_ready treated as a clean synchronous input.

What to notice:

• Tracking capacity gates only operations that need a response. A write that expects nothing back does not consume an entry, so gating it on tracking capacity would throttle throughput for no reason. Getting this wrong is a real and easily-missed performance bug.
• The fault is sticky. Hardware sets it; software clears it. A self-clearing fault lets work resume before anyone has established why it occurred.
• The decrement is guarded against underflow. A spurious or duplicated completion cannot wrap the counter to its maximum — which would silently appear as full capacity and stall the device permanently.

Production RTL would additionally require: per-class tracking with independent limits, completion matching against specific outstanding entries, ordering rule enforcement, timeout handling for operations that never complete, and the normative status reporting Module 8 covers.

8. Assertions at the Admission Boundary

// SVA over the illustrative admission module. Implementation invariants for
// THIS design — not PCIe protocol requirements.
 
// P1 — no work is accepted while the path is unusable. Accepting work that
// cannot be issued strands it: the device's logic believes it was handed off.
property p_no_accept_when_path_down;
  @(posedge clk) disable iff (!rst_n)
  !link_ready |-> !(req_valid && req_ready);
endproperty
a_no_accept_path_down : assert property (p_no_accept_when_path_down);
 
// P2 — no response-requiring work is accepted with tracking exhausted.
// Overrunning the tracker means a returning response has no entry to match,
// which surfaces far away as a completion that cannot be delivered.
property p_no_accept_when_full;
  @(posedge clk) disable iff (!rst_n)
  (status.outstanding_full && req_needs_response) |-> !(req_valid && req_ready);
endproperty
a_no_accept_full : assert property (p_no_accept_when_full);
 
// P3 — occupancy never exceeds the configured limit.
property p_occupancy_bounded;
  @(posedge clk) disable iff (!rst_n)
  status.outstanding_count <= MAX_OUTSTANDING;
endproperty
a_occupancy_bounded : assert property (p_occupancy_bounded);
 
// P4 — a latched fault blocks new work until explicitly cleared.
property p_fault_blocks;
  @(posedge clk) disable iff (!rst_n)
  status.error_latched |-> !(req_valid && req_ready);
endproperty
a_fault_blocks : assert property (p_fault_blocks);
 
// P5 — accepted work is presented downstream on the following cycle. The
// holding slot means issue is registered, not coincident with acceptance.
property p_accept_then_issue;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && req_ready) |=> tx_valid;
endproperty
a_accept_then_issue : assert property (p_accept_then_issue);
 
// P6 — the held item is stable while the downstream has not accepted it.
// This is the source-side half of the decoupled handshake contract.
property p_held_stable;
  @(posedge clk) disable iff (!rst_n)
  (tx_valid && !tx_ready) |=> tx_valid;
endproperty
a_held_stable : assert property (p_held_stable);

P1 catches stranding. Simulation with a permanently-available path never exercises it — the bug requires availability to be withdrawn while the device has work, which a directed test rarely produces.

P2 and P3 catch tracker overrun. This is a classic case simulation misses because a testbench issuing a few operations at a time never approaches the limit; it needs deliberate saturation.

P4 catches faults that do not actually block, which produces the confusing situation of a device reporting an error while continuing to issue work.

P5 is deliberately shown with its own limitation stated. It holds only because this model issues on acceptance; a buffered design needs a liveness property with an explicit environment assumption, and asserting the strong form there would fail on ordinary backpressure.

9. Topology-Aware Verification

The specific contribution of this chapter to a verification plan is that the environment must model the path, not just the Link.

Baseline — direct attach. Functional correctness with no intermediate component and no sibling traffic. Establishes that the device works.

Behind a Switch, siblings idle. Adds a forwarding hop. Should behave essentially as baseline; a difference here indicates sensitivity to hop count or latency, which is worth knowing.

Behind a Switch, siblings active. The case that matters and the one most often skipped. Generate sustained sibling traffic and confirm the device under test remains correct — throughput will fall, and that is expected. What must not happen is a functional failure: dropped operations, tracker corruption, timeouts mishandled, or forward progress lost.

Shared upstream saturation. Drive the upstream segment to saturation and hold it. This produces sustained backpressure at the device, exercising the admission logic under conditions a lightly-loaded test never reaches.

Failure isolation. Two distinct scenarios that must be distinguishable in the environment:

• Take down the device's own Link. Expect the device to become unavailable and its siblings unaffected.
• Take down the shared upstream Link. Expect all devices below the Switch to be affected together.

If a testbench cannot tell these apart, it will misattribute real failures.

Relocation. Where the environment supports it, run the same device RTL at different depths and compare. Differences are topology effects, and quantifying them validates that your model of the fabric is right.

Coverage worth defining: tracking occupancy across its range including full; requests offered while the path is unavailable; fault set and cleared with work in flight; sibling traffic at several intensities; and each failure-isolation scenario.

10. Debugging by Placement

Six signatures, each with the topology evidence that discriminates.

Works direct-attached, fails behind a Switch. The device is sensitive to something the Switch introduces — additional latency, backpressure patterns, or the interleaving of sibling traffic. Check whether it fails with siblings idle: if it does, the sensitivity is to hop count or latency; if only with siblings active, it is to congestion behaviour.

The device's Link is healthy but throughput collapses. Look upstream. A healthy Link says the first hop is fine and says nothing about the rest of the path. Check the shared upstream segment and what else is behind it.

All siblings degrade together. Implicates the smallest element they share — the Switch's upstream Link or the Switch itself. It is not a device problem, and investigating any individual device wastes time.

One device disappears, siblings unaffected. Implicates something local to that device: its own Link, its own connection, or the device itself. The siblings' health is the evidence that rules out everything shared.

The device's own request source stalls. Work is not being accepted locally. Check admission conditions in order: is the path available, is tracking exhausted, is a fault latched? Each has a different cause, and the status fields distinguish them directly — which is why exposing them is worth the logic.

Transactions leave the device but never reach the root. The device did its job. Investigate the path: the Switch's forwarding, the upstream segment, or a hop beyond. Chapter 3.4's boundary method applies, bisecting at the Switch.

11. Common Misconceptions

12. Understanding Check

13. Summary

An Endpoint is a leaf, and its path shapes what you observe. Attachment decides hop count, which segments the traffic traverses, who shares them, and which failures affect it alone versus a group.

Local Link capability is not end-to-end bandwidth. A dedicated Link guarantees one hop. Siblings behind one Switch have independent Links and a shared upstream segment, so their achievable throughput is coupled even though their connections are not.

Placement determines failure domains, and the relationship inverts usefully: the set of affected devices identifies the topology element they share. One device affected implicates something local; a group implicates the smallest element containing all of them.

For implementation, placement becomes hardware at the admission boundary — local work must be gated on path availability, on finite tracking capacity for operations that expect a response, and on latched faults. For verification, the environment must model the path: testing with siblings idle does not exercise the conditions the device will operate under, and the two failure-isolation cases must be distinguishable.

Avoid the generalisation that direct attachment is fastest. The defensible claim is that direct attachment removes intermediate hops and intermediate sharing; whether that changes observed performance depends on load, capability, and workload.

Hold the model: an Endpoint's observable behaviour is shaped by the path connecting it to the Root Complex.

14. What Comes Next

This chapter treated the Switch as a fixed feature of an Endpoint's path. Chapter 4.3 — Switches opens it: how ports are organised and oriented, where forwarding decisions happen, where traffic actually converges inside the component, and why adding ports does not add upstream capacity.

Chapter 4.4 then cascades them, and 4.5 applies everything to real systems.

Revisit Root Complex Topology for the anchors these paths begin at, or Endpoint for the architectural role this chapter placed. Browse the full path on the PCIe tutorials index.