Skip to content

PCIe · Module 9

Host Access — From a System Address to a Resource Offset

After assignment the host never touches the BAR again. It issues ordinary memory-space accesses to the assigned address, the hierarchy routes them, and the Function converts a system address into a resource offset. Plus the definitive MMIO debug ladder.

Chapter 9.5 left the system in a state where nothing has happened yet: a resource described, sized, placed, programmed, and permitted.

What actually happens when the host performs an access to that mapped address?

1. The BAR Leaves the Story

Chapter 9.1 opened Module 9 by insisting that a BAR is not the device's memory. Six chapters later the same distinction has a sharper form.

During enumeration the BAR was the object being manipulated. Software read it, probed it, wrote it, read it back. Every access in Chapters 9.4 and 9.5 targeted the register itself.

During operation the BAR is not accessed at all. Not once per transaction, not once per burst, not at all. Its programmed value became a comparator operand (Chapter 9.5 §11's active_base) and the runtime path reads that operand — a wire, not a register access.

Enumeration accesses the BAR. Operation accesses the window the BAR describes. Different address spaces, different mechanisms, different consumers of the same 32 or 64 bits.

Why this matters beyond terminology. A driver author who believes MMIO involves the BAR will reach for configuration-space calls when they want data movement, and will expect BAR reads to reflect device activity. Both are §15's misconceptions, both produce confident bug reports, and both dissolve the moment the two paths are held apart.

2. Two Kinds of Access, Compared

DimensionConfiguration accessHost resource access
What it targetsthe Function's configuration spacethe resource behind an assigned window
How the target is namedidentity — Bus, Device, Function, plus a register offset (7.7)an address in memory or I/O space
Address spaceconfigurationmemory (or I/O, 9.3)
Who issues itenumeration software, drivers doing control-plane workany host agent doing data-plane work
Whenat discovery, and occasionally afterwardsconstantly
Permission gatenone of the Command Register's space enablesMemory Space Enable / I/O Space Enable (8.4)
Reachesthe BAR, the Command Register, the headerthe registers, memory, or queues behind the window
Routingby identity, through the hierarchyby address, through every parent routing window

The routing row is the one that explains the most confusing symptom in Module 9. Configuration access is routed by identity, so it keeps working perfectly when address routing is broken. A Function whose parent windows do not forward its assigned range will read back a flawless BAR, a flawless Command Register, and a flawless header — while every memory access to it vanishes.

That combination — configuration fine, MMIO dead — is not a contradiction. It is what a routing fault looks like, and §14's ladder is built around recognising it fast.

3. The End-to-End Path

Host access to an assigned window. The host CPU issues a load at the assigned address. The Root Complex routes it by address. The switch path forwards it through parent routing windows to the Function. The Function decode computes an offset equal to the address minus the base and presents it to the internal resource, which returns data. A response travels back through the switch path and Root Complex to the host, completing the load. The host then issues a store at the assigned address, which is routed and forwarded the same way, and the Function writes the payload at the computed offset.A host access reaching an assigned windowHost CPURoot ComplexSwitch pathFunction decodeInternalresourceload at assignedaddressrouted by addressforwarded if inwindowoffset = address -basedataresponse - Module 10load completesstore at assignedaddressrouted and forwardedwrite payload atoffset
Figure 1 — a host read and a host write reaching an assigned window. Both travel by address through the hierarchy rather than by identity. The Function's decoder converts the system address into an offset inside its own resource. The read's return path is shown as a single step because the transaction that carries it is Module 10's subject, not this chapter's.

Three things to read out of the figure.

The host's action is ordinary. A load and a store. Nothing in the CPU's instruction says "PCIe"; the address is what makes it PCIe, and the address is what the host allocator chose in Chapter 9.5.

Every hop between the host and the Function can drop it. "Forwarded if in window" is not decoration — each bridge applies its own routing window (§7), and an access outside any one of them stops there.

The read and the write are not symmetric. The read has a return path; the write does not appear to. That asymmetry is real, it is the reason the two have different failure modes (§14's second scenario), and it belongs to Module 10 — §8 says exactly how much of it this chapter claims.

4. From a Virtual Address to the Assigned Range

Software does not usually hold the assigned physical address directly, and the gap between what a driver writes and what appears on the Link is worth stating once — then scoping out.

The chain, at the level this chapter needs:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
driver's pointer  (a virtual address)
   → CPU address translation
   → a host physical address
   → the range assigned to this BAR in Chapter 9.5
   → routed by address through the hierarchy
   → the Function

The operating system's job is to make the second-to-last step true. A driver asks to map its device's window; the OS establishes a virtual mapping whose translation lands inside the assigned physical range, with the attributes MMIO requires. From then on, a store through that pointer becomes an access to the assigned address.

5. Address to Offset — the Arithmetic

This is the bridge from configuration to datapath, and it is small enough to derive completely.

Given a window whose base is B and whose size is S, and an inbound host address A:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
in-window   ⟺   B ≤ A < B + S
offset      =   A − B

The offset selects a location inside the resource. A is a system-wide coordinate; offset is a device-local one, and the Function only ever needs the second.

A worked access

The window. Base B = 0x8000_0000, size S = 64 KiB = 0x1_0000. Aligned, since 0x8000_0000 mod 0x1_0000 = 0 (Chapter 9.5 §3).

Host addressIn window?Offset
0x8000_0000yes — the first address0x0000
0x8000_0120yes0x0120
0x8000_FFFFyes — the last address0xFFFF
0x8001_0000no — this is B + S, one past the end
0x7FFF_FFFFno — one below the base

0x8001_0000 is the address people get wrong. It looks like it belongs — it differs from the base only in a bit above the window — and it is exactly one byte past the last address the window covers. It is the boundary a mask error moves, and §13 tests it on both sides.

6. Permission Is Checked on Every Access

Memory Space Enable is not a one-time gate that assignment passes through. It is an operand of the decode, consulted on every inbound access, exactly as Chapter 8.4 described.

The acceptance condition, conceptually:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
the window is programmed
  AND the space is enabled
  AND the address falls inside the window
     → the access belongs to this resource

All three are independent and all three are required. A programmed window with the enable cleared serves nothing. An enabled space with no programmed window has nothing to serve. A matching address with either missing is not a match.

And "conceptually" is doing real work in that box. The condition above says whether an access belongs to this resource. It does not say how the interface flow-controls that access, and conflating the two is a common RTL mistake — §11 keeps them apart deliberately, because they answer different questions and depend on different things.

7. Local Decode Is Not Reachability

The Function can answer one question: is this address mine? It cannot answer the question that actually determines whether the system works: did the access get here?

Every bridge between the Root Complex and the Function applies its own routing window. An access is forwarded downstream only if its address falls inside the range that bridge has been programmed to forward (Chapter 7.4, Chapter 7.8 §5, Chapter 9.5 §9).

PropertyOwnerEvidence available
The Function claims this addressthe Functionthe BAR; the decoder's hit output
The access reaches the Functionthe hierarchyparent windows, level by level

The failure mode is silence with a perfect local configuration. The BAR reads back correctly. The Command Register reads back correctly. hit never asserts, because nothing ever arrives. Nothing inside the Function distinguishes "no access arrived" from "an access arrived and missed" unless the interface is instrumented to show the difference — which is why §11 exposes a miss output and §14 asks for it early.

8. Reads and Writes Are Not the Same Shape

At the level this chapter owns, one asymmetry matters and the rest is deferred.

A write carries a payload to the Function. The host supplies an address and data; the Function applies it at the computed offset. Whatever the Function does about telling the host it happened is not visible in the write itself.

A read must produce data back. The host supplies an address and expects a value. Something has to travel in the reverse direction, carrying the data and identifying which request it answers.

9. Microarchitecture — The Resource Front End

The block sitting between the Function's PCIe side and its actual registers has a small, well-defined job, and separating its two halves is what makes it verifiable.

The decode half answers is this mine, and where inside me? It consumes the window state from Chapter 9.5, the permission bit from Chapter 8.4, and the inbound address. It produces a claim and an offset.

The ownership half answers who holds this request now? It takes an accepted request, registers it, and presents it downstream under a handshake, absorbing the difference between the rate requests arrive and the rate the resource consumes them.

10. Interface Discipline

The front end is the first place in Module 9 with a genuine decoupled interface on both sides, so the rules it follows are worth stating before the code.

  • Valid is asserted because data exists, never because the far side is ready. res_valid is a register set at acceptance, and it does not consult res_ready to decide whether to assert.
  • Ready depends on state, not on the offer. req_ready is a function of whether the ownership stage is occupied and whether the downstream will drain it. It never depends on req_valid, so there is no combinational path from a requester's valid to its own ready.
  • Payload is stable while stalled. Once res_valid is high, every field held with it is stable until the handshake. This is a property of the design, and P5 asserts it.
  • A transfer happens on valid && ready, once. No partial transfers, no re-offers of a request already taken.
  • The requester owes the same discipline. Its valid must hold and its payload must be stable until the handshake. That is an assumption on the environment, and §12 states it as one rather than pretending it is a property of this module.

11. RTL — The Resource Front End

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Turns an inbound memory-space request into an offset inside
// one BAR-backed resource, under a valid/ready handshake on both sides.
// Window semantics and the meaning of Memory Space Enable: NORMATIVE.
// Interface shape, data width, and the miss output: illustrative.
import bar_size_pkg::*;   // Chapter 9.4 section 10 — one mask definition
 
module bar_resource_frontend #(
  parameter longint unsigned RESOURCE_SIZE = 64'h0000_0000_0001_0000, // 64 KiB
  // Derived. Not intended to be overridden.
  parameter int              OFF_W         = $clog2(RESOURCE_SIZE)
) (
  input  logic        clk,
  input  logic        rst_n,
 
  // Window state, from Chapter 9.5. READ ONLY here — there is no write port
  // from this side, which is what makes "an access cannot move the window"
  // structural rather than a promise (Chapter 9.1).
  input  logic [63:0] active_base,
  input  logic        base_valid,
  input  logic        mem_space_enable,     // Command Register bit 1
 
  // Inbound memory-space request.
  input  logic        req_valid,
  output logic        req_ready,
  input  logic [63:0] req_addr,
  input  logic        req_write,
  input  logic [31:0] req_wdata,
  input  logic [3:0]  req_be,
 
  // Internal resource request. Same discipline, one stage of ownership.
  output logic             res_valid,
  input  logic             res_ready,
  output logic [OFF_W-1:0] res_offset,
  output logic             res_write,
  output logic [31:0]      res_wdata,
  output logic [3:0]       res_be,
 
  // A request this window does not claim. One-cycle pulse at the handshake.
  // What the Function DOES about it is out of scope (Chapter 9.1 section 6).
  output logic        miss,
 
  // Snapshots, for checking. Not needed by the datapath.
  output logic [63:0] base_at_accept,
  output logic [63:0] addr_at_accept
);
 
  localparam logic [63:0] SIZE = 64'(RESOURCE_SIZE);
  localparam logic [63:0] LOW  = SIZE - 64'd1;
  localparam logic [63:0] MASK = ~LOW;
 
  generate
    if (!size_is_pow2(RESOURCE_SIZE))
      $error("RESOURCE_SIZE must be a nonzero power of two");
    if (!size_meets_pcie_endpoint_min(RESOURCE_SIZE))
      $error("A PCI Express Endpoint memory resource must be >= 128 bytes");
  endgenerate
 
  // ---- Decode half: is this mine, and where inside me? ----------------
  // Three independent conditions, exactly as section 6 states them.
  wire window_open = base_valid && mem_space_enable;
  wire addr_hit    = ((req_addr & MASK) == (active_base & MASK));
  wire mine        = window_open && addr_hit;   // payload-dependent, no valid
 
  // Because the base is aligned to the size (Chapter 9.5), A - B is exactly
  // the low bits of A. No subtractor, and P4 checks the equivalence against
  // the true subtraction so a misaligned base cannot hide here.
  wire [OFF_W-1:0] offset_comb = req_addr[OFF_W-1:0];
 
  // ---- Ownership half: one request in flight --------------------------
  logic             res_valid_q;
  logic [OFF_W-1:0] res_offset_q;
  logic             res_write_q;
  logic [31:0]      res_wdata_q;
  logic [3:0]       res_be_q;
  logic [63:0]      base_snap_q, addr_snap_q;
 
  // READY DEPENDS ON STATE AND DOWNSTREAM READY ONLY. Never on req_valid, so
  // there is no combinational path from the requester's valid to its ready;
  // and never on req_addr, so a request this window does not claim is still
  // consumed rather than deadlocking the interface (section 9).
  assign req_ready = !res_valid_q || res_ready;
 
  wire xfer   = req_valid && req_ready;   // a request leaves the requester
  wire accept = xfer &&  mine;            // ...and enters the resource path
  assign miss = xfer && !mine;            // ...or is not ours
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      res_valid_q  <= 1'b0;
      res_offset_q <= '0;
      res_write_q  <= 1'b0;
      res_wdata_q  <= 32'h0000_0000;
      res_be_q     <= 4'h0;
      base_snap_q  <= 64'h0;
      addr_snap_q  <= 64'h0;
    end else begin
      // Drain first, then refill — so a full stage with res_ready high can
      // hand off and take a new request in the same cycle.
      if (res_valid_q && res_ready) res_valid_q <= 1'b0;
 
      if (accept) begin
        res_valid_q  <= 1'b1;
        res_offset_q <= offset_comb;
        res_write_q  <= req_write;
        res_wdata_q  <= req_wdata;
        res_be_q     <= req_be;          // byte enables PROPAGATED, not dropped
        base_snap_q  <= active_base;     // snapshot: the base in force at accept
        addr_snap_q  <= req_addr;
      end
    end
  end
 
  assign res_valid      = res_valid_q;
  assign res_offset     = res_offset_q;
  assign res_write      = res_write_q;
  assign res_wdata      = res_wdata_q;
  assign res_be         = res_be_q;
  assign base_at_accept = base_snap_q;
  assign addr_at_accept = addr_snap_q;
 
endmodule

Classification: synthesizable.

Interface and register semantics:

DimensionBehaviour
Resetstage empty; res_valid low; snapshots cleared
req_readystate and res_ready only — never req_valid, never req_addr
res_validset at acceptance; independent of res_ready
Payload stabilityevery res_* field is stable while res_valid is high and res_ready is low
Throughputone request per cycle sustained, because the stage drains and refills in the same cycle
Byte enablespropagated — carried to the resource unchanged, not accepted and dropped
Window stateread only; there is no write port from this side
Base snapshottaken at acceptance, so a later base change cannot retroactively alter an in-flight offset
Missone-cycle pulse; never enters the resource path; response behaviour out of scope
Offset width$clog2(RESOURCE_SIZE) — exactly the bits the resource can use

What it teaches — five things:

  1. The two meanings of "accepted" are separate signals. xfer is the handshake; accept is the claim. Fusing them into one req_ready gated on the address is the deadlock of §9.
  2. The offset needs no subtractor, and that is a consequence of alignment. req_addr[OFF_W-1:0] is the whole computation. It is correct because Chapter 9.5 guarantees an aligned base, which is why P4 checks the guarantee rather than assuming it.
  3. The offset is computed and registered at acceptance. A base change afterwards cannot alter a request already in flight, because the derived value was captured, not recomputed.
  4. Byte enables are carried. A front end that accepted req_be and dropped it would turn a partial write into a full-width one — silent data corruption at the exact granularity software chose deliberately.
  5. There is no path from the runtime side into the window state. Chapter 9.1's thesis, still structural three chapters later: the ports are inputs.

Deliberately simplified: one window; one 32-bit data beat per request rather than a burst; a one-deep ownership stage; no response path for reads; no behaviour defined for a miss; no I/O-space channel (Chapter 9.3 §8 shows the space-qualified form).

Production implication: a real Endpoint front end decodes several windows, handles multi-dword requests with a length and a data stream, buffers more than one request, generates the return transaction for reads (Module 10), and defines a response for requests that match no window.

12. Assertions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SVA over bar_resource_frontend. Implementation invariants for THIS design
// plus the normative behaviours it implements — not claims about PCIe beyond
// sections 5 and 6. Every property refers to explicit RTL state or inputs.
 
// ENVIRONMENT ASSUMPTIONS — stated as assumptions, not smuggled in as
// properties of this module. The requester owes valid-stability; without it
// no downstream property about ownership can hold.
assume property (@(posedge clk) disable iff (!rst_n)
  (req_valid && !req_ready) |=> req_valid);
assume property (@(posedge clk) disable iff (!rst_n)
  (req_valid && !req_ready) |=> ($stable(req_addr) && $stable(req_write)
                                 && $stable(req_wdata) && $stable(req_be)));
 
// CLAIM — P1: a request that entered the resource path was inside the window
// at the moment it was accepted. Checked against the SNAPSHOTS, so a later
// base change cannot make a correct acceptance look wrong.
property p_only_in_window_accepted;
  @(posedge clk) disable iff (!rst_n)
  res_valid |-> ((addr_at_accept >= base_at_accept)
              && ((addr_at_accept - base_at_accept) < SIZE));
endproperty
a_in_window : assert property (p_only_in_window_accepted);
 
// PERMISSION — P2: no request is claimed while the space is disabled.
// Chapter 8.4's Command Register bit 1, composed with this design.
property p_disabled_claims_nothing;
  @(posedge clk) disable iff (!rst_n)
  !mem_space_enable |-> !accept;
endproperty
a_needs_enable : assert property (p_disabled_claims_nothing);
 
// PERMISSION — P3: no request is claimed before the window is programmed.
property p_unprogrammed_claims_nothing;
  @(posedge clk) disable iff (!rst_n)
  !base_valid |-> !accept;
endproperty
a_needs_base : assert property (p_unprogrammed_claims_nothing);
 
// CORRECTNESS — P4: the offset equals the TRUE subtraction, in full width.
// The design computes a bit-slice; this property computes A - B. They agree
// if and only if the base is aligned, so P4 is simultaneously an offset check
// and an alignment check on state this module does not own.
property p_offset_is_true_difference;
  @(posedge clk) disable iff (!rst_n)
  res_valid |-> ({{(64-OFF_W){1'b0}}, res_offset}
                 == (addr_at_accept - base_at_accept));
endproperty
a_offset_exact : assert property (p_offset_is_true_difference);
 
// INTERFACE — P5: payload is stable while stalled. The handshake discipline
// of section 10, made checkable.
property p_payload_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
  (res_valid && !res_ready) |=> (res_valid && $stable(res_offset)
                                 && $stable(res_write) && $stable(res_wdata)
                                 && $stable(res_be) && $stable(base_at_accept)
                                 && $stable(addr_at_accept));
endproperty
a_payload_held : assert property (p_payload_stable_under_stall);
 
// INTERFACE — P6: the only reason to stall the requester is a full stage that
// is not draining. Catches a ready that acquired a dependence on the address
// or on req_valid, which is section 9's deadlock.
property p_stall_only_when_full;
  @(posedge clk) disable iff (!rst_n)
  !req_ready |-> (res_valid && !res_ready);
endproperty
a_stall_reason : assert property (p_stall_only_when_full);
 
// CONSERVATION — P7: one claimed request produces one internal request.
// The stage is occupied on the cycle after an acceptance...
property p_accept_produces_request;
  @(posedge clk) disable iff (!rst_n)
  accept |=> res_valid;
endproperty
a_accept_fills : assert property (p_accept_produces_request);
 
// CONSERVATION — P8: ...and it empties on handoff unless immediately refilled,
// so a request is never delivered twice.
property p_handoff_empties;
  @(posedge clk) disable iff (!rst_n)
  (res_valid && res_ready && !accept) |=> !res_valid;
endproperty
a_no_duplicate : assert property (p_handoff_empties);
 
// CONSERVATION — P9: a miss never becomes an internal request. Together with
// P7 this partitions every handshake into exactly one outcome.
property p_miss_never_enters;
  @(posedge clk) disable iff (!rst_n)
  (miss && !res_valid) |=> !res_valid;
endproperty
a_miss_dropped : assert property (p_miss_never_enters);
 
property p_outcomes_exclusive;
  @(posedge clk) disable iff (!rst_n)
  !(accept && miss);
endproperty
a_one_outcome : assert property (p_outcomes_exclusive);
 
// SAFETY — P10: byte enables reach the resource unchanged.
property p_byte_enables_propagated;
  @(posedge clk) disable iff (!rst_n)
  accept |=> (res_be == $past(req_be) && res_wdata == $past(req_wdata)
              && res_write == $past(req_write));
endproperty
a_be_carried : assert property (p_byte_enables_propagated);
 
// SAFETY — P11: the offset never exceeds the resource. Structural here, since
// res_offset is exactly $clog2(SIZE) bits wide, and asserted so a later width
// change that breaks it fails loudly.
property p_offset_within_resource;
  @(posedge clk) disable iff (!rst_n)
  res_valid |-> ({{(64-OFF_W){1'b0}}, res_offset} < SIZE);
endproperty
a_offset_bounded : assert property (p_offset_within_resource);
 
// SAFETY — P12: no output is ever unknown.
property p_outputs_never_unknown;
  @(posedge clk) disable iff (!rst_n)
  !$isunknown({req_ready, res_valid, miss});
endproperty
a_no_x : assert property (p_outputs_never_unknown);

P4 is the property that earns its place twice. The design computes req_addr[OFF_W-1:0]; the property computes addr_at_accept − base_at_accept at full width and requires equality including the upper bits. Those two expressions agree exactly when the base is aligned to the size — which is Chapter 9.5's guarantee, imported from a module this one does not own. So P4 checks the offset and checks that the guarantee it depends on is actually holding. A misaligned base makes the difference nonzero above bit OFF_W−1, and the property fires immediately rather than after someone notices data landing in the wrong register.

P6 is how a deadlock becomes a failing assertion instead of a hang. The dangerous edit to this module is gating req_ready on mine, which looks like a tightening and is a liveness bug: a requester offering an address this window does not claim would never see ready. P6 says the only legitimate reason to withhold ready is a full stage that is not draining. Adding an address dependence violates it on the first non-matching request.

P1 and P4 both read the snapshots rather than the live inputs, and that is deliberate. active_base may legitimately change while a request is in flight (Chapter 9.5 §11 commits on an enable edge). A property written against the live base would fail on a perfectly correct design the moment a commit happened mid-flight. Asserting against what was true at acceptance is the honest contract, and it is why the snapshots exist as outputs.

P7, P8 and P9 partition every handshake. Each xfer is either an accept — which fills the stage — or a miss — which does not. P10 covers the payload of the first case, P8 prevents the stage from being drained twice, and P9 prevents a miss from leaking into the resource. Together they are the conservation argument for the whole module.

13. Verification

Monitors observe: both handshakes with all payload fields; the window state and permission input; mine, accept, and miss; and both snapshots.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// VERIFICATION-ONLY. Independent reference model. Deliberately shares NOTHING
// with the design: it computes membership by comparison and the offset by
// subtraction, where the design uses a mask and a bit-slice.
function automatic bit ref_in_window(input logic [63:0] addr,
                                     input logic [63:0] base,
                                     input logic [63:0] size);
  // Written as two comparisons rather than base + size, which would wrap for
  // a window at the very top of the address space (Chapter 9.5 section 12).
  if (addr < base) return 1'b0;
  return (addr - base) < size;
endfunction
 
function automatic logic [63:0] ref_offset(input logic [63:0] addr,
                                           input logic [63:0] base);
  return addr - base;   // valid only once ref_in_window has returned 1
endfunction

The scoreboard must not reuse the design's mask or its bit-slice. Both of the design's shortcuts are correct only under an alignment guarantee supplied by another module; a checker that adopted the same shortcuts would agree with the design about exactly the case where the guarantee has failed.

Window boundaries

  • The first address. Exactly base. Must be claimed; offset 0.
  • The last address. base + size − 1. Must be claimed; offset size − 1. The boundary a mask error moves.
  • One past the end. base + size. Must not be claimed. In the worked example this is 0x8001_0000, and it is the address most likely to be wrongly accepted.
  • One below the base. Must not be claimed.
  • A window at the very top of the address space. Verify the last address is still claimed and nothing wraps — the masked form has no end-address arithmetic, and this proves it rather than assuming it.
  • An address inside the window with every low bit set. Verify the offset is size − 1 and P11 holds.

Permission and programming

  • Memory Space Enable clear, window otherwise perfect. Verify nothing is claimed at any address (P2) and that miss pulses instead — the distinction that tells a debugger the access arrived.
  • base_valid low. Verify nothing is claimed (P3).
  • Enable asserted mid-stream. Verify claims begin on the correct cycle and no request is lost or duplicated at the transition.
  • Enable cleared mid-stream with a request in the ownership stage. Verify the in-flight request completes normally — it was claimed under the permission that was in force — and that no new request is claimed.

Handshake and backpressure

  • Continuous writes with res_ready high. Verify one request per cycle sustained and no dead cycles.
  • res_ready held low for many cycles. Verify res_valid stays high, every payload field is stable (P5), req_ready falls, and the requester's offer is preserved.
  • res_ready toggling every other cycle. Verify no request is dropped or duplicated (P7, P8).
  • A read request. Verify res_write is low and the offset is computed identically — the read path through this module is the same path.
  • Byte enables. All fifteen nonzero patterns, plus zero. Verify each reaches the resource unchanged (P10).
  • Requests offered and withdrawn is not tested, because the environment assumption forbids it — verify instead that the assumption itself is exercised and never violated by the stimulus generator.

Interaction with the window state

  • BAR reprogrammed while idle. Verify the new base takes effect for subsequent requests and the old window no longer claims.
  • Base committed while a request is in the ownership stage. Verify the in-flight offset is unchanged (the snapshot) and that P1 and P4 still hold — this is the test the snapshots exist for, and a design that recomputed the offset from the live base would fail it.
  • Sustained traffic with no configuration activity. Verify the window state never changes — the runtime side has no write port, and this makes the absence checkable.
  • Reset with a request in flight. Verify the stage clears and nothing is delivered afterwards.

Miss behaviour

  • An address outside the window with the stage empty. Verify miss pulses for exactly one cycle, req_ready stays high, and nothing enters the resource path (P9).
  • A stream alternating in-window and out-of-window addresses. Verify each is classified correctly and the interface never stalls on a miss.

Coverage should include: addresses at both boundaries and one beyond each; resource sizes from the 128-byte Endpoint minimum upward, including one whose window ends at the top of the address space; both permission inputs in all four combinations; reads and writes; all byte-enable patterns; res_ready continuously high, continuously low, and randomly toggled; and a base commit landing in each of the stage's states.

14. Debugging

Symptom: configuration space reads correctly, the BAR reads correctly, and MMIO never reaches the register block

This is the definitive Module 9 ladder. Every rung is cheaper than the one below it, and the order is chosen so that the most common causes fall out first.

  1. Is the host using the right address? Compare the physical address the driver's mapping resolves to against the assigned base. A stale mapping, or the base of a different Function, produces this exactly. The driver's pointer is not the BAR value (§4) — resolve the translation, do not compare the numbers.
  2. Does the BAR hold what was written? Read it back and compare against the value software wrote, not against a range. A mismatch is Chapter 9.5 §3's silent rounding and the window is not where anyone thinks.
  3. Is Memory Space Enable set? Command Register bit 1 (8.4). It resets to 0, programming does not set it, and this is the single most common cause. One configuration read.
  4. Do the parent routing windows contain the assigned range? Every bridge between the Root Complex and the Function applies its own (§7). Configuration access keeps working when this is wrong, because it routes by identity — which is precisely why the symptom is so misleading.
  5. Does any request reach the Function at all? This is the rung that splits the investigation in half. Watch req_valid — or miss. If nothing arrives, the fault is at rungs 1, 2, or 4 and nothing inside the Function will reveal it. If requests arrive and miss, the fault is local and rungs 6–8 apply.
  6. Does the local decode claim the address? Check minebase_valid, mem_space_enable, and the masked comparison. Compute membership by hand from the base and size.
  7. Is the offset right? Compare res_offset against A − B computed by hand. A disagreement means the base is not aligned (§5), which sends you back to rung 2.
  8. Is the resource accepting? res_valid high with res_ready low for a long time is backpressure, not a decode fault — and it is a resource problem, not a PCIe one.
  9. For reads, does the response ever return? Module 10 territory; see the next scenario.

The single observation that halves the search: rung 5. "Did an access arrive?" separates the hierarchy (rungs 1, 2, 4) from the Function (rungs 6–8), and those two halves need completely different instruments. Everything above rung 5 is a configuration read; everything below needs visibility inside the device. Do the cheap half first.

Symptom: writes work, reads hang

The asymmetry is the diagnosis. Writes reaching the resource proves an enormous amount at once: the host's address is right, the BAR is right, the enable is set, the parent windows forward, the request arrives, the decode claims it, and the offset is usable. Rungs 1 through 8 of the ladder above are all confirmed by a working write.

So the fault is on the path a write does not use: the return direction (§8).

What that narrows it to. Something must carry the read data back and identify which request it answers. Candidates live in the Function's response generation, in its correlation of a response to a request, and in the outbound path that carries it — none of which a write exercises.

Why this chapter stops here. The return transaction's structure, its identification, and the rules governing it are Module 10's subject. What this chapter contributes is the inference: a working write is a proof of the entire forward path, so a hanging read is a response-path problem and should be investigated as one from the first minute.

The habit worth forming. Before debugging a read failure, issue a write to the same window and confirm it lands. It costs one transaction and eliminates eight rungs.

Symptom: an access to offset 0x120 reaches internal register 0x100

The displacement is 0x20 — one bit. That is the signature, and it separates the candidates immediately.

First, classify the error. Issue accesses across the window and record the displacement each time.

ObservationPoints at
The displacement is constant across all addressesthe base — a misaligned or stale base, since A − B shifts every offset equally
The displacement varies with the address and always clears the same bitthe offset path — a dropped or mis-wired bit between res_offset and the resource
Offsets are scaled by 4 (or divided by 4)a byte-versus-dword unit confusion between the front end and the resource

For the constant case, the arithmetic is exact. If B = 0x8000_0020 instead of 0x8000_0000, then A − B for A = 0x8000_0120 is 0x100. A misaligned base by 0x20 produces precisely this symptom — and it also produces the divergence P4 exists to catch, because the bit-slice form would have said 0x120.

Which is the fastest confirmation available: compute both forms and compare. req_addr[OFF_W-1:0] versus addr_at_accept − base_at_accept. They are equal if and only if the base is aligned, so a disagreement identifies the fault class in one observation and sends you to Chapter 9.5 §15's first scenario.

For the varying case, check widths at the boundary. res_offset is $clog2(RESOURCE_SIZE) bits; a resource whose index port is narrower silently drops the top bits, and one wired with an off-by-one slice drops a middle bit. The signature — always the same bit cleared — is what distinguishes it from a base error.

Why the stale-base variant is worth naming separately. If the base was committed while a request was in flight and the offset were recomputed from the live base rather than from the snapshot, the displacement would appear only on the requests that straddled the commit — intermittent, load-dependent, and untraceable. §11 captures the offset at acceptance precisely so this cannot happen, and §13 tests it.

15. Common Misconceptions

  • "The host reads configuration space repeatedly to access the device." Configuration space is used at enumeration and for control-plane work. Operational access goes to the assigned address in memory space and never touches the BAR.
  • "The BAR value is the address of the register inside the Endpoint." It is the address of the window in the host's address space. The register's location inside the resource is the offset, and the two differ by the base.
  • "The CPU virtual address equals the BAR value." They are related by an OS-controlled translation (§4). Expecting numeric equality produces "the address is wrong" reports about a system that is behaving correctly.
  • "If the Function's decode is right, the access will arrive." Local decode correctness and end-to-end reachability are different properties with different owners (§7). Every parent routing window must also contain the range.
  • "Memory Space Enable programs the BAR." It grants permission to respond. Programming and permission are independent (8.4, 9.5 §8), and both are required.
  • "Host access bypasses the Transaction Layer." Every operational access crosses the Link as a transaction. What that transaction contains is Module 10; that it exists is why Module 10 is next.
  • "Reads and writes have the same response behaviour." They do not, which is why they fail differently (§14's second scenario). The mechanism is deferred, the asymmetry is not.
  • "The resource offset is the PCIe address." The offset is device-local and always less than the resource size. The PCIe address is system-wide. Handing an absolute address to an internal register file indexes far outside it.
  • "The BAR register changes on every MMIO access." It is not read, written, or otherwise touched at run time. Its value reached a comparator once; P-properties in Chapter 9.1 and here assert the absence of any path from the runtime side back into it.
  • "A successfully mapped address proves the driver is talking to the right Function." It proves an address was mapped. Two Functions of the same device, or two identical cards, have different bases, and a driver holding the wrong one gets a perfectly functional mapping to somebody else's window.

16. Understanding Check

17. Module 9 Complete

Six chapters, and the lifecycle closes.

9.1 What BARs Are — a configuration-space control point associated with an addressable window, not the storage behind it.

9.2 Memory BARs — attribute bits below, address bits above, and a 64-bit form spanning two consecutive dwords that describe one resource.

9.3 IO BARs — the other address space, and the fact that a number is not a complete coordinate without it.

9.4 BAR Sizing — the size is not stored anywhere; it is the shape of the register, exposed by a probe.

9.5 Address Assignment — choosing a legal base and committing it as one coherent address.

9.6 Host Access — the assigned address in use, and its conversion to a device-local offset.

The through-line, stated once: a resource exists; a BAR describes where it will be reached; software discovers how much space it needs; the host chooses an address and programs it; the Command Register permits it; the host accesses the assigned range; and the Function converts that system address into an offset inside itself. Seven steps, six chapters, and every failure in §14's ladder is one of them going wrong.

18. What's Next

Module 9 followed a host access to the edge of the Function and stopped at a question it could not answer: what actually crossed the Link?

The host issued a load. Something carried that operation across a serial link, through switches, to a Function that had never heard of it — and something carried the data back. That something is a Transaction Layer Packet, and building it is the Transaction Layer's job.

Chapter 10.1 — Requests opens Module 10: what the Transaction Layer owns, what a request is, which address spaces requests target, who is permitted to issue them, and what categories of information a request must carry. Chapter 10.2 covers the completions that answer them, 10.3 and 10.4 the two lifecycles a transaction can have, and 10.5 a full exchange across the fabric.

The idea to carry forward: the host accesses an address; the Function receives an offset; and everything in between is a transaction that Module 10 is about to open up.