Skip to content

PCIe · Module 30

Integration Checklist — Two Reasonable Assumptions

The IP is correct, the SoC is correct, and the product writes to the wrong memory. Integration failures are disagreements between individually reasonable assumptions, and every one produces perfectly legal PCIe traffic.

30.3 asked whether the environment could disprove the design. It could — inside its own testbench, where every assumption was written by one team. This gate asks what survives contact with an SoC whose assumptions were written by six.

1. What This Gate Owns

Integration failures are disagreements between individually reasonable assumptions.

Three readings, and the third is why this gate needs to exist separately.

Nothing is broken, and that is the defining property. The PCIe IP meets its specification. The SoC meets its. The driver is correct against its documentation. Every component passes its own review and the product fails — which is 29.6's composition problem moved from the fabric into the chip.

Which means the review technique is different. 30.2 inspected a module against a contract. This gate inspects a seam against two documents, and the finding is usually that the two documents describe the same thing differently — or that one of them never mentions it.

And the characteristic symptom is legal traffic with a wrong outcome. §5's flagship produces perfectly well-formed TLPs that write to the wrong physical memory. No error is signalled anywhere, because at the protocol layer nothing went wrong.

This gate ownsOwned elsewhere
seams — address, reset, interrupt, clock, power, firmware, debugthe subsystem's internal contracts — 30.1
whether the SoC honours the subsystem's stated assumptionsRTL faithfulness — 30.2
whether readiness composes correctly across layersenvironment capability — 30.3
system-level debug reachabilityperformance closure — 30.5

2. Contract Boundaries

A block diagram of a PCIe subsystem integrated into an SoC. Software and firmware sit at the top, below them an IOMMU and address translation layer, then the PCIe subsystem containing configuration and BAR logic, DMA, interrupts, reset and power, and debug. Below that is the external PCIe link to an endpoint. Contract boundaries are marked between software and address translation, between address translation and the subsystem, and between the subsystem and the link.Software / driversupplies addressesFirmwarepublishes readinessIOMMU /translationTHE address contractConfig · BARthe SoC address mapPCIe subsystemDMA · interruptsReset · power ·clocksscope must be agreedExternal linkalways looks fine12
A PCIe subsystem drawn inside its SoC, with the boundaries at which two teams must agree. The vertical path is the data path from software through address translation to the external link; the five blocks alongside are the contracts that cross it. Every failure in this chapter sits on one of these boundaries rather than inside any block, which is why reviewing the blocks individually never finds them.

3. The Integration Contract Matrix

The artefact this gate produces. One row per seam; a blank cell is a finding, not an omission.

ContractProducerConsumerOwnerClock domainReset scopeConfig sourceDV evidenceSW-visibleSilicon observableDisposition
DMA address spacedriverDMA enginemust be one ownercorefunctiondriver + IOMMUsystem test with translation activeyesfault address registerFAIL if unstated
BAR → SoC address mapSoC mapconfig logicSoC archconfigfunctionintegration paramsenumeration testyesBAR readbackPASS
subsystem readinessfirmwaredriverfirmwarecoresee §9firmwarereadiness-race testyesready registerFAIL if link-only
interrupt vector → CPUsubsysteminterrupt controllerSoC archasyncfunctionintegration paramsend-to-end deliveryyespending registerPASS
reset scopeSoC reset ctrlsubsystemSoC archasyncreset treereset-under-loadpartlyreset causeFAIL if unmapped
link readinessPHY/LTSSMfirmwarePCIe IPlinkrecoverylink-up testyesLTSSM statePASS
error escalationsubsystemSoC error aggSoC archcorefunctionpolicyinjected errorsyesfirst faultPASS
completion timeoutsubsystemdriverarchitecturecorefunctionconfigtimeout testyestimeout countPASS
clock enable / gatingpower ctrlsubsystemSoC powermultiplepower policygate under loadnoactivity counterFAIL if untested
debug reachsubsystemSoC debugSoC archcorereachability testnothe registersPASS

Three readings.

"Must be one owner" in row 1 is the entire chapter in a cell. An address contract with two owners is an address contract with two answers — and §5 is what that costs.

The "Clock domain" and "Reset scope" columns exist because they are the two dimensions that vanish in a standalone testbench. A subsystem-level environment runs one clock and one reset by construction; the SoC has several of each, and the seams between them are where the interesting cases live.

And "Silicon observable" is the column that gets left blank under schedule pressure. Every blank there is a future debug session with no evidence — 30.1 §11 already required these; this gate checks they survived integration.

4. Address and BAR Integration

Do not re-teach BARs — 9.1, 9.4, 9.5 and 9.6 own the mechanism. Check the agreement.

QuestionThe disagreement it finds
does the SoC address map match the BAR sizes requested?a BAR that cannot be placed at its required alignment
what does the SoC return for an unimplemented offset inside a BAR?silent aliasing, which reads as working (25.5)
does the driver's register map match the RTL's?two documents, both maintained, diverging quietly
are prefetchable and non-prefetchable regions correctly separated?speculative reads with side effects
which registers are safe to touch while traffic is live?30.2 §9's partial-configuration hazard
does anything else in the SoC decode the same range?two decoders, one wins, non-deterministically

And the single most effective check here is mechanical: diff the driver's header against the RTL's register definitions, automatically, in CI. Two hand-maintained copies of one map will diverge, and the divergence is discovered by a field that reads back wrong months later.

5. The DMA and IOMMU Contract

The highest-severity seam in this chapter, because it produces perfectly legal traffic that lands in the wrong memory.

The question is one sentence and it must have exactly one answer: what address space do the addresses the driver hands the device live in?

PossibilityImplication
physicalthe device writes where it is told; no translation
IOVA — translated by an IOMMUthe address is meaningless without translation
guest-physical under virtualisationanother translation layer
restricted / windowedvalid only within a programmed aperture

Wrong integration — the driver supplies an IOVA and the DMA engine treats it as physical.

StepWhat happensWho is wrong
0driver allocates a buffer, maps it, obtains an IOVAnobody
1driver programs the descriptor with the IOVAcorrect per its documentation
2DMA engine issues a memory write to that addresscorrect per its documentation
3the address is not translated — it goes out as-isthe seam
4the write is a perfectly legal PCIe memory write✓ well-formed, ✓ routed, ✓ acknowledged
5it lands on whatever physical memory that number names
6no PCIe error, no completion error, no fault
7symptom: unrelated memory corruption, or nothing at allarbitrarily far away

First divergence: step 3, and it is not a step anybody performs — it is a step nobody performs, which is why no code review finds it.

Four readings.

Both documents are correct in isolation. The driver's says "the descriptor takes a DMA address obtained from the mapping API." The RTL's says "the descriptor address is issued as the memory address." Neither is wrong; together they are a silent memory corruptor.

The symptom is the worst available. With an IOMMU actively enforcing, the access is blocked and you get a fault with an address — annoying and diagnosable. Without enforcement, the write succeeds into unrelated memory, and the failure surfaces as instability somewhere else entirely, possibly much later.

Which makes the IOMMU-enabled configuration the easier one to debug, and it should be tested first. A fault register naming the offending address converts this from a multi-week investigation into a one-line finding — so "is the IOMMU-enforcing path in the test matrix?" is a checklist item with unusually high value.

And the durable fix is to make the address space part of the interface, not the prose. Name the field dma_iova rather than dma_addr; state the space in the register description; assert it in the driver. A field whose name states its space cannot be misread by the next integrator.

6. RTL — Aperture Validation and Bounds

§5's failure is silent because nothing checks. The containment is cheap, and it converts an unbounded corruption into a contained, attributed, reported event.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ILLUSTRATIVE. An aperture guard at the DMA engine's address output. It cannot
// fix an address-space disagreement — nothing in hardware can — but it converts
// a silent write into somebody else's memory into a REPORTED containment event
// with the offending address captured.
localparam int AW = 64;
 
logic [AW-1:0] ap_base_q, ap_limit_q;   // programmed by firmware
logic          ap_valid_q;              // written
logic          ap_checked_q;            // validated: limit > base, aligned, mapped
 
logic [AW-1:0] first_viol_addr_q;       // FIRST, not last — evidence survives
logic          first_viol_valid_q;
logic [31:0]   viol_count_q;
 
// In range means BOTH bounds and the whole burst — checking only the start
// address lets a transfer begin legally and run off the end, which is the
// subtle version of the same bug.
wire in_range = ap_valid_q && ap_checked_q
             && (dma_addr >= ap_base_q)
             && ((dma_addr + AW'(dma_bytes)) <= ap_limit_q);
 
// The request is only emitted if it is contained. Containment is a gate, not a
// warning — a violation that is merely counted still corrupts memory.
assign dma_req_valid = dma_req_pending && in_range;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    first_viol_addr_q <= '0; first_viol_valid_q <= 1'b0; viol_count_q <= '0;
  end else if (dma_req_pending && !in_range) begin
    // First-fault semantics: the first violation is the diagnostic one, and a
    // stream of subsequent violations must not overwrite it (30.1 §13 Q40).
    if (!first_viol_valid_q) begin
      first_viol_addr_q  <= dma_addr;
      first_viol_valid_q <= 1'b1;
    end
    viol_count_q <= viol_count_q + 32'd1;
  end
end
 
// MANDATORY. English: no DMA request leaves the subsystem outside the validated
// aperture. This is the property that bounds §5's blast radius.
a_dma_within_aperture: assert property (
  @(posedge clk) disable iff (!rst_n)
    dma_req_valid |-> in_range
);
 
// MANDATORY. English: no DMA is issued before the aperture has been both
// programmed AND validated. Catches §8's readiness race at the point of harm.
a_no_dma_before_aperture: assert property (
  @(posedge clk) disable iff (!rst_n)
    dma_req_valid |-> (ap_valid_q && ap_checked_q)
);
 
// MANDATORY. English: the first violation address is sticky until cleared.
a_first_viol_sticky: assert property (
  @(posedge clk) disable iff (!rst_n || viol_clear)
    $rose(first_viol_valid_q) |=> always first_viol_valid_q
);

Architecture. A gate, not a monitor. A violation that is counted but still emitted has not been contained — the write already happened, and the counter merely documents it.

State. Aperture bounds, two qualifying bits, and first-fault capture. ap_checked_q is separate from ap_valid_q because programmed and validated are different facts — §9's readiness aggregation requires both for the same reason.

Event. The check runs on every request; the first-violation capture fires once, and viol_count_q counts the rest.

Contract. Firmware must set ap_checked_q only after verifying the aperture against the SoC address map and the IOMMU's mapping — this hardware cannot verify translation, and claiming otherwise would be the same overreach §5 is about. What it can guarantee is containment within a programmed range.

Failure. Checking only dma_addr and not dma_addr + dma_bytes lets a legal-looking transfer run off the end of the aperture, which is the subtle version of §5 and is harder to spot because the start address is correct. Last-violation-wins instead of first destroys the diagnostic in exactly the burst of violations that follows the first one.

DV/debug. first_viol_addr_q is the register that makes §5 diagnosable without an enforcing IOMMU — it names the offending address from inside the subsystem. One register, and it turns a multi-week memory-corruption investigation into a one-line finding, which is the same trade every Module 29 chapter ended on.

7. Reset Ownership Across the Seam

30.1 §6 built the subsystem's reset matrix. This gate asks whether the SoC's reset tree implements it.

ResetWho asserts itWhat the subsystem expectsThe integration failure
SoC cold resetpower sequencereverything clearsdeassertion not synchronised to each domain
PCIe block resetSoC reset controllerper 30.1 §6mapped to the wrong scope — clears more or less than intended
function-level resetsoftware, via configthis function's statesiblings affected
link Recovery (18.5)the linkretain live requestsSoC treats it as an error and resets the block
software resetdrivera defined sequencedriver and RTL disagree on what it clears
power-domain transitionpower controllerstate retained or restoredretention list incomplete

Three readings.

Row 4 is the classic and it is expensive. The SoC's error aggregator sees a link event, classifies it as a fault, and resets the block — while the subsystem's architecture said Recovery retains live requests. The result is 30.2 §7's orphaned-work timeline, triggered by the platform rather than by the RTL.

Row 3 is 29.6 §14's blast radius, one level down. A function-level reset that disturbs a sibling function is the same failure as a subtree reset disturbing a sibling endpoint, and it has the same cause: scope decided without reference to ownership.

And row 1's deassertion synchronisation is the one that produces impossible-looking bugs. Different domains leaving reset on different cycles means the first cycle of operation differs between them — and the symptom is a rare, ratio-dependent startup failure that disappears under instrumentation.

8. Readiness Is a Hierarchy, Not a Bit

The most under-modelled contract in the whole gate: ready means different things at five different layers, and they complete in order.

Layer"Ready" meansCompletes when
1 — physical / linkthe link reached L0 (18.6)training completes
2 — enumeratedthe device has been discovered and BARs assigned (7.1)enumeration finishes
3 — configuredthe subsystem's own configuration is committedfirmware programs it
4 — resourcedDMA apertures mapped, IOMMU entries installed, interrupts routeddriver setup completes
5 — functionalthe device can actually serve a requestall of the above, and only then

Exposing layer 1 as though it were layer 5 is the single most common readiness bug, and it is attractive precisely because layer 1 is the easiest to observe.

Wrong integration — firmware publishes readiness at link-up.

TimeEventready bitReality
0link reaches L0layer 1 only
1 msfirmware sets ready1layers 3, 4 incomplete
1.1 msdriver issues a request1DMA aperture not yet programmed
1.2 msdevice performs a DMA to an unmapped address1§5's failure, triggered by a race
5 msfirmware finishes configuration1too late
laterworks perfectly on every subsequent boot1the race is timing-dependent

Three readings.

The intermittency is the defining feature. A slow boot hides it entirely; a fast one exposes it. So it appears on one machine, in one lab, at one temperature — and the natural conclusion is that the hardware is marginal.

Readiness must be an aggregate whose terms are each individually observable. A single bit tells you it is not ready; five bits tell you which layer is missing, which is the difference between a bug report and a diagnosis.

And the aggregation belongs in hardware where the terms are hardware. §9 is that register. Firmware may add its own term, but it must not be the only term — firmware asserting readiness on its own behalf is precisely the failure above.

9. RTL — Readiness Aggregation

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ILLUSTRATIVE. Readiness as a conjunction of individually observable terms.
// The design goal is that "not ready" is always answerable with "because of
// WHICH term", which is what converts a boot race into a one-read diagnosis.
typedef struct packed {
  logic link_l0;        // layer 1 — from the LTSSM
  logic enumerated;     // layer 2 — BARs assigned
  logic cfg_committed;  // layer 3 — 30.2 §9's commit, not a shadow write
  logic dma_mapped;     // layer 4 — aperture programmed AND validated
  logic irq_routed;     // layer 4 — vectors installed
  logic fw_ready;       // layer 5 — firmware's own term, one of six
} ready_terms_t;
 
ready_terms_t ready_q;
logic         func_ready_q;
 
// Software reads THIS to find out which term is missing. One register.
assign ready_status = ready_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    ready_q <= '0; func_ready_q <= 1'b0;
  end else begin
    // Each term is set by its own owner and, critically, CLEARED by the event
    // that invalidates it. A term that only ever sets is a latch for a
    // condition that can go away — which is how a stale ready survives a reset.
    ready_q.link_l0       <= ltssm_state_is_l0;              // level, not a pulse
    ready_q.enumerated    <= bars_assigned;
    ready_q.cfg_committed <= cfg_valid_q;                    // 30.2 §9
    ready_q.dma_mapped    <= dma_aperture_valid && dma_aperture_checked;
    ready_q.irq_routed    <= irq_vectors_valid;
    ready_q.fw_ready      <= fw_ready_wr ? fw_ready_data : ready_q.fw_ready;
 
    func_ready_q <= &ready_q;                                // all six, always
  end
end
 
// MANDATORY. English: no request is accepted unless the function is ready.
// This is the property that makes the race impossible rather than unlikely.
a_no_traffic_before_ready: assert property (
  @(posedge clk) disable iff (!rst_n)
    req_fire |-> func_ready_q
);
 
// MANDATORY. English: readiness is not sticky. If a term goes away — link left
// L0, configuration invalidated — readiness must drop in the next cycle.
a_ready_not_sticky: assert property (
  @(posedge clk) disable iff (!rst_n)
    (!(&ready_q)) |=> !func_ready_q
);

Architecture. A conjunction, not a bit. Each term has exactly one owner and each is separately readable, so "not ready" is always answerable with "because of which term."

State. Six terms plus the aggregate. dma_mapped requires _valid && _checked because a programmed aperture that was never validated is the §5 failure waiting for a trigger.

Event. Every term is assigned from a level, not a pulse. A term set by a pulse cannot be cleared by the condition going away, which produces a readiness bit that survives events that should invalidate it — including a link drop.

Contract. Firmware owns exactly one term. The programming model must say so, or the next integrator will find fw_ready and use it as the readiness bit, reproducing §8 exactly.

Failure. Any term latched rather than levelled → stale readiness after a link event. func_ready_q computed from a subset → the race returns, now harder to see because a readiness register exists and looks thorough.

DV/debug. a_no_traffic_before_ready is the property; the negative test is issue traffic with one term deasserted and confirm it fires. In silicon, ready_status turns "the device didn't come up" into "dma_mapped is 0"one read.

10. Interrupts Across the Seam

QuestionThe disagreement
does each MSI-X vector reach the intended CPU handler?vector table programmed by one team, routed by another
is data visible before the interrupt is observed?the producer-consumer inversion, now with an SoC interconnect in between
what happens to pending interrupts across each reset?30.1 §13 Q34's row, at SoC scope
does interrupt moderation interact with the SoC's aggregator?double moderation, or none
is there a path for a lost interrupt to be recovered?a poll fallback, or a permanent hang

And the ordering question deserves emphasis because the SoC changes it. 19.2's mechanism is a memory write, which is why it is ordered with respect to data. An SoC interconnect between the subsystem and memory may reorder, buffer or route differently — so the property that held at the subsystem boundary must be re-argued at the SoC boundary, not assumed to carry.

11. Clocks, Power and Gating

QuestionThe failure
which clocks does the subsystem need, and are they all present before reset deassertion?a domain starting with no clock latches an undefined state
can any clock be gated while traffic is outstanding?outstanding requests stall forever; the peer times out
does a power-domain transition preserve the state 30.1 §6 says must survive?an incomplete retention list
are single-cycle events crossed safely at every SoC boundary?30.2 §12, now with domains you do not control
is reset deassertion synchronised per domain?§7 row 1

Two readings.

Gating with work outstanding is the highest-severity item here and it is easy to arrive at: the power controller sees an idle clock activity signal, not an idle protocol state. "No transitions on this bus" and "no requests outstanding" are different facts, and only the second is safe to gate on. The subsystem must export an outstanding-work indication to the power controller — which is a contract row, and it is frequently missing.

And an activity counter is the cheap observable that settles it. "Was the clock gated while outstanding_q was non-zero?" is a one-bit sticky answer, and without it a gating-induced timeout is indistinguishable from a link problem.

12. Software and Firmware Contracts

ContractFailure when unstated
the reset sequence the driver must performa driver reset that leaves hardware half-configured
which registers are safe to touch while live30.2 §9 partial configuration
the readiness protocol§8
the descriptor ownership handoff point30.1 §13 Q17 — the producer-consumer class
the error-clearing protocolfirst fault cleared before it is read
the address space of every address the driver supplies§5
what the driver must do after each reset typerecovery that leaves the device unusable

And the reviewable artefact is a single document that both teams signed, not two documents that agree today. Every row here is a place where "reasonable" differs between a hardware engineer and a driver author — and neither will discover it by reading their own side.

13. Debug Reachability

RequirementWhy
30.1 §11's observables are readable from the SoC, in the failing statea register behind a gated clock is unreadable exactly when needed
a first-fault register survives the escalation that follows the fault25.1
reset cause is recorded§7 — otherwise "who reset us?" is unanswerable
readiness terms are individually readable§8, §9
a shared timestamp exists across subsystem, SoC and software events29.6 §15: correlation needs a common time base
counters survive a functional resetevidence outliving the event

The sharpest version of this gate's question: in the failing state, is the evidence readable? A debug register in a powered-down or clock-gated domain is absent precisely when the failure occurs — and that is a design finding, not a lab inconvenience.

14. The Integration Checklist — 46 Questions

Address and BAR

#QuestionWhyEvidenceFAIL if
1Does the SoC address map accommodate every BAR's size and alignment?9.4the map with BARs placed"it'll fit"
2Does anything else decode the same range?non-deterministic winnera decode auditnot checked
3What is returned for unimplemented offsets inside a BAR?silent aliasing (25.5)a defined responseundefined
4Is the driver's register map diffed against the RTL's, automatically?§4the CI checktwo hand-maintained copies
5Are prefetchable and non-prefetchable regions correctly separated?speculative reads with side effectsthe classificationmixed
6Which registers are safe to write while traffic is live?30.2 §9per-register statementunstated

DMA and address translation

#QuestionWhyEvidenceFAIL if
7What address space do driver-supplied addresses live in?§5one stated answertwo documents that never mention it
8Does the field name state its space?§5dma_iova vs dma_addra generic name
9Is the IOMMU-enforcing configuration in the test matrix?§5 — it is the diagnosable casethe testonly the permissive path tested
10Is there a fault-address register readable after a translation fault?§5the registernone
11Is the DMA aperture validated, not just programmed?§9's _checked termthe validation stepprogrammed only
12Does the device ever generate an address outside the aperture?containmenta bounds check + counterunchecked
13Is coherency assumed anywhere without being stated?assumption, not propertythe explicit statementassumed
14Is scatter-gather alignment/length/termination agreed with the driver?20.4the joint documentinherited from an example
15Where exactly does descriptor ownership transfer?30.1 §13 Q17one named eventtwo readings

Reset

#QuestionWhyEvidenceFAIL if
16Is every reset in 30.1 §6's matrix mapped to an SoC reset?§7the mappingunmapped rows
17Does the SoC treat link Recovery as an error?§7 row 4the error policyit resets the block
18Does a function-level reset disturb a sibling function?§7 row 3the scope testuntested
19Is reset deassertion synchronised per clock domain?§7 row 1the synchronisersshared async deassert
20Is reset cause recorded and readable afterwards?§13the registernot recorded
21Is reset tested with work outstanding at SoC level?30.3 §10the testidle-only
22Does the driver's reset sequence match the hardware's expectation?§12the joint sequencetwo versions

Readiness

#QuestionWhyEvidenceFAIL if
23Is readiness an aggregate of all five layers?§8the conjunctionlink-up published as ready
24Is each term individually readable?§9ready_statusone bit
25Are terms driven from levels, not pulses?§9the assignmentslatched terms
26Does firmware own exactly one term?§9the programming modelfirmware owns the bit
27Is traffic-before-ready impossible, or merely unlikely?§9a_no_traffic_before_readya software convention
28Is the boot race tested at varying boot speeds?§8's intermittencythe timing sweepone boot path

Interrupts

#QuestionWhyEvidenceFAIL if
29Does every vector reach its intended handler?§10end-to-end delivery per vectorvector 0 tested only
30Is data visible before the interrupt is observed at SoC level?§10the re-argued orderingassumed to carry
31What happens to pending interrupts across each reset?§10the matrix rowblank
32Is there a recovery path for a lost interrupt?§10poll fallback or timeoutpermanent hang
33Does moderation interact correctly with the SoC aggregator?§10the combined behaviourdouble moderation

Clocks and power

#QuestionWhyEvidenceFAIL if
34Are all clocks present before reset deassertion?§11the sequenceunspecified
35Can a clock be gated with work outstanding?§11outstanding-work export to the power controllergated on bus idleness
36Is there an activity/violation counter for gating?§11the sticky bitnone
37Does the retention list cover everything 30.1 §6 says must survive?§11the list, diffedpartial
38Are single-cycle events crossed safely at SoC boundaries?30.2 §12the schemebare pulses

Errors, software, debug

#QuestionWhyEvidenceFAIL if
39Does the SoC error aggregator classify PCIe events correctly?§7 row 4the classification tableeverything is fatal
40Does first fault survive the escalation it triggers?§13the retentioncleared by the response
41Is the error-clearing protocol agreed with software?§12the joint documentdriver clears before reading
42Is there one signed document per software contract?§12the documenttwo agreeing documents
43Are the observables readable in the failing state?§13powered/clocked-domain analysisbehind a gated clock
44Is there a shared timestamp across subsystem, SoC and software?29.6 §15the time basethree unrelated clocks
45Do counters survive a functional reset?§13the reset rowcleared
46Has the full stack been run — real driver, real firmware, real IOMMU?§1the system runsubsystem testbench only

15. Misconceptions

"The IP is verified and the SoC is verified, so integration is a formality." §1: both are correct and the product writes to the wrong memory. The failures live on the seams.

"A DMA address is a DMA address." §5: physical, IOVA, guest-physical and windowed are four different things, and choosing wrong produces legal PCIe traffic into unrelated memory with no error.

"The IOMMU makes it safer to debug." Correct, and it is the reason to test that path first — an enforcing IOMMU turns a silent corruption into a fault with an address.

"Link up means the device is ready." §8: that is layer 1 of five. Publishing it as readiness is the most common readiness bug, and its symptom is a boot race that appears on one machine.

"It works on every boot." §8: the race is timing-dependent. A slow boot hides it completely.

"The clock is gated when the bus is idle." §11: bus idleness and zero outstanding requests are different facts, and only the second is safe to gate on.

"Recovery is an error, so we reset the block." §7: 30.1 §6 says Recovery retains live requests. The platform reset then orphans them.

"We have debug registers." §13: readable in the failing state, or not at all.

16. Understanding Check

Q1. A driver hands the device an address, the device performs a legal memory write, and unrelated memory is corrupted. What happened?

An address-space disagreement, and nobody did anything wrong (§5). The driver supplied an IOVA obtained from a mapping API — correct per its documentation. The DMA engine issued that number as the memory address — correct per its documentation. The failing step is the one nobody performs: translation. The resulting TLP is well-formed, correctly routed and acknowledged, so there is no PCIe error, no completion error and no fault — which is why no code review and no protocol analyser finds it.

Two things follow. The IOMMU-enforcing configuration is the easier one and should be tested first — enforcement turns silent corruption into a fault with an offending address, converting a multi-week investigation into a one-line finding. And the durable fix is to put the address space into the interface rather than the prose: name the field dma_iova, state the space in the register description, and assert it in the driver. A field whose name states its space cannot be misread by the next integrator.

Q2. Firmware sets a ready bit when the link reaches L0. What is wrong, and why does it pass most testing?

It publishes layer 1 of a five-layer hierarchy as though it were layer 5 (§8). Ready means different things at the link, enumerated, configured, resourced and functional layers, and they complete in order. Publishing at link-up lets the driver issue a request while the DMA aperture is not yet programmed — which triggers §5's failure as a race.

It passes most testing because the race is timing-dependent: a slow boot hides it entirely, a fast one exposes it, so it appears on one machine in one lab and reads as marginal hardware. The fix is an aggregate whose terms are individually observable (§9): six terms, each with one owner, each driven from a level rather than a pulse — a latched term cannot be cleared when its condition goes away, which produces a readiness bit that survives a link drop. Firmware owns exactly one of the six, and the programming model must say so or the next integrator will use fw_ready as the readiness bit and reproduce the original bug. a_no_traffic_before_ready makes the race impossible rather than unlikely, and ready_status turns "it didn't come up" into "dma_mapped is 0" — one read.

Q3. The SoC power controller gates the PCIe clock when the bus is idle. What can go wrong?

Bus idleness and zero outstanding work are different facts, and only the second is safe to gate on (§11). Non-posted requests can be outstanding with no bus transitions at all while their Completions are in flight (10.4) — the peer is working, and nothing is moving locally. Gating there stalls the outstanding requests indefinitely, and the failure surfaces as completion timeouts (25.7) that look exactly like a link problem.

The missing artefact is a contract row: the subsystem must export an outstanding-work indication to the power controller, and §3's matrix is where that absence becomes visible. The cheap observable is a sticky bit"was the clock ever gated while outstanding_q was non-zero?" — which separates a gating-induced timeout from a genuine link event in one read. Without it the two are indistinguishable.

Q4. Why does this gate exist separately, given that 30.2 and 30.3 both passed?

Because those gates inspect components against contracts, and this one inspects seams against two documents (§1). Every failure in this chapter has the same shape: the IP meets its specification, the SoC meets its, the driver is correct against its documentation, and the product fails — which is 29.6's composition problem moved from the fabric into the chip. The finding is usually that two documents describe one thing differently, or that neither mentions it.

Two dimensions make it structurally invisible earlier. A subsystem testbench runs one clock and one reset by construction; the SoC has several of each, and §3's "Clock domain" and "Reset scope" columns are exactly where the interesting cases live. No amount of subsystem verification reaches them, which is why question 46 — has the full stack been run with real driver, real firmware and a real enforcing IOMMU? — is the one that subsumes many of the others. And the symptom to expect throughout is legal traffic with a wrong outcome, because at the protocol layer nothing went wrong.

17. Module 30 So Far

GateAsksPassing means
30.1 Architectureis it decisive?two engineers cannot implement incompatible behaviour
30.2 RTLis it faithful?simultaneous legal events do not break ownership
30.3 Verificationcan it fail?every contract has an independent checker seen to fire
30.4 Integrationdoes it survive the SoC?the seams agree, and the evidence is readable

The remaining gates — 30.5 Performance Checklist, 30.6 Debug Checklist and 30.7 Interview Checklist — are planned. 30.5 inherits every performance assumption 30.1 §10 required to be declared, and it can only verify what this module made someone write down.

The through-line across all four. 30.1 fails on blanks, 30.2 on coincidences, 30.3 on circularity, and 30.4 on disagreement. None of them is about whether the design is clever. Every one asks the same underlying question in a different register: is there evidence, and would anyone have noticed if there were not?