DMA Failures — Where Did Ownership Stop Moving?

Chapter 25.5 asked who owns an address. This chapter asks who owns a descriptor — and the answer moves, thousands of times per second, between two agents that never stop running.

A DMA failure is an ownership handoff that was not atomic. Every fault in this chapter is one of: a handoff that never completed, a handoff that completed twice, or a handoff whose payload was read before it was written.

1. Sources, Scope, and the Boundary With Module 20 and 23.3

2. Ownership, and Why It Is the Only Useful Frame

A descriptor is a contract between two agents that share memory. Software fills it in and hands it to hardware. Hardware performs it and hands it back. Neither agent stops running while the other works, and there is no lock — the handoff is expressed as a bit in memory.

That is the entire mechanism, and it is why the failure modes are what they are:

software owns      software may write every field
   |  handoff: software's last write is the ownership bit
hardware owns      software must not touch ANY field
   |  handoff: hardware's last write is the ownership bit
software owns      hardware must not touch ANY field

Two properties make this work, and each is a fault when it is missing.

The handoff must be atomic with respect to the payload. The ownership bit must become visible after every field it describes. If the order inverts, there is a window in which the other agent sees "yours now" over data that is not yet written — §5, and §12 measured 411 such reads.

And ownership must be exclusive at every instant — 23.6 §4's law. Two owners is not a race that usually resolves correctly; it is a descriptor being written by one agent while executed by the other.

The frame pays off immediately in what it tells you to measure. "The DMA is broken" is not a question. "Where did ownership stop moving?" is, and it has a mechanical answer: the ring has a head and a tail, each descriptor has an owner, and one of them stopped advancing. §10's instruments exist to make that answer readable, and §15 case 1 shows the freeze point is usually not the fault point.

3. Stalled and Corrupting Are Different Faults

§12 measured every fault against both classes:

Five of six faults do not stop the engine. That ratio is the practical content of this section: the instinct to look for something that stopped will miss most DMA bugs, because most of them keep running.

And the one that does stop is misleading about where. In the measured run, the engine that never released a descriptor posted 16 jobs and retired 1 before wedging. The head pointer froze at the entry after the fault, not at the fault — so a debugger reading the ring finds the freeze one descriptor downstream of the defect. §15 case 1 is entirely about that offset.

The two classes also differ in how long they hide. A stall is reported by a user within seconds. A corruption is reported when a downstream computation misbehaves — often on a different machine, often days later, and often as "intermittent". Every "intermittent DMA bug" in §15 is a conservation failure that nobody was measuring, and the class is only intermittent because the measurement is missing.

4. Four Conservation Laws

These are the instruments that catch the corrupting class. Each is a quantity that must balance, each is cheap, and each catches a different fault.

Law 1 — bytes described equal bytes moved, in both directions.

sum(descriptor.length)  ==  sum(bytes actually transferred)

The two-sided form is not pedantry. §12 measured a stale-length read producing a deficit of −38,080 — the engine moved more than was described, because it read a length belonging to a previous, longer descriptor. A check written moved <= described reports success on that run. The overrun is a write past the end of a buffer, which is the most damaging fault in the chapter, and the one-sided check is blind to it by construction.

Law 2 — every job's completion is signalled exactly once.

for every submitted job:  status pulses == 1

Not "at least once", which a duplicate satisfies; not "at most once", which a loss satisfies. §12 measured the two failures separately — 1,797 jobs never signalled, and 1,797 signalled twice — and they produce opposite symptoms from the same law. A driver that counts completions sees too few in one case and too many in the other; only the per-job form distinguishes them.

Law 3 — no descriptor is written while hardware owns it.

clobbered == 0

This is 23.6 §4's exclusivity law made countable. §12 measured 12,464 clobbered descriptors under a broken ring-full test, and note what else that row shows: 12,464 jobs also never signalled. One violation, two symptoms, which is why §15 insists on reading the laws together rather than one at a time.

Law 4 — no field is read before it is written.

stale == 0

The ordering window of §5. It is the only law of the four that measures a timing relationship rather than a count, and it is the only one that cannot be checked from a memory dump after the fact.

Together the four laws separate all six faults in §12's table — no two faults produce the same signature across the four. That separation is the reason to instrument all four rather than the one that seems relevant.

5. The Ordering Window

The bug is one line of code in the wrong order, and it is the hardest fault in the chapter to reproduce.

CORRECT                          WRONG
  write length                     write ownership = HW
  write address                    write length
  write flags                      write address
  write ownership = HW             write flags

In the wrong order there is a window — between the ownership write and the last field write — in which the other agent may legally read a descriptor it now owns, containing fields that belong to the previous occupant of that slot.

Three things make this fault distinctive.

It is rare and load-dependent. §12 measured 411 stale reads in a run of 89,625 jobs — under half a percent. It requires the consumer to sample in a window a few cycles wide, so it appears only when the engine is keeping up with software, which is to say only under load, which is to say never during debugging.

Its symptom is a wrong length, not a wrong address. The stale field is whichever one had not yet been written, and length is the one that does the most damage: the transfer is the wrong size. If the previous occupant's length was larger, the engine overruns the buffer — §12's negative deficit.

And it cannot be seen in a memory dump. By the time anything stops to be inspected, every field has been written. The descriptor in memory is correct. Only an instrument sampling at the moment of the read can catch it, which is what §10's desc_order_check is and why law 4 is stated as a timing property.

One clarification, because this is frequently confused. The fix is a producer-side ordering obligation — every payload field visible before the ownership bit. PCIe's ordering rules (13.4) govern transactions on the link; they do not substitute for the producer writing its own fields in the right order. A design that relies on link ordering to repair a producer that writes the ownership bit first has misidentified whose obligation it is.

6. Full and Empty Look Identical

A ring with head and tail pointers has one genuinely ambiguous state: head == tail means both empty and full, and every ring implementation needs a way to tell them apart.

Getting it wrong in the "full reads as empty" direction is the worst outcome available, because the producer then writes over descriptors the consumer has not yet consumed.

§12 measured it, and the row is the ugliest in the chapter:

Read the first line carefully. The broken ring posts more jobs than the healthy one. A throughput measurement shows this configuration outperforming the correct one by 11%, because it never blocks — it simply overwrites. Any performance instrument, in isolation, reports this fault as an improvement.

And the two 12,464s are the same 12,464 descriptors seen through two different laws — clobbered by law 3, never signalled by law 2. That correspondence is diagnostic: when the clobber count equals the never-signalled count, the ring-full test is the fault, and §15 case 4 uses exactly that equality.

The correct discriminators are standard and this chapter takes no position among them — an ownership bit per entry (as §10 uses), an extra wrap bit on each pointer, or keeping one slot always empty. What matters for debugging is that a discriminator exists and is checked, and P14 asserts the property rather than the mechanism: the producer must never write an entry that hardware owns.

7. Retirement Must Be Exactly Once

A Memory Write is Posted and receives no Completion (12.2, 10.5). The device therefore cannot learn from the link that its data arrived — it signals completion by writing a status, and that write is itself Posted.

Everything awkward about DMA completion follows from that.

The status write must not overtake the data it describes. If software can observe "job done" before the payload is visible, it reads a buffer that is partly stale. The ordering rules that prevent this are owned by 13.4; the design obligation is to issue the status write after the data writes on a path where that ordering is guaranteed, and P17 states it.

And the status must be signalled exactly once — law 2. §12 measured both violations at the same rate on the same traffic:

Both look like "the completion count is wrong". A driver that maintains a single outstanding counter reports "1,797 jobs never finished" in the first case and "1,797 extra completions" in the second — and in a design where the counter saturates or wraps, the two can look identical.

The double-retirement case is the more dangerous of the two, and it is worth being explicit about why. A duplicated completion tells software that a buffer is free when it may have been reissued in the interval. Software then hands that buffer to something else while the engine is still writing to it — a use-after-free with a DMA engine as the writer. The lost-pulse case merely leaks a buffer.

8. The Ring, Drawn

The cycle is the instrument. Every fault in §12 attaches to exactly one arrow, and naming the arrow is the diagnosis. A DMA bug report that does not name an arrow has not localised anything — which is the same standard 25.1 §2 set for the layer chain and 25.5 §3 set for the five boundaries.

9. The Waveform

Four readings.

In the first fetch sw_wr_len precedes sw_wr_own. That is the whole correctness requirement of §5, and it is one cycle wide in the trace.

In the second, desc_own_hw is already high at cycle 6 and sw_wr_len does not land until cycle 8. The engine fetches at cycle 7 — legally, since it owns the entry — and captures the previous occupant's length.

xfer_active looks identical in both transfers. The engine is not malfunctioning; it is faithfully executing a descriptor that was handed to it too early. Nothing on the datapath distinguishes the two, which is why the fault survives datapath-level review.

And len_stale is a derived signal, not a natural one. It exists because §10 builds it. Without it this trace shows two successful transfers — and that is exactly what a memory dump taken afterwards would also show, since by cycle 8 the descriptor in memory is correct (§5).

10. RTL — The Ownership and Conservation Instruments

Block 1 — the package: descriptor shape, ownership encoding, and the fault taxonomy.

package dma_dbg_pkg;
 
  // Which handoff in Figure 1 failed.  Ordered as the cycle proceeds, so
  // "first failing handoff" is a priority select over the four.
  typedef enum logic [2:0] {
    DMA_OK          = 3'd0,
    DMA_ORDER       = 3'd1,   // payload read before it was written (§5)
    DMA_BYTES       = 3'd2,   // bytes moved != bytes described (§4 law 1)
    DMA_STATUS      = 3'd3,   // signalled zero or twice     (§4 law 2)
    DMA_OWNERSHIP   = 3'd4,   // written while hardware-owned (§4 law 3)
    DMA_NO_RELEASE  = 3'd5    // never handed back — the stall (§3)
  } dma_fault_e;
 
  function automatic int unsigned gw(input int unsigned n);
    return (n <= 1) ? 1 : $clog2(n);
  endfunction
 
  // Ownership is one bit with an explicit polarity constant rather than a
  // bare 1'b1, because a polarity inversion is a silent, total failure and
  // naming it makes the inversion reviewable (mutation 3).
  localparam logic OWN_SW = 1'b0;
  localparam logic OWN_HW = 1'b1;
 
  typedef struct packed {
    logic [31:0] length;      // bytes this descriptor describes
    logic [63:0] address;     // host buffer address
    logic [15:0] flags;
    logic        owner;       // OWN_SW or OWN_HW — written LAST by the producer
  } desc_t;
 
endpackage

Block 2 — the ownership tracker. The direct instrument for law 3, and the module bar_first_reject's counterpart from 25.5 §10: it records the first violation and holds it.

module dma_owner_track #(
  parameter int unsigned RING_DEPTH = 64
)(
  input  logic clk,
  input  logic rst_n,
  // observed writes, one port per agent
  input  logic                               sw_write,
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0] sw_write_idx,
  input  logic                               hw_write,
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0] hw_write_idx,
  // the ownership bit of every entry, as it currently stands
  input  logic [RING_DEPTH-1:0]              owner_bits,
  input  logic                               clear,
  output logic                               violation,
  output logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0] first_idx,
  output logic                               captured,
  output logic [31:0]                        violation_count
);
 
  import dma_dbg_pkg::*;
 
  logic sw_touched_hw_owned, hw_touched_sw_owned;
 
  always_comb begin
    // The exclusivity law of 23.6 §4, stated as two symmetric checks.
    // Checking only one direction is mutation 8, and it passes on a design
    // where hardware writes an entry it has already released.
    sw_touched_hw_owned = sw_write && (owner_bits[sw_write_idx] == OWN_HW);
    hw_touched_sw_owned = hw_write && (owner_bits[hw_write_idx] == OWN_SW);
    violation           = sw_touched_hw_owned || hw_touched_sw_owned;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      captured <= 1'b0; first_idx <= '0; violation_count <= '0;
    end else if (clear) begin
      captured <= 1'b0; first_idx <= '0; violation_count <= '0;
    end else if (violation) begin
      if (!captured) begin
        captured  <= 1'b1;
        first_idx <= sw_touched_hw_owned ? sw_write_idx : hw_write_idx;
      end
      if (violation_count != 32'hFFFF_FFFF)
        violation_count <= violation_count + 32'd1;
    end
  end
 
endmodule

Block 3 — the byte conservation counter, two-sided. §4 law 1, and the reason the deficit is a signed quantity.

module dma_byte_conserve (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        desc_accepted,      // a descriptor entered the engine
  input  logic [31:0] desc_length,
  input  logic        beat_valid,         // one data beat moved
  input  logic [15:0] beat_bytes,
  input  logic        clear,
  output logic [63:0] described,
  output logic [63:0] moved,
  output logic signed [64:0] deficit,     // described - moved, SIGNED
  output logic        overrun,            // moved > described  <-- the fatal case
  output logic        shortfall           // moved < described
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      described <= '0; moved <= '0;
    end else begin
      if (desc_accepted) described <= described + 64'(desc_length);
      if (beat_valid)    moved     <= moved     + 64'(beat_bytes);
    end
  end
 
  // Signed, and reported as two separate flags.  A one-sided check written
  // as (moved <= described) reports SUCCESS on the run that measured a
  // deficit of -38,080 in §12 — an engine writing past a buffer end.
  always_comb begin
    deficit   = $signed({1'b0, described}) - $signed({1'b0, moved});
    overrun   = (moved > described);
    shortfall = (moved < described);
  end
 
endmodule

Block 4 — the exactly-once retirement checker. §4 law 2 and §7, per job rather than in aggregate.

module dma_retire_once #(
  parameter int unsigned NJOB_TRACK = 256   // job IDs tracked concurrently
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                    job_submit,
  input  logic [dma_dbg_pkg::gw(NJOB_TRACK)-1:0]  submit_id,
  input  logic                                    job_status,
  input  logic [dma_dbg_pkg::gw(NJOB_TRACK)-1:0]  status_id,
  input  logic                                    clear,
  output logic                                    dup_status,
  output logic                                    orphan_status,
  output logic [31:0]                             dup_count,
  output logic [31:0]                             orphan_count,
  output logic [31:0]                             outstanding
);
 
  import dma_dbg_pkg::*;
 
  // Two bits per tracked job: submitted, and already signalled.  Counting
  // pulses in aggregate cannot distinguish "1,797 lost" from "1,797
  // duplicated" — §7 measured both at the same rate on the same traffic.
  logic [NJOB_TRACK-1:0] submitted, signalled;
 
  always_comb begin
    dup_status    = job_status && submitted[status_id] && signalled[status_id];
    orphan_status = job_status && !submitted[status_id];
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      submitted <= '0; signalled <= '0;
      dup_count <= '0; orphan_count <= '0; outstanding <= '0;
    end else begin
      if (job_submit) begin
        submitted[submit_id] <= 1'b1;
        signalled[submit_id] <= 1'b0;
        outstanding          <= outstanding + 32'd1;
      end
      if (job_status) begin
        if (dup_count    != 32'hFFFF_FFFF && dup_status)    dup_count    <= dup_count + 32'd1;
        if (orphan_count != 32'hFFFF_FFFF && orphan_status) orphan_count <= orphan_count + 32'd1;
        if (submitted[status_id] && !signalled[status_id]) begin
          signalled[status_id] <= 1'b1;
          outstanding          <= outstanding - 32'd1;
        end
      end
    end
  end
 
endmodule

Block 5 — the ordering-window detector. §5 law 4, the only law that measures a timing relationship and the only one that cannot be checked after the fact.

module dma_order_check #(
  parameter int unsigned RING_DEPTH = 64
)(
  input  logic clk,
  input  logic rst_n,
  // Per-entry write-completion tracking, driven by the producer's bus.
  input  logic                                     wr_len,
  input  logic                                     wr_addr,
  input  logic                                     wr_own,
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]   wr_idx,
  // The consumer's fetch.
  input  logic                                     fetch,
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]   fetch_idx,
  input  logic                                     clear,
  output logic                                     order_violation,
  output logic                                     stale_fetch,
  output logic [31:0]                              stale_count
);
 
  import dma_dbg_pkg::*;
 
  // For each entry: have the payload fields been written since the last
  // time this slot was handed to hardware?
  logic [RING_DEPTH-1:0] len_written, addr_written, hw_owned;
 
  always_comb begin
    // The producer-side fault: ownership handed over before every payload
    // field landed.  This is the fault, and it is detectable at the write.
    order_violation = wr_own && !(len_written[wr_idx] && addr_written[wr_idx]);
    // The consumer-side consequence: a fetch of an entry whose payload was
    // incomplete when ownership transferred.  §12 measured 411 of these.
    stale_fetch     = fetch && hw_owned[fetch_idx] &&
                      !(len_written[fetch_idx] && addr_written[fetch_idx]);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      len_written <= '0; addr_written <= '0; hw_owned <= '0; stale_count <= '0;
    end else begin
      if (wr_len)  len_written[wr_idx]  <= 1'b1;
      if (wr_addr) addr_written[wr_idx] <= 1'b1;
      if (wr_own) begin
        hw_owned[wr_idx] <= 1'b1;
        // Clear the payload-written marks for the NEXT occupancy of this
        // slot, so the check is per-occupancy rather than per-lifetime.
        len_written[wr_idx]  <= 1'b0;
        addr_written[wr_idx] <= 1'b0;
      end
      if (stale_fetch && stale_count != 32'hFFFF_FFFF)
        stale_count <= stale_count + 32'd1;
    end
  end
 
endmodule

Block 6 — ring occupancy with an unambiguous full/empty discriminator. §6, and the producer gate that P14 asserts.

module dma_ring_occupancy #(
  parameter int unsigned RING_DEPTH = 64
)(
  input  logic clk,
  input  logic rst_n,
  input  logic push,                                    // producer posts
  input  logic pop,                                     // consumer releases
  input  logic [RING_DEPTH-1:0] owner_bits,
  output logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0] head,
  output logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0] tail,
  output logic full,
  output logic empty,
  output logic push_illegal                             // §6's fatal case
);
 
  import dma_dbg_pkg::*;
 
  // The discriminator used here is the per-entry ownership bit: the slot at
  // `tail` is available iff software owns it.  A wrap bit or a
  // keep-one-empty scheme is equally valid; what matters is that SOME
  // discriminator exists and the producer is gated on it (§6).
  always_comb begin
    full         = (owner_bits[tail] == OWN_HW);
    empty        = (owner_bits[head] == OWN_SW);
    // A push into a hardware-owned slot overwrites unconsumed work.
    // §12 measured 12,464 of these, and the same 12,464 jobs were also
    // never signalled — the correspondence §15 case 4 keys on.
    push_illegal = push && full;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      head <= '0; tail <= '0;
    end else begin
      if (push && !full) tail <= (tail == gw(RING_DEPTH)'(RING_DEPTH-1)) ? '0 : tail + 1'b1;
      if (pop  && !empty) head <= (head == gw(RING_DEPTH)'(RING_DEPTH-1)) ? '0 : head + 1'b1;
    end
  end
 
endmodule

Block 7 — the stall locator. §3's observation that the freeze point is not the fault point, turned into an instrument that records both.

module dma_stall_locate #(
  parameter int unsigned RING_DEPTH  = 64,
  parameter int unsigned STALL_LIMIT = 32'd100_000   // DEBUG HEURISTIC
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                    any_progress,   // any beat, any retire
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]  head,
  input  logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]  tail,
  input  logic                                    clear,
  output logic                                    stalled,
  output logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]  stall_head,
  output logic [dma_dbg_pkg::gw(RING_DEPTH)-1:0]  stall_tail,
  output logic [31:0]                             idle_cycles
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      idle_cycles <= '0; stalled <= 1'b0; stall_head <= '0; stall_tail <= '0;
    end else if (any_progress) begin
      idle_cycles <= '0;
      stalled     <= 1'b0;
    end else begin
      if (idle_cycles != 32'hFFFF_FFFF) idle_cycles <= idle_cycles + 32'd1;
      // Capture the pointers at the MOMENT the threshold is crossed, once.
      // Reading them later shows where the ring settled, not where it stopped.
      if (idle_cycles == STALL_LIMIT && !stalled) begin
        stalled    <= 1'b1;
        stall_head <= head;
        stall_tail <= tail;
      end
    end
  end
 
endmodule

Block 8 — the aggregate report. The four laws in one readable structure, because §4's separation argument only pays off if all four are read together.

module dma_law_report #(
  parameter int unsigned RING_DEPTH = 64
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        overrun,
  input  logic        shortfall,
  input  logic        dup_status,
  input  logic        orphan_status,
  input  logic        owner_violation,
  input  logic        stale_fetch,
  input  logic        stalled,
  output dma_dbg_pkg::dma_fault_e first_fault,
  output logic        captured
);
 
  import dma_dbg_pkg::*;
 
  dma_fault_e this_fault;
 
  // Priority follows Figure 1's cycle order, so the reported fault is the
  // EARLIEST handoff that failed.  A later handoff failing is very often a
  // consequence of an earlier one — §12's ring-full row produced both a
  // clobber and a missing status from one defect.
  always_comb begin
    if      (stale_fetch)                  this_fault = DMA_ORDER;
    else if (overrun || shortfall)         this_fault = DMA_BYTES;
    else if (dup_status || orphan_status)  this_fault = DMA_STATUS;
    else if (owner_violation)              this_fault = DMA_OWNERSHIP;
    else if (stalled)                      this_fault = DMA_NO_RELEASE;
    else                                   this_fault = DMA_OK;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      first_fault <= DMA_OK; captured <= 1'b0;
    end else if (!captured && (this_fault != DMA_OK)) begin
      first_fault <= this_fault; captured <= 1'b1;
    end
  end
 
endmodule

11. Assertions

Ownership properties — §4 law 3, 23.6 §4's law made checkable.

// P1 — software never writes a hardware-owned entry.
property p1_sw_respects_hw;
  @(posedge clk) disable iff (!rst_n)
    sw_write |-> (owner_bits[sw_write_idx] == dma_dbg_pkg::OWN_SW);
endproperty
a_p1: assert property (p1_sw_respects_hw);
 
// P2 — hardware never writes a software-owned entry.  The symmetric half;
// checking only P1 is mutation 8 and it passes on a real defect.
property p2_hw_respects_sw;
  @(posedge clk) disable iff (!rst_n)
    hw_write |-> (owner_bits[hw_write_idx] == dma_dbg_pkg::OWN_HW);
endproperty
a_p2: assert property (p2_hw_respects_sw);
 
// P3 — ownership changes only at a handoff, never spontaneously.
property p3_owner_stable;
  @(posedge clk) disable iff (!rst_n)
    (!wr_own && !hw_release) |=> $stable(owner_bits);
endproperty
a_p3: assert property (p3_owner_stable);
 
// P4 — an entry handed to hardware is eventually handed back.  This is the
// stall (§3), and it is the only liveness property in the chapter.
property p4_eventual_release;
  @(posedge clk) disable iff (!rst_n)
    (wr_own) |-> s_eventually (owner_bits[wr_idx] == dma_dbg_pkg::OWN_SW);
endproperty
a_p4: assert property (p4_eventual_release);

Ordering properties — §5 law 4.

// P5 — every payload field is written before ownership transfers.  This is
// the producer-side obligation, and it is where the fault actually lives.
property p5_payload_before_ownership;
  @(posedge clk) disable iff (!rst_n)
    wr_own |-> (len_written[wr_idx] && addr_written[wr_idx]);
endproperty
a_p5: assert property (p5_payload_before_ownership);
 
// P6 — a fetched entry has complete payload fields.  The consumer-side
// consequence, stated separately because a design can violate P5 in a
// window narrow enough that P6 rarely fires (§5: 411 in 89,625).
property p6_fetch_sees_complete;
  @(posedge clk) disable iff (!rst_n)
    fetch |-> !stale_fetch;
endproperty
a_p6: assert property (p6_fetch_sees_complete);
 
// P7 — the per-occupancy marks are cleared at handoff, so the check is
// about THIS occupancy rather than any previous one (mutation 12).
property p7_marks_clear_on_handoff;
  @(posedge clk) disable iff (!rst_n)
    wr_own |=> (!len_written[$past(wr_idx)] && !addr_written[$past(wr_idx)]);
endproperty
a_p7: assert property (p7_marks_clear_on_handoff);

Conservation properties — §4 law 1, both directions.

// P8 — the engine never moves more bytes than were described.  This is the
// half a one-sided check omits, and §12 measured it at -38,080.
property p8_no_overrun;
  @(posedge clk) disable iff (!rst_n)
    !overrun;
endproperty
a_p8: assert property (p8_no_overrun);
 
// P9 — at quiescence, described equals moved exactly.  Stated at
// quiescence because in flight work legitimately unbalances the counters.
property p9_conserve_at_idle;
  @(posedge clk) disable iff (!rst_n)
    (ring_empty && !xfer_active) |-> (described == moved);
endproperty
a_p9: assert property (p9_conserve_at_idle);
 
// P10 — a descriptor's beats sum to its length.  Per job, which is what
// catches a dropped final partial beat (§12: 2,872,352 bytes).
property p10_per_job_bytes;
  @(posedge clk) disable iff (!rst_n)
    job_done |-> (job_bytes_moved == job_desc_length);
endproperty
a_p10: assert property (p10_per_job_bytes);
 
// P11 — a zero-length descriptor moves no beats.  The degenerate case,
// which a beat-count-based engine frequently gets wrong by one.
property p11_zero_length_no_beats;
  @(posedge clk) disable iff (!rst_n)
    (desc_accepted && (desc_length == 32'd0)) |-> !beat_valid;
endproperty
a_p11: assert property (p11_zero_length_no_beats);
 
// P12 — the final beat of a transfer may be partial; it must not be dropped.
property p12_final_partial_beat;
  @(posedge clk) disable iff (!rst_n)
    (job_last_beat && (job_residue != 16'd0)) |-> (beat_valid && (beat_bytes == job_residue));
endproperty
a_p12: assert property (p12_final_partial_beat);

Ring properties — §6.

// P13 — full and empty are never simultaneously true.
property p13_full_empty_exclusive;
  @(posedge clk) disable iff (!rst_n)
    !(full && empty);
endproperty
a_p13: assert property (p13_full_empty_exclusive);
 
// P14 — the producer never pushes into a full ring.  §6's fatal case, and
// the property that catches the fault whose throughput looks BETTER.
property p14_no_push_when_full;
  @(posedge clk) disable iff (!rst_n)
    push |-> !full;
endproperty
a_p14: assert property (p14_no_push_when_full);
 
// P15 — the consumer never pops an empty ring.
property p15_no_pop_when_empty;
  @(posedge clk) disable iff (!rst_n)
    pop |-> !empty;
endproperty
a_p15: assert property (p15_no_pop_when_empty);
 
// P16 — pointers advance by exactly one, and wrap correctly.
property p16_pointer_step;
  @(posedge clk) disable iff (!rst_n)
    (push && !full) |=> (tail == (($past(tail) == RING_DEPTH-1) ? '0 : $past(tail) + 1));
endproperty
a_p16: assert property (p16_pointer_step);

Retirement properties — §4 law 2 and §7.

// P17 — the status write follows the data it describes.  A Memory Write is
// Posted (§1), so this ordering is the ONLY thing that makes the status
// meaningful (13.4 owns the rules that make it hold on the link).
property p17_status_after_data;
  @(posedge clk) disable iff (!rst_n)
    job_status |-> $past(job_last_beat_done);
endproperty
a_p17: assert property (p17_status_after_data);
 
// P18 — no job is signalled twice.
property p18_no_duplicate_status;
  @(posedge clk) disable iff (!rst_n)
    !dup_status;
endproperty
a_p18: assert property (p18_no_duplicate_status);
 
// P19 — no status arrives for a job never submitted.
property p19_no_orphan_status;
  @(posedge clk) disable iff (!rst_n)
    !orphan_status;
endproperty
a_p19: assert property (p19_no_orphan_status);
 
// P20 — every submitted job is eventually signalled.  P18 and P20 together
// are "exactly once"; either alone is satisfied by the opposite fault.
property p20_eventual_status;
  @(posedge clk) disable iff (!rst_n)
    job_submit |-> s_eventually (signalled[$past(submit_id)]);
endproperty
a_p20: assert property (p20_eventual_status);
 
// P21 — outstanding never goes negative (it is unsigned, so: never wraps).
property p21_outstanding_no_wrap;
  @(posedge clk) disable iff (!rst_n)
    (outstanding == 32'd0) |-> !(job_status && !job_submit);
endproperty
a_p21: assert property (p21_outstanding_no_wrap);

Instrument-integrity properties — the debug hardware must itself be trustworthy.

// P22 — the first fault is sticky.
property p22_first_fault_sticky;
  @(posedge clk) disable iff (!rst_n)
    captured |=> (captured && $stable(first_fault));
endproperty
a_p22: assert property (p22_first_fault_sticky);
 
// P23 — stall pointers are captured at the threshold crossing, once.
// Reading them later shows where the ring settled, not where it stopped.
property p23_stall_capture_once;
  @(posedge clk) disable iff (!rst_n)
    stalled |=> ($stable(stall_head) && $stable(stall_tail));
endproperty
a_p23: assert property (p23_stall_capture_once);
 
// P24 — counters saturate rather than wrapping.
property p24_counters_saturate;
  @(posedge clk) disable iff (!rst_n)
    (violation_count == 32'hFFFF_FFFF) |=> (violation_count == 32'hFFFF_FFFF);
endproperty
a_p24: assert property (p24_counters_saturate);
 
// P25 — progress clears the idle counter, so `stalled` means "no progress",
// not "no submissions".  Mutation 27 conflates them.
property p25_progress_clears_idle;
  @(posedge clk) disable iff (!rst_n)
    any_progress |=> (idle_cycles == 32'd0);
endproperty
a_p25: assert property (p25_progress_clears_idle);

Cover — the anti-vacuity set.

// P26 — the four laws of §4 are evaluated together, not independently.  A
// design reporting one law clean while another is violated is reporting a
// partial view, and §4's separation argument only holds if all four are read.
property p26_laws_jointly_clean;
  @(posedge clk) disable iff (!rst_n)
    (ring_empty && !xfer_active) |->
      (!overrun && !shortfall && !dup_status && !orphan_status &&
       !owner_violation && !stale_fetch);
endproperty
a_p26: assert property (p26_laws_jointly_clean);
 
// P26's covers — the rare events must actually occur, or the properties above are
// satisfied without ever evaluating.  §12 measured the ordering window at
// 411 events in 89,625 jobs; c3 is the cover that proves it was exercised.
c1_ring_full:      cover property (@(posedge clk) disable iff (!rst_n) full);
c2_ring_empty:     cover property (@(posedge clk) disable iff (!rst_n) empty);
c3_order_window:   cover property (@(posedge clk) disable iff (!rst_n)
                     wr_own && !(len_written[wr_idx] && addr_written[wr_idx]));
c4_partial_beat:   cover property (@(posedge clk) disable iff (!rst_n)
                     job_last_beat && (job_residue != 16'd0));
c5_zero_length:    cover property (@(posedge clk) disable iff (!rst_n)
                     desc_accepted && (desc_length == 32'd0));
c6_wrap:           cover property (@(posedge clk) disable iff (!rst_n)
                     push && (tail == RING_DEPTH-1));
c7_concurrent:     cover property (@(posedge clk) disable iff (!rst_n) push && pop);

12. Measured Behaviour

200,000 steps, 16-entry ring, drained before measurement:

Four readings.

The healthy row is exactly zero on all five fault metrics, and the model asserts this before producing any other row. The second defect above was caught precisely because that assertion failed.

The minus sign is the most important character in the table. A stale length read from a previous, longer occupant caused the engine to move 38,080 bytes more than described. Every one of those bytes is a write past the end of a buffer, and a conservation check written moved <= described reports the run as clean.

The never-release row shows the stall's misleading geometry. 16 posted, 1 retired — the engine stopped on the first entry it failed to release, and everything downstream is a consequence. §15 case 1 is about not mistaking the freeze point for the fault point.

And the ring-full row posts 11% more jobs than the healthy one. A throughput measurement, alone, reports this defect as an improvement (§6).

13. Executable Counterexamples

Counterexample A — the one-sided conservation check (violates P8).

// A byte-conservation monitor that only ever reports a shortfall.  This is
// the single most common form of the check in real designs, and §12's
// stale-length run passes it while writing 38,080 bytes past buffer ends.
module ce_a_one_sided_conserve (
  input  logic        clk, rst_n,
  input  logic        desc_accepted, input logic [31:0] desc_length,
  input  logic        beat_valid,    input logic [15:0] beat_bytes,
  output logic        conserved
);
  logic [63:0] described, moved;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin described <= '0; moved <= '0; end
    else begin
      if (desc_accepted) described <= described + 64'(desc_length);
      if (beat_valid)    moved     <= moved     + 64'(beat_bytes);
    end
  end
  assign conserved = (moved <= described);          // <-- one-sided
endmodule
 
// Failing stimulus: a descriptor slot whose previous occupant had
// length = 4096 is handed over with ownership written BEFORE length, and
// the engine fetches in that window while the new length is 64.
// Golden: DMA_ORDER, and 4032 bytes moved beyond the described extent.
// This:   conserved = 1 throughout.  The monitor reports success.
// P8 fails (overrun is high).  P5 and P6 fail at the fetch.
// Observable consequence: 4032 bytes written past the end of a 64-byte
// host buffer, with every driver-visible counter reading correct.

Counterexample B — the aggregate retirement counter (violates P18 and P20).

// Completion tracking by a single outstanding counter rather than per job.
// It cannot distinguish a lost status from a duplicated one — §12 measured
// both at exactly 1,797 on the same traffic.
module ce_b_aggregate_retire (
  input  logic clk, rst_n,
  input  logic job_submit, input logic job_status,
  output logic [31:0] outstanding, output logic balanced
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) outstanding <= '0;
    else begin
      case ({job_submit, job_status})
        2'b10: outstanding <= outstanding + 32'd1;
        2'b01: outstanding <= outstanding - 32'd1;   // <-- no identity check
        default: ;
      endcase
    end
  end
  assign balanced = (outstanding == 32'd0);
endmodule
 
// Failing stimulus: job 7 is signalled twice and job 9 is never signalled.
// Golden: dup_status on job 7, and job 9 outstanding forever.
// This:   outstanding returns to 0.  balanced = 1.  Both faults cancel.
// P18 fails (a duplicate occurred).  P20 fails (job 9 never signalled).
// Observable consequence: software frees job 9's buffer on job 7's second
// completion, while the engine may still be writing to it — a use-after-free
// with a DMA engine as the writer (§7).

Both counterexamples are instruments rather than engines, and that is the point. Neither design under observation is wrong in these examples — the monitors are. A DMA that is being watched by a one-sided conservation check and an aggregate completion counter is, for debugging purposes, not being watched at all.

14. Verification — Mutations

Thirty-four mutations. Every "Caught by" entry names a property from §11.

Mutations 13 and 28 are the pair worth study. Neither breaks a DMA engine — both break the instrument watching it, and both are the default form of the check in practice. §13's counterexamples are exactly these two, and they are in the table to make the point that a mutation campaign must mutate the monitors as well as the design.

Mutations 33 and 34 break the testbench. Their symptom is that everything passes, and they are the reason P26 exists (24.4 §5).

15. Debugging

Case 1 — the ring stopped and the head pointer sits at descriptor N.

Do not start at descriptor N. The engine stopped on the first entry it failed to complete, and the pointer you are reading is where it settled. §12 measured 16 posted against 1 retired — the freeze is 15 entries downstream of the defect. Read the stall-capture registers (§10 Block 7), which latch head and tail at the moment the idle threshold was crossed rather than when you read them (P23), and inspect the entry at the captured head. Confidence: high, and this is the single most common process error in DMA debugging.

Case 2 — throughput is normal, driver-visible counters are normal, and data is intermittently wrong.

Corrupting class. Every counter-based instrument is useless here by construction. Read the four conservation laws (§4). §12's signatures separate the candidates completely: a nonzero deficit with zero stale reads is a dropped beat; a negative deficit with nonzero stale reads is the ordering window; equal clobber and missing-status counts is the ring-full test. Confidence: high — no two faults in §12 share a signature across the four laws.

Case 3 — the byte-conservation check reports clean and buffers are being overrun.

Check whether the check is two-sided (§4 law 1, counterexample A, mutation 13). A monitor written moved <= described reports success on precisely the fault that overruns. §12 measured a deficit of −38,080 on such a run. Read the monitor's source before reading any more of the engine's. Confidence: high; this is a five-minute test that resolves the case outright.

Case 4 — the clobber count and the never-signalled count are equal.

That equality is the fingerprint of a broken ring-full test (§6, §12: 12,464 and 12,464). One defect produces both symptoms — descriptors overwritten before consumption are also never completed. Do not debug them as two faults. Confirm by checking whether the producer is gated on the slot's ownership at all (P14). Confidence: high when the counts match exactly.

Case 5 — the DMA is faster than it was before the "fix", and something is subtly wrong.

Suspect the ring-full test immediately. §12 measured the broken configuration posting 99,869 jobs against a healthy 89,761 — 11% faster, because it never blocks; it overwrites. A throughput improvement with no design change to the datapath is a red flag, not a result. Confidence: moderate to high; check P14 first, then law 3.

Case 6 — occasional transfers are the wrong length, and the descriptor in memory is correct.

The ordering window (§5). The memory dump is correct by the time you take it — every field has been written by then, which is why this fault survives post-mortem analysis indefinitely. §12 measured it at 411 events in 89,625 jobs. Only a live instrument sampling at the fetch can catch it (§10 Block 5). Confidence: high if the wrong lengths match the previous occupant of the same slot — check that correspondence explicitly, because it is nearly conclusive.

Case 7 — the driver reports more completions than it submitted.

Law 2, duplication side (§7, mutation 26). Treat this as urgent rather than as an accounting curiosity. A duplicated completion frees a buffer that may have been reissued, giving software a buffer the engine is still writing to. Read the per-job tracker (§10 Block 4); an aggregate counter cannot tell you which job duplicated (counterexample B). Confidence: high.

Case 8 — one job never completes; every other job is fine.

Law 2, loss side. The engine is healthy — a lost status pulse affects one job and leaves the pipeline running, which is why §12 shows 87,766 jobs still signalled correctly on that run. Look for a status path that can drop a pulse under backpressure rather than for anything wrong with the transfer. This case leaks a buffer; it does not corrupt one, and distinguishing it from case 7 is the entire value of per-job tracking.

Case 9 — the last few bytes of every transfer are stale.

The final partial beat (§4 law 1, mutations 16 and 17). The arithmetic is almost always a truncating division turning a length into a beat count. §12 measured a 2,872,352-byte deficit. Confirm with a directed transfer whose length is deliberately not a multiple of the beat width (cover c4), which mutation 34 shows a random testbench may never generate. Confidence: high.

Case 10 — everything passes and you do not trust it.

Read the covers before the assertions (§11 P26, mutations 33 and 34). A ring exercised below half occupancy never reaches full, never wraps, and never opens the ordering window — so P14, P16 and P5 are all vacuously satisfied. §12's ordering fault occurs 411 times in 89,625 jobs; a short directed test can miss it entirely while reporting complete assertion coverage. This is 24.4 §5's rule and it is the right first move whenever a clean result is surprising.

16. Misconceptions

"The DMA is stuck, so something stopped." Five of the six faults measured in §12 do not stop the engine (§3). The instinct to look for a freeze misses most DMA bugs.

"The descriptor in memory is correct, so the descriptor was correct." Not for the ordering window (§5). By the time you dump memory, every field has been written; the fault was a read that happened earlier. 412 events out of 89,625 leave no trace in memory at all.

"Bytes moved never exceeds bytes described — that would be nonsense." §12 measured −38,080. A stale longer length makes it not only possible but routine, and it is the most damaging fault in the chapter (§4 law 1).

"Completion counting catches lost and duplicated statuses." In aggregate, they cancel (counterexample B). §12 measured both at exactly 1,797 on the same traffic, and an aggregate counter returns to zero in both cases. Per-job tracking is the only form that distinguishes them.

"A duplicated completion is harmless — the job did finish." It tells software a buffer is free when it may have been reissued (§7). The result is a use-after-free with a DMA engine as the writer.

"Higher throughput after a ring change means the change was good." §12's broken ring-full test posted 11% more jobs than the correct one, by overwriting unconsumed descriptors (§6). Throughput alone cannot distinguish a faster ring from one that skips work.

"PCIe ordering rules protect the descriptor handoff." They govern transactions on the link (13.4). They do not substitute for a producer writing its own fields in the right order (§5) — that obligation is the producer's and no link rule discharges it.

"The assertions all pass." Check whether they evaluated (§11 P26, §15 case 10). A ring run at low occupancy satisfies most of this chapter's properties without ever exercising one.

17. Understanding Check

Q1. A DMA has stopped and the head pointer sits at descriptor N. Why is descriptor N probably not the problem?

Because the pointer shows where the ring settled, not where it stopped (§15 case 1). The engine halted on the first entry it failed to complete and everything after that is a consequence — §12's never-release run posted 16 jobs and retired 1, putting the freeze 15 entries downstream of the defect. The instrument that answers this is the stall-capture register (§10 Block 7, P23), which latches head and tail at the moment the idle threshold was crossed rather than when software reads them.

Q2. Your byte-conservation monitor reports clean and buffers are being overrun. What is wrong?

The check is one-sided (§4 law 1, mutation 13, counterexample A). Written moved <= described, it reports success on exactly the fault that overruns. §12 measured a deficit of −38,080 on a stale-length run — the engine moved more than the descriptors described, every excess byte a write past a buffer end. Only moved == described, reported as two separate flags, catches it, which is why P8 asserts !overrun independently of any shortfall check.

Q3. Occasional transfers use the wrong length, and the descriptor in memory is correct every time you look. Explain.

The ordering window (§5). The producer wrote the ownership bit before the length, the engine fetched inside that window, and captured the previous occupant's length. By the time you dump memory the length has been written, so memory is correct — the fault was a read that happened earlier and left no residue. §12 measured 411 such fetches in 89,625 jobs. The confirming correspondence is that the wrong lengths match the previous occupant of the same slot, and the only instrument that catches it live is a fetch-time check (§10 Block 5, P6).

Q4. Lost and duplicated completions produce opposite symptoms. Why does an aggregate outstanding counter see neither?

Because they cancel (§7, counterexample B). §12 measured 1,797 lost and 1,797 duplicated on the same traffic; a single counter incremented on submit and decremented on status returns to zero in both cases and reports "balanced". Exactly-once is a per-job property, and it needs two bits per job — submitted and signalled — which is what P18 and P20 together assert and what a single counter structurally cannot express.

Q5. Why is a duplicated completion more dangerous than a lost one?

Because loss leaks a buffer and duplication hands out a live one (§7). A lost status leaves one job outstanding forever — the engine is healthy, the rest of the traffic is unaffected, and §12 shows 87,766 jobs still completing normally on that run. A duplicate tells software a buffer is free when the slot may have been reissued in the interval, so software hands that buffer to something else while the engine is still writing to it. That is a use-after-free with a DMA engine as the writer.

Q6. A ring change makes the DMA 11% faster. Why is that suspicious?

Because a broken ring-full test never blocks — it overwrites (§6, §15 case 5). §12 measured the broken configuration posting 99,869 jobs against a healthy 89,761, along with 12,464 clobbered descriptors and 12,464 jobs never signalled. A throughput gain with no datapath change is a signal to check law 3, not a result to report. P14 is the property; the equal clobber and missing counts are the fingerprint.

Q7. Which of the four conservation laws cannot be checked from a memory dump, and why does that matter?

Law 4, the ordering window (§4, §5). The other three are counts — bytes, status pulses, ownership writes — and can be reconciled after the fact. Law 4 measures a timing relationship between two writes and a read, and all the evidence is gone once the writes complete. That is why it needs live hardware (§10 Block 5) and why the fault survives post-mortem analysis indefinitely (§15 case 6).

Q8. Every assertion in your DMA testbench passes. What do you check before believing it?

The covers (§11 P26, §15 case 10, mutations 33 and 34). A ring exercised below half occupancy never reaches full, never wraps its pointers, and never opens the ordering window — so P14, P16, P5 and P6 are satisfied vacuously, having never evaluated a consequent. §12's ordering fault occurred 411 times in 89,625 jobs, which a short directed test can easily miss entirely. A property that never evaluated is not a property that held, and c1, c3, c4 and c6 are what distinguish the two.

18. Module 25 So Far

Six chapters, and the object of study keeps moving inward.

25.5 and this chapter are a pair, and the pairing is worth naming. Both are about ownership; they differ in whether it moves. A BAR is a static claim — programmed once, checked millions of times, and wrong in the same way every time. A descriptor is a claim that changes hands thousands of times a second, which adds the two failure modes a static claim cannot have: a handoff that does not complete, and a handoff that completes twice.

And 25.5 §18's generalisation held. It predicted that the layer which looks redundant is the one converting silent corruption into visible rejection, and §4's law 1 is exactly that layer for DMA: the two-sided byte check is redundant right up until a stale length makes the engine overrun, at which point it is the only instrument in the system that notices.

One more shape recurs, and it is the chapter's most transferable result. §13's counterexamples are both monitors, not engines. A one-sided conservation check and an aggregate completion counter are the default forms of those instruments, and both are blind to the exact fault they exist to catch. Mutating the design is half a verification plan; the other half is mutating what watches it.

25.7 takes the next question. This chapter's engine originated requests and this chapter never asked what happens when one is never answered. A non-posted request that never completes is its own failure class — and the timer that reports it is very often not the timer that should have.