Skip to content

PCIe · Module 26

Network Adapters — When a PCIe Limit Becomes Packet Loss

A NIC cannot tell the wire to wait. Slowing only the host path dropped 23,495 frames with every PCIe counter clean — and one ownership oracle caught a fault the other was structurally blind to.

Chapter 26.3 described a device that chose when to work — a command waits in a queue until the controller fetches it. This chapter removes that.

Packets arrive whether or not the adapter is ready. A NIC bridges a network that does not wait and a host memory path that sometimes does, and the consequence of a slow PCIe path is not a longer queue — it is frames discarded on the wire, reported by the network stack, with every PCIe counter clean.

1. Sources, Scope, and the SmartNIC Boundary

2. Two Independent Paths, Neither of Which Waits

A NIC is two data paths that share a Link, a buffer pool and an interrupt mechanism, and otherwise run independently.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
RECEIVE                                TRANSMIT
network ingress                        host writes descriptor + data
  → packet buffer                        → doorbell
  → PCIe DMA WRITE to host memory        → NIC DMA READ of the descriptor
  → status/completion published          → NIC DMA READ of the packet data
  → notification                         → packet buffer
                                         → network egress
                                         → status published → notification

The asymmetry that defines the chapter is in the first line of each.

On transmit, the host decides when work exists. If the NIC is slow, descriptors accumulate in a ring and the host eventually blocks. Nothing is lost.

On receive, the network decides. Frames arrive at whatever rate the far end sends them, and the adapter has a finite buffer. When the PCIe path cannot drain that buffer fast enough, frames are discarded — and §15 measured exactly that, with the MAC and the network unchanged in every row.

This is the chapter's central asymmetry, and it is why §5 exists as its own section: it is the one place in Module 26 where a PCIe-side limit produces a statistic in a completely different subsystem's counters.

3. Descriptor Rings, in Both Directions

The ownership discipline is 25.6 §2's, applied twice with the roles reversed.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
RX ring                                TX ring
software posts EMPTY buffers           software posts FULL buffers
hardware fills them                    hardware drains them
hardware returns them with status      hardware returns them with status
software consumes the data             software frees the memory

In both cases the law is the same: exactly one owner at every instant, and the handoff is a write to memory that must become visible after everything it describes.

Two ring-specific hazards recur, and §16 measures both.

A descriptor must not be recycled before its transfer completes. On RX that means the host buffer is still being written; returning the descriptor early tells software the data is ready when it is not. On TX it means the packet data is still being read, and freeing the buffer lets software overwrite a packet mid-transmission.

And a completion must return to the queue that issued it. With many queues, a completion credited to the wrong ring corrupts two rings at once — draining one and inflating the other. §16 measured 3,284 such events, and found that the obvious conservation check cannot see them at all.

4. The Block Diagram

A host with CPU, DRAM containing transmit and receive descriptor rings and packet buffers, and a root complex, connects over PCIe to a NIC. Inside the NIC a queue manager, DMA engines, receive and transmit packet buffers, packet processing and a MAC connect to the network. An interrupt path with coalescing returns to the host.Host CPURX ring + buffersTX ring + buffersRoot ComplexPCIe EndpointQueue managerRX DMATX DMARX packet bufferTX packet bufferPacket processing +MACNetworkInterrupt +coalescingdrainfillafter status12
Figure 1 — receive and transmit as separate flows across one Link. On receive the network fills a packet buffer which is drained by DMA writes into host memory. On transmit the adapter fetches descriptors and packet data by DMA read. Both paths publish status before any interrupt is raised, and the interrupt path is shared and coalesced.

One edge in the figure is the whole of §5. The mac → rxb fill arrow has no back-pressure toward the network; the rxb → rxd drain arrow depends entirely on the PCIe and host path. When the second is slower than the first, the buffer between them is the only slack in the system, and it is finite.

5. How a PCIe Limit Becomes Packet Loss

Nothing on the network side changed between these rows. Same offered load, same MAC, same frames. The drops are entirely a consequence of the host-facing path.

The reporting is what makes this hard to debug. The frames are discarded at the packet buffer, so the counter that increments is a receive drop in the network statistics — the same counter that increments for a genuine network overrun. Meanwhile every PCIe error counter reads zero, because nothing about PCIe failed: no error, no timeout, no credit deadlock. The Link was simply not fast enough, or the host memory path behind it was not.

Three candidate causes sit behind a slow drain, and separating them is §18 case 2:

  • the Link — insufficient bandwidth, or credit exhaustion (22.3);
  • the host memory path — the fabric and memory controller behind the Root Complex (26.1 §4);
  • the descriptor supply — software not posting free buffers fast enough, which starves the DMA engine regardless of how fast the Link is.

The third is the one that looks most like a PCIe problem and is not. A NIC with no free descriptors cannot write to host memory no matter how much bandwidth is available, and §9 Block 3 makes that state separately observable for exactly this reason.

What PCIe does not specify is the response. Whether the adapter drops, applies network-side flow control, or buffers more deeply is IMPLEMENTATION POLICY and depends on the network protocol in use. This chapter takes no position on it and claims only the causal chain: a PCIe-side limit can manifest as network-side loss.

6. The PCIe Cost of a Frame

A frame does not cost its own length in PCIe traffic. It costs more, and at small sizes it costs much more.

§14 measured it with normalized 16-byte descriptor and status structures (IMPLEMENTATION POLICY — real formats differ and are not reproduced here):

frame bytespayloaddescriptorstatusPCIe totalratio
64641616961.50×
12812816161601.25×
25625616162881.12×
51251216165441.06×
15181518161615501.02×
90009000161690321.00×

Two consequences.

Sizing a PCIe path from average network throughput understates it by up to 50%. A NIC saturating its wire with small frames generates half again as much PCIe traffic as the payload figure suggests, and that traffic is spread over many more transactions — which matters more than the byte count, because per-transaction overheads and Tag limits scale with count rather than size (22.4).

And frame-size mix changes the answer entirely. A test at 9000-byte frames and a deployment at 64-byte frames are different PCIe workloads at the same wire rate. A throughput measurement without a stated frame-size distribution is not a measurement22.1's discipline, applied here.

Real adapters may add per-packet metadata, split headers, or write status in batches, all of which change the fixed cost. This chapter models the structure, not any product's exact overhead.

7. The Instruments, Named

InstrumentAnswers
nic_queue_ctxper-queue ring state, isolated between queues
nic_rx_allocwhich free descriptor owns this arriving frame?
nic_rx_completepublish status, then allow notification
nic_tx_fetchdescriptor fetch and packet fetch, kept separate
nic_queue_arbiterfair service without cross-queue coupling
nic_buffer_accountbuffer occupancy, and whether a frame can be accepted
nic_vector_snapshotthe queue→vector mapping at event ownership
nic_coalesceinterrupt batching policy (device policy, not PCIe)
nic_countersis the limit the Link, the host, or descriptor supply?

8. Same-Cycle Audit

9. RTL — Rings, Buffers, and Notification

Block 1 — the package. COMPILE-TIME.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
package nic_pkg;
 
  typedef enum logic [0:0] { DIR_RX = 1'b0, DIR_TX = 1'b1 } ring_dir_e;
 
  // Why a frame was not delivered. Distinguishing these is the whole of §5:
  // "dropped" is a network statistic, and only the controller knows which of
  // three unrelated causes produced it.
  typedef enum logic [2:0] {
    DROP_NONE      = 3'd0,
    DROP_BUFFER    = 3'd1,   // packet buffer full — the PCIe path is behind
    DROP_NODESC    = 3'd2,   // no free host descriptor — software is behind
    DROP_QDISABLED = 3'd3,   // the queue is not enabled
    DROP_ERROR     = 3'd4    // network-side error; not a PCIe matter
  } drop_reason_e;
 
  function automatic int unsigned gw(input int unsigned n);
    return (n <= 1) ? 1 : $clog2(n);
  endfunction
 
  // A DMA request as the engine owns it: a SNAPSHOT taken when ownership
  // transfers. 25.6 §5 established why this is copied and not referenced.
  typedef struct packed {
    logic [15:0] queue_id;
    logic [15:0] desc_idx;
    ring_dir_e   dir;
    logic [63:0] host_addr;
    logic [15:0] bytes;
  } nic_xfer_t;
 
endpackage

Block 2 — per-queue context with isolation. SYNTHESIZABLE. §11's requirement.

Input owner: configuration writes and doorbells. Output owner: this module. Isolation: every queue's state is indexed; no field is shared, which is what makes a stalled queue unable to corrupt another. Reset: clears enables and pointers; the host re-posts descriptors.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_queue_ctx #(
  parameter int unsigned NQUEUE = 8,
  parameter int unsigned MAXRING = 4096
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                 cfg_valid,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       cfg_qid,
  input  logic                                 cfg_enable,
  input  logic [nic_pkg::gw(MAXRING+1)-1:0]    cfg_depth,
  input  logic                                 db_valid,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       db_qid,
  input  logic [31:0]                          db_tail,
  input  logic                                 head_adv,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       head_qid,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       q_sel,
  output logic                                 q_enabled,
  output logic                                 q_has_work,
  output logic [nic_pkg::gw(MAXRING)-1:0]      q_head, q_tail,
  output logic                                 qid_error
);
 
  import nic_pkg::*;
 
  logic [NQUEUE-1:0] en;
  logic [gw(MAXRING+1)-1:0] depth [NQUEUE];
  logic [gw(MAXRING)-1:0]   head  [NQUEUE], tail [NQUEUE];
 
  // Every piece of per-queue state is INDEXED, never shared. A single global
  // head/tail pair across queues is mutation 11, and its symptom is that one
  // stalled queue stops every other — the cross-talk §16's identity oracle
  // measures.
  assign qid_error = (db_valid && (db_qid >= gw(NQUEUE)'(NQUEUE))) ||
                     (cfg_valid && (cfg_qid >= gw(NQUEUE)'(NQUEUE)));
 
  assign q_enabled  = en[q_sel];
  assign q_head     = head[q_sel];
  assign q_tail     = tail[q_sel];
  assign q_has_work = en[q_sel] && (head[q_sel] != tail[q_sel]);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      en <= '0;
      for (int i = 0; i < NQUEUE; i++) begin
        depth[i] <= '0; head[i] <= '0; tail[i] <= '0;
      end
    end else begin
      if (cfg_valid && !qid_error) begin
        en[cfg_qid]    <= cfg_enable;
        depth[cfg_qid] <= cfg_depth;
        head[cfg_qid]  <= '0;
        tail[cfg_qid]  <= '0;
      end
      if (db_valid && !qid_error && en[db_qid] && (db_tail < 32'(depth[db_qid])))
        tail[db_qid] <= gw(MAXRING)'(db_tail);
      if (head_adv && en[head_qid])
        head[head_qid] <= (head[head_qid] == gw(MAXRING)'(depth[head_qid]-1))
                        ? '0 : head[head_qid] + 1'b1;
    end
  end
 
endmodule

Block 3 — buffer accounting and the drop decision. SYNTHESIZABLE. §5's mechanism, with the reason recorded.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_buffer_account #(
  parameter int unsigned BUFCAP = 256
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                   pkt_arrive,
  input  logic                                   q_enabled,
  input  logic                                   desc_available,
  input  logic                                   drain_done,      // a frame left for host memory
  output logic                                   accept,
  output nic_pkg::drop_reason_e                  drop_reason,
  output logic [nic_pkg::gw(BUFCAP+1)-1:0]       occupancy,
  output logic [31:0]                            c_drop_buffer, c_drop_nodesc, c_drop_qdis
);
 
  import nic_pkg::*;
 
  // Acceptance depends on BUFFER SPACE only. Descriptor availability is a
  // separate question answered later, because a frame with buffer space but
  // no descriptor can WAIT — dropping it is throwing away a frame the
  // adapter had room for (§8 audit A, mutation 14).
  always_comb begin
    accept      = pkt_arrive && q_enabled && (occupancy < gw(BUFCAP+1)'(BUFCAP));
    drop_reason = DROP_NONE;
    if (pkt_arrive && !q_enabled)                                   drop_reason = DROP_QDISABLED;
    else if (pkt_arrive && (occupancy >= gw(BUFCAP+1)'(BUFCAP)))    drop_reason = DROP_BUFFER;
  end
 
  // The three drop counters are SEPARATE. §5's whole point is that "receive
  // drops" is one network statistic covering three unrelated causes: the
  // PCIe path is behind (DROP_BUFFER), software is behind (DROP_NODESC), or
  // the queue is down. Merging them makes §18 case 2 unanswerable.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      occupancy <= '0; c_drop_buffer <= '0; c_drop_nodesc <= '0; c_drop_qdis <= '0;
    end else begin
      case ({accept, drain_done})
        2'b10: occupancy <= occupancy + 1'b1;
        2'b01: occupancy <= occupancy - 1'b1;
        default: ;
      endcase
      if (drop_reason == DROP_BUFFER    && c_drop_buffer != '1) c_drop_buffer <= c_drop_buffer + 1'b1;
      if (drop_reason == DROP_QDISABLED && c_drop_qdis   != '1) c_drop_qdis   <= c_drop_qdis + 1'b1;
      if (pkt_arrive && q_enabled && !desc_available && c_drop_nodesc != '1)
        c_drop_nodesc <= c_drop_nodesc + 1'b1;
    end
  end
 
endmodule

Block 4 — the RX descriptor allocator. SYNTHESIZABLE. §3's ownership discipline.

Lifetime: a descriptor is owned by hardware from allocation until its status is accepted, not until its last data beat.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_rx_alloc #(
  parameter int unsigned NQUEUE  = 8,
  parameter int unsigned NINFLT  = 32
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                 frame_ready,     // a buffered frame needs a home
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       frame_qid,
  input  logic [15:0]                          frame_bytes,
  input  logic                                 desc_avail,
  input  logic [15:0]                          desc_idx,
  input  logic [63:0]                          desc_host_addr,
  output logic                                 alloc_valid,
  output nic_pkg::nic_xfer_t                   alloc_xfer,
  input  logic                                 alloc_ready,
  input  logic                                 status_accepted, // the completion was published
  input  logic [nic_pkg::gw(NINFLT)-1:0]       status_slot,
  output logic                                 recycle_valid,
  output logic [15:0]                          recycle_idx,
  output logic [nic_pkg::gw(NINFLT+1)-1:0]     inflight
);
 
  import nic_pkg::*;
 
  logic [NINFLT-1:0] busy;
  logic [15:0]       held_idx [NINFLT];
  logic [gw(NINFLT)-1:0] alloc_slot;
 
  always_comb begin
    alloc_slot = '0;
    for (int i = NINFLT-1; i >= 0; i--) if (!busy[i]) alloc_slot = gw(NINFLT)'(i);
 
    alloc_valid = frame_ready && desc_avail && (busy != '1);
    alloc_xfer  = '{queue_id:  16'(frame_qid),
                    desc_idx:  desc_idx,
                    dir:       DIR_RX,
                    host_addr: desc_host_addr,
                    bytes:     frame_bytes};
 
    // The descriptor returns to software ONLY when its status has been
    // ACCEPTED — not when the last data beat moved. Recycling on the beat is
    // counterexample A: software sees a buffer marked ready while the write
    // to it may still be in flight (§8 audit B).
    recycle_valid = status_accepted && busy[status_slot];
    recycle_idx   = held_idx[status_slot];
 
    inflight = '0;
    for (int i = 0; i < NINFLT; i++) inflight += gw(NINFLT+1)'(busy[i]);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      busy <= '0;
      for (int i = 0; i < NINFLT; i++) held_idx[i] <= '0;
    end else begin
      if (alloc_valid && alloc_ready) begin
        busy[alloc_slot]     <= 1'b1;
        held_idx[alloc_slot] <= desc_idx;
      end
      if (recycle_valid) busy[status_slot] <= 1'b0;
    end
  end
 
endmodule

Block 5 — the TX fetch engine. SYNTHESIZABLE. Descriptor fetch and packet fetch kept separate.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_tx_fetch #(
  parameter int unsigned NQUEUE = 8
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                 q_has_work,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       q_sel,
  input  logic [63:0]                          ring_base,
  input  logic [15:0]                          ring_head,
  // stage 1: fetch the DESCRIPTOR
  output logic                                 dfetch_valid,
  output logic [63:0]                          dfetch_addr,
  input  logic                                 dfetch_ready,
  input  logic                                 dfetch_data_valid,
  input  logic [63:0]                          dfetch_pkt_addr,
  input  logic [15:0]                          dfetch_pkt_bytes,
  // stage 2: fetch the PACKET DATA, only after the descriptor returned
  output logic                                 pfetch_valid,
  output nic_pkg::nic_xfer_t                   pfetch_xfer,
  input  logic                                 pfetch_ready,
  output logic                                 head_adv
);
 
  import nic_pkg::*;
  localparam int unsigned DESC_BYTES = 16;      // IMPLEMENTATION POLICY
 
  typedef enum logic [1:0] {T_IDLE, T_DFETCH, T_PFETCH} st_e;
  st_e st;
  logic [63:0] pkt_addr;
  logic [15:0] pkt_bytes, cur_qid;
 
  // Two DISTINCT fetches. The descriptor tells the adapter where the packet
  // is; the packet fetch cannot be issued until that answer returns. Merging
  // them is mutation 17 — it issues a read against an address the adapter
  // has not yet been told.
  always_comb begin
    dfetch_valid = (st == T_DFETCH);
    dfetch_addr  = ring_base + 64'(ring_head) * DESC_BYTES;
    pfetch_valid = (st == T_PFETCH);
    pfetch_xfer  = '{queue_id: cur_qid, desc_idx: ring_head, dir: DIR_TX,
                     host_addr: pkt_addr, bytes: pkt_bytes};
    head_adv     = dfetch_valid && dfetch_ready;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      st <= T_IDLE; pkt_addr <= '0; pkt_bytes <= '0; cur_qid <= '0;
    end else begin
      unique case (st)
        T_IDLE:   if (q_has_work) begin st <= T_DFETCH; cur_qid <= 16'(q_sel); end
        T_DFETCH: if (dfetch_data_valid) begin
                    pkt_addr  <= dfetch_pkt_addr;
                    pkt_bytes <= dfetch_pkt_bytes;
                    st        <= T_PFETCH;
                  end
        T_PFETCH: if (pfetch_valid && pfetch_ready) st <= T_IDLE;
        default:  st <= T_IDLE;
      endcase
    end
  end
 
endmodule

Block 6 — the multi-queue arbiter. SYNTHESIZABLE. §11's isolation requirement.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_queue_arbiter #(
  parameter int unsigned NQUEUE = 8
)(
  input  logic clk,
  input  logic rst_n,
  input  logic [NQUEUE-1:0]              q_request,
  input  logic [NQUEUE-1:0]              q_blocked,   // this queue cannot proceed
  output logic [NQUEUE-1:0]              q_grant,
  output logic                           any_grant,
  output logic [nic_pkg::gw(NQUEUE)-1:0] grant_qid
);
 
  import nic_pkg::*;
  logic [gw(NQUEUE)-1:0] rr;
  logic [NQUEUE-1:0]     eligible;
  logic                  found;
  logic [gw(NQUEUE)-1:0] sel;
 
  always_comb begin
    // A BLOCKED queue is skipped rather than held at the head of the
    // arbiter. This is the isolation property: one queue that cannot make
    // progress must not prevent every other queue from being served
    // (mutation 20), which is 25.8 §15's per-class lesson applied to queues.
    eligible = q_request & ~q_blocked;
    sel = rr; found = 1'b0;
    for (int k = 0; k < NQUEUE; k++) begin
      automatic logic [gw(NQUEUE)-1:0] i = gw(NQUEUE)'((rr + k) % NQUEUE);
      if (!found && eligible[i]) begin sel = i; found = 1'b1; end
    end
    any_grant = found;
    grant_qid = sel;
    q_grant   = '0;
    if (found) q_grant[sel] = 1'b1;
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) rr <= '0;
    else if (any_grant) rr <= (sel == gw(NQUEUE)'(NQUEUE-1)) ? '0 : sel + 1'b1;
  end
 
endmodule

Block 7 — the RX completion owner. SYNTHESIZABLE. Status before notification.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_rx_complete #(
  parameter int unsigned NINFLT = 32
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                dma_last_beat,
  input  logic [nic_pkg::gw(NINFLT)-1:0]      dma_slot,
  input  logic [15:0]                         dma_bytes,
  output logic                                status_valid,
  output logic [nic_pkg::gw(NINFLT)-1:0]      status_slot,
  output logic [15:0]                         status_bytes,
  input  logic                                status_ready,
  input  logic                                status_accepted,
  output logic                                evt_valid,
  output logic [nic_pkg::gw(NINFLT)-1:0]      evt_slot,
  input  logic                                evt_ready
);
 
  import nic_pkg::*;
 
  typedef enum logic [1:0] {C_IDLE, C_STATUS, C_EVENT} st_e;
  st_e st;
  logic [gw(NINFLT)-1:0] slot;
  logic [15:0]           bytes;
 
  // The order is: durable status record first, notification second. An
  // interrupt is not the authoritative completion state (19.x), and this
  // module enforces the sequence rather than assuming it.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      st <= C_IDLE; slot <= '0; bytes <= '0;
    end else begin
      unique case (st)
        C_IDLE:   if (dma_last_beat) begin slot <= dma_slot; bytes <= dma_bytes; st <= C_STATUS; end
        C_STATUS: if (status_valid && status_ready && status_accepted) st <= C_EVENT;
        C_EVENT:  if (evt_valid && evt_ready) st <= C_IDLE;
        default:  st <= C_IDLE;
      endcase
    end
  end
 
  assign status_valid = (st == C_STATUS);
  assign status_slot  = slot;
  assign status_bytes = bytes;
  assign evt_valid    = (st == C_EVENT);
  assign evt_slot     = slot;
 
endmodule

Block 8 — the queue→vector snapshot. SYNTHESIZABLE. 19.3's rule (§8 audit C).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_vector_snapshot #(
  parameter int unsigned NQUEUE = 8,
  parameter int unsigned NVEC   = 16
)(
  input  logic clk,
  input  logic rst_n,
  input  logic                                 evt_valid,
  input  logic [nic_pkg::gw(NQUEUE)-1:0]       evt_qid,
  input  logic [nic_pkg::gw(NVEC)-1:0]         vec_table [NQUEUE],  // host-writable, LIVE
  output logic                                 int_req,
  output logic [nic_pkg::gw(NVEC)-1:0]         int_vector,
  output logic [nic_pkg::gw(NQUEUE)-1:0]       int_qid,
  input  logic                                 int_ack
);
 
  import nic_pkg::*;
 
  logic                    pending;
  logic [gw(NVEC)-1:0]     snap_vec;
  logic [gw(NQUEUE)-1:0]   snap_qid;
 
  // The mapping is SNAPSHOTTED when the event takes ownership, and the
  // pending request is driven from the snapshot. Reading vec_table live
  // while the request is stalled sends the notification to a vector the
  // host has since reprogrammed (mutation 24, §8 audit C).
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending <= 1'b0; snap_vec <= '0; snap_qid <= '0;
    end else begin
      if (evt_valid && !pending) begin
        pending  <= 1'b1;
        snap_vec <= vec_table[evt_qid];
        snap_qid <= evt_qid;
      end else if (int_req && int_ack) begin
        pending <= 1'b0;
      end
    end
  end
 
  assign int_req    = pending;
  assign int_vector = snap_vec;
  assign int_qid    = snap_qid;
 
endmodule

Block 9 — interrupt coalescing. SYNTHESIZABLE. IMPLEMENTATION POLICY, not a PCIe requirement.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_coalesce #(
  parameter int unsigned PKT_THRESH = 16,      // IMPLEMENTATION POLICY
  parameter int unsigned TIMER_LIMIT = 200     // IMPLEMENTATION POLICY
)(
  input  logic clk,
  input  logic rst_n,
  input  logic        completion_event,
  input  logic        int_taken,
  output logic        int_request,
  output logic [15:0] batch_count,
  output logic [31:0] c_interrupts, c_events
);
 
  // PCIe specifies NOTHING about interrupt batching. This is a device policy
  // that trades latency for interrupt rate, and §17 measured its effect:
  // one interrupt per packet gave 71,991 interrupts for 71,991 packets;
  // a threshold of 16 gave 4,499; a threshold of 64 gave 1,124.
  logic [15:0] cnt;
  logic [31:0] timer;
  logic        thresh_hit, timer_hit;
 
  // Both terms use the NEXT value. `cnt == PKT_THRESH` on the old value is
  // one packet late, and `>` instead of `>=` waits for the 17th packet —
  // §17 measured that off-by-one firing 4,234 times instead of 4,499.
  assign thresh_hit  = ((cnt + 16'd1) >= 16'(PKT_THRESH)) && completion_event;
  assign timer_hit   = ((timer + 32'd1) >= 32'(TIMER_LIMIT)) && (cnt != 0);
  assign int_request = thresh_hit || timer_hit;
  assign batch_count = cnt;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cnt <= '0; timer <= '0; c_interrupts <= '0; c_events <= '0;
    end else begin
      if (completion_event && c_events != '1) c_events <= c_events + 1'b1;
      // Both conditions true in one cycle close the batch ONCE (§8 audit D).
      if (int_request && int_taken) begin
        cnt <= '0; timer <= '0;
        if (c_interrupts != '1) c_interrupts <= c_interrupts + 1'b1;
      end else begin
        if (completion_event) cnt <= cnt + 16'd1;
        if (cnt != 0) timer <= timer + 32'd1;
      end
    end
  end
 
endmodule

Block 10 — attribution counters. VERIFICATION-ONLY. §5's three candidate causes.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
module nic_counters (
  input  logic clk,
  input  logic rst_n,
  input  logic rx_beat, input logic [15:0] rx_bytes,
  input  logic tx_beat, input logic [15:0] tx_bytes,
  input  logic stall_link,       // credit or Tag exhaustion
  input  logic stall_hostmem,    // the write is accepted slowly upstream
  input  logic stall_nodesc,     // no free host descriptor
  input  logic clear,
  output logic [63:0] c_rx_bytes, c_tx_bytes,
  output logic [31:0] c_link, c_hostmem, c_nodesc
);
 
  // §5's three candidates, counted separately. "RX drops" is one network
  // statistic; these three counters are what turn it into a diagnosis, and
  // the third — no free descriptor — is a SOFTWARE limit that looks exactly
  // like a PCIe one from the network side.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      c_rx_bytes <= '0; c_tx_bytes <= '0;
      c_link <= '0; c_hostmem <= '0; c_nodesc <= '0;
    end else begin
      if (rx_beat && c_rx_bytes != '1) c_rx_bytes <= c_rx_bytes + 64'(rx_bytes);
      if (tx_beat && c_tx_bytes != '1) c_tx_bytes <= c_tx_bytes + 64'(tx_bytes);
      if (stall_link    && c_link    != '1) c_link    <= c_link + 1'b1;
      if (stall_hostmem && c_hostmem != '1) c_hostmem <= c_hostmem + 1'b1;
      if (stall_nodesc  && c_nodesc  != '1) c_nodesc  <= c_nodesc + 1'b1;
    end
  end
 
endmodule

10. Assertions

Queue isolation properties — §11.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P1 — every queue's state is independent: configuring one leaves others alone.
property p1_queue_isolation;
  @(posedge clk) disable iff (!rst_n)
    (cfg_valid && (cfg_qid != q_sel)) |=> ($stable(head[q_sel]) && $stable(tail[q_sel]));
endproperty
a_p1: assert property (p1_queue_isolation);
 
// P2 — an out-of-range queue id is reported, never wrapped.
property p2_qid_range;
  @(posedge clk) disable iff (!rst_n)
    ((db_valid && (db_qid >= NQUEUE)) || (cfg_valid && (cfg_qid >= NQUEUE))) |-> qid_error;
endproperty
a_p2: assert property (p2_qid_range);
 
// P3 — a disabled queue accepts no work.
property p3_disabled_inert;
  @(posedge clk) disable iff (!rst_n)
    !q_enabled |-> !q_has_work;
endproperty
a_p3: assert property (p3_disabled_inert);
 
// P4 — pointers stay within the configured depth.
property p4_pointers_bounded;
  @(posedge clk) disable iff (!rst_n)
    q_enabled |-> ((q_head < cfg_depth) && (q_tail < cfg_depth));
endproperty
a_p4: assert property (p4_pointers_bounded);
 
// P5 — configuring a queue resets its pointers.
property p5_cfg_resets;
  @(posedge clk) disable iff (!rst_n)
    (cfg_valid && !qid_error) |=> ((head[$past(cfg_qid)] == '0) && (tail[$past(cfg_qid)] == '0));
endproperty
a_p5: assert property (p5_cfg_resets);

Buffer and drop properties — §5.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P6 — occupancy never exceeds capacity.
property p6_occupancy_bounded;
  @(posedge clk) disable iff (!rst_n)
    (occupancy <= BUFCAP);
endproperty
a_p6: assert property (p6_occupancy_bounded);
 
// P7 — occupancy never underflows.
property p7_no_underflow;
  @(posedge clk) disable iff (!rst_n)
    (occupancy == 0) |-> !drain_done;
endproperty
a_p7: assert property (p7_no_underflow);
 
// P8 — a frame is accepted only if there is room for it.
property p8_accept_needs_room;
  @(posedge clk) disable iff (!rst_n)
    accept |-> (occupancy < BUFCAP);
endproperty
a_p8: assert property (p8_accept_needs_room);
 
// P9 — a dropped frame always has exactly one recorded reason. §5's whole
// argument: "RX drops" covers three unrelated causes.
property p9_drop_reason_unique;
  @(posedge clk) disable iff (!rst_n)
    (pkt_arrive && !accept) |-> (drop_reason != DROP_NONE);
endproperty
a_p9: assert property (p9_drop_reason_unique);
 
// P10 — a frame is never dropped for lack of a descriptor while buffer space
// exists. It waits instead (§8 audit A).
property p10_nodesc_does_not_drop;
  @(posedge clk) disable iff (!rst_n)
    (pkt_arrive && q_enabled && (occupancy < BUFCAP) && !desc_available)
      |-> (accept && (drop_reason == DROP_NONE));
endproperty
a_p10: assert property (p10_nodesc_does_not_drop);

Descriptor ownership properties — §3.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P11 — a descriptor is allocated only when free.
property p11_alloc_when_free;
  @(posedge clk) disable iff (!rst_n)
    (alloc_valid && alloc_ready) |-> !busy[alloc_slot];
endproperty
a_p11: assert property (p11_alloc_when_free);
 
// P12 — a descriptor is recycled only after its STATUS was accepted, never
// on the last data beat. Counterexample A; §16 measured 479,609 violations.
property p12_recycle_after_status;
  @(posedge clk) disable iff (!rst_n)
    recycle_valid |-> status_accepted;
endproperty
a_p12: assert property (p12_recycle_after_status);
 
// P13 — a host buffer is not reused while hardware owns it.
property p13_no_reuse_while_owned;
  @(posedge clk) disable iff (!rst_n)
    (busy[s] && !recycle_valid) |=> busy[s];
endproperty
a_p13: assert property (p13_no_reuse_while_owned);
 
// P14 — the in-flight count agrees with the busy vector at all times. This
// is the CONSERVATION oracle of §16, and it is blind to cross-talk.
property p14_inflight_consistent;
  @(posedge clk) disable iff (!rst_n)
    (inflight == $countones(busy));
endproperty
a_p14: assert property (p14_inflight_consistent);
 
// P15 — a completion returns to the queue that issued it. This is the
// IDENTITY oracle of §16, and it is blind to premature recycling. Both are
// required; neither is sufficient.
property p15_completion_queue_identity;
  @(posedge clk) disable iff (!rst_n)
    recycle_valid |-> (alloc_xfer_qid_of[status_slot] == status_qid);
endproperty
a_p15: assert property (p15_completion_queue_identity);
 
// P16 — the transfer request is immutable while it waits.
property p16_xfer_stable;
  @(posedge clk) disable iff (!rst_n)
    (alloc_valid && !alloc_ready) |=> (alloc_valid && $stable(alloc_xfer));
endproperty
a_p16: assert property (p16_xfer_stable);

TX fetch properties — §9 Block 5.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P17 — the packet fetch follows the descriptor fetch, never precedes it.
property p17_packet_after_descriptor;
  @(posedge clk) disable iff (!rst_n)
    pfetch_valid |-> $past(dfetch_data_valid, 1) or (st == T_PFETCH);
endproperty
a_p17: assert property (p17_packet_after_descriptor);
 
// P18 — the packet address comes from the fetched descriptor.
property p18_packet_addr_from_desc;
  @(posedge clk) disable iff (!rst_n)
    (dfetch_data_valid) |=> (pkt_addr == $past(dfetch_pkt_addr));
endproperty
a_p18: assert property (p18_packet_addr_from_desc);
 
// P19 — the head advances once per descriptor fetched.
property p19_head_once;
  @(posedge clk) disable iff (!rst_n)
    head_adv |-> (dfetch_valid && dfetch_ready);
endproperty
a_p19: assert property (p19_head_once);

Arbitration properties — §11.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P20 — at most one queue granted per cycle.
property p20_grant_onehot0;
  @(posedge clk) disable iff (!rst_n)
    $onehot0(q_grant);
endproperty
a_p20: assert property (p20_grant_onehot0);
 
// P21 — a blocked queue is never granted, and never blocks the others.
property p21_blocked_skipped;
  @(posedge clk) disable iff (!rst_n)
    q_blocked[i] |-> !q_grant[i];
endproperty
a_p21: assert property (p21_blocked_skipped);
 
// P22 — an eligible queue is eventually granted (no starvation).
property p22_no_starvation;
  @(posedge clk) disable iff (!rst_n)
    (q_request[i] && !q_blocked[i]) |-> s_eventually q_grant[i];
endproperty
a_p22: assert property (p22_no_starvation);
 
// P23 — a queue that is blocked does not prevent another from being served.
// 25.8 §15's per-class lesson, applied to queues.
property p23_isolation_under_block;
  @(posedge clk) disable iff (!rst_n)
    ((q_request & ~q_blocked) != '0) |-> any_grant;
endproperty
a_p23: assert property (p23_isolation_under_block);

Notification properties — §9 Blocks 7–9.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P24 — the event never precedes the status publication.
property p24_event_after_status;
  @(posedge clk) disable iff (!rst_n)
    evt_valid |-> $past(status_accepted);
endproperty
a_p24: assert property (p24_event_after_status);
 
// P25 — the vector is snapshotted at event ownership and held.
property p25_vector_snapshot_stable;
  @(posedge clk) disable iff (!rst_n)
    (int_req && !int_ack) |=> (int_req && $stable(int_vector) && $stable(int_qid));
endproperty
a_p25: assert property (p25_vector_snapshot_stable);
 
// P26 — a batch closes exactly once even when both triggers fire together.
property p26_batch_closes_once;
  @(posedge clk) disable iff (!rst_n)
    (int_request && int_taken) |=> (batch_count == 16'd0);
endproperty
a_p26: assert property (p26_batch_closes_once);
 
// P27 — the coalescing count is exact: the threshold fires ON the Nth event.
// §17 measured `>` instead of `>=` firing 4,234 times instead of 4,499.
property p27_threshold_exact;
  @(posedge clk) disable iff (!rst_n)
    (completion_event && ((batch_count + 1) == PKT_THRESH)) |-> int_request;
endproperty
a_p27: assert property (p27_threshold_exact);

Cover — the anti-vacuity set.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P28 — a frame reaches exactly one outcome: accepted into the buffer, or
// dropped with a recorded reason. A frame that produces neither has been
// lost with no counter naming it, which makes §5's three-way attribution
// unavailable for that frame forever.
property p28_frame_outcome_total;
  @(posedge clk) disable iff (!rst_n)
    pkt_arrive |-> (accept ^ (drop_reason != DROP_NONE));
endproperty
a_p28: assert property (p28_frame_outcome_total);
 
// P28's covers — the load-dependent states must occur. §15 measured ZERO drops at two
// of four drain rates, which makes P8, P9 and P10 vacuous against a test
// whose host keeps up.
c1_buffer_full: cover property (@(posedge clk) disable iff (!rst_n) occupancy == BUFCAP);
c2_drop_buffer: cover property (@(posedge clk) disable iff (!rst_n) drop_reason == DROP_BUFFER);
c3_no_desc:     cover property (@(posedge clk) disable iff (!rst_n)
                  pkt_arrive && q_enabled && !desc_available);
c4_queue_block: cover property (@(posedge clk) disable iff (!rst_n) q_blocked != '0);
c5_vec_rewrite: cover property (@(posedge clk) disable iff (!rst_n)
                  int_req && !int_ack && vec_write);
c6_both_trig:   cover property (@(posedge clk) disable iff (!rst_n) thresh_hit && timer_hit);
c7_all_inflt:   cover property (@(posedge clk) disable iff (!rst_n) busy == '1);
c8_ring_wrap:   cover property (@(posedge clk) disable iff (!rst_n)
                  head_adv && (q_head == cfg_depth - 1));

11. Multi-Queue Isolation

A NIC with many queues has a failure mode a single-queue device cannot have: one queue's problem becoming every queue's problem.

Three places where coupling creeps in, all of which §9's RTL guards:

Shared state. A single head/tail pair, a single in-flight counter, or a single descriptor pool indexed without the queue id couples every queue to every other. P1 asserts per-queue independence, and mutation 11 is the version that shares.

Arbitration. An arbiter that holds a blocked queue at the head of its rotation stops serving anyone. §9 Block 6 skips blocked queues explicitly, and P23 is the property that a blocked queue does not prevent an eligible one from being served — which is 25.8 §15's per-class liveness lesson in a new setting.

And completion routing. A completion credited to the wrong queue corrupts two rings simultaneously. §16 measured 3,284 such events — and found that the natural conservation check is structurally incapable of detecting them, which is the subject of §16.

12. Interrupt Coalescing Is a Policy

PCIe specifies nothing about how often a device interrupts. Batching completions and raising one interrupt for many is IMPLEMENTATION POLICY, and §17 measured the trade:

policypacketsinterruptspackets/interrupt
threshold 1 (one per packet)71,99171,9911.0
threshold 16, no timer71,9914,49916.0
threshold 16 + 200-step timer71,9914,49916.0
threshold 64 + 200-step timer71,9911,12464.0

The timer exists for the tail of a burst. Without it, the last packets of a flow wait for a threshold that will never be reached until more traffic arrives. With it, latency is bounded, and the interrupt rate is unchanged at steady load — rows 2 and 3 are identical because the threshold is reached before the timer expires whenever traffic is continuous.

The arithmetic must use the next value. §17 measured > instead of >= firing 4,234 times instead of 4,499 — every batch one packet larger than configured, and the last packet of each burst waiting for the timer instead. This is 25.4 §7's comparison rule in a new place, and P27 is the property.

Adaptive schemes exist and are not modelled here. §9 Block 9 shows the contract — count, timer, exactly-once batch closure — not a tuning algorithm.

13. What Limits a NIC

candidate limitsignature
the PCIe Linkcredit/Tag stalls; drops scale with frame count, not bytes
host memory pathhost-memory stalls; drops appear at high rates only
descriptor supplyDROP_NODESC counter rises; the Link is idle
queue count and arbitrationone queue starves while others run
interrupt rateCPU saturated by interrupts; §12's policy is mis-tuned
the network itselfgenuine overrun; not a PCIe matter

§15's measurement covers rows 2 and 3, and §9 Block 10's separate counters are what distinguish them. The third row is the one most often misdiagnosed as a PCIe problem: an adapter with no free descriptors cannot write to host memory regardless of available bandwidth, and from the network side it looks identical to a bandwidth shortfall.

14. Measured Behaviour — PCIe Bytes per Network Byte

frame bytespayloaddescriptor readstatus writePCIe totalratio
64641616961.50×
12812816161601.25×
25625616162881.12×
51251216165441.06×
15181518161615501.02×
90009000161690321.00×

A 64-byte frame costs 1.50× its own length in PCIe traffic. A 9000-byte frame costs 1.004×. Sizing from average network throughput alone understates the PCIe requirement by up to 50%, and the transaction count difference is larger still.

15. Measured Behaviour — The Drain Rate Decides

host + PCIe drain ratearriveddelivereddroppedbuffer at end
1.00/step — faster than the wire107,905107,88400
0.90/step — matched to the wire108,037107,971045
0.70/step — slower than the wire107,90984,14323,495255
0.40/step — much slower107,96548,12159,580256

Three readings.

Rows 1 and 2 drop nothing, and row 2 sits with a partly-full buffer — the system is at its operating limit and absorbing bursts. That is a healthy state, and it is one instrument-reading away from the failing one.

Row 3 drops 23,495 frames with the MAC unchanged. The network offered the same load; the buffer could not drain. The counter that increments is a receive drop, and every PCIe error counter reads zero.

And row 4 saturates the buffer completely. Once occupancy pins at capacity, the drop rate is simply the difference between arrival and drain — the buffer has stopped providing any slack at all.

16. Measured Behaviour — Two Oracles, Two Blind Spots

Each oracle is structurally blind to the fault the other catches, and the reasons are worth stating precisely.

Conservation catches the premature free because returning a descriptor to the pool while hardware still owns it inflates that ring's accounting — free + owned exceeds RING and stays wrong.

Conservation is blind to cross-talk because moving a completion between rings decrements and increments the same wrong ring. That ring's sum is unchanged. The originating ring keeps its owned entry, so its sum is also unchanged. Both rings satisfy the invariant while both are corrupted.

Identity catches cross-talk by construction, and is blind to the premature free, which misroutes nothing — the descriptor goes back to the correct ring, just too early.

The general result matters more than the specific one:

An invariant that holds under a fault is not evidence the fault is absent. Two fault classes need two oracles, and running one is running none for half the space.

This is the third time Module 25 and 26 have produced this shape25.6 §13's one-sided conservation check, 25.7 §13's epoch-derived diagnostic, and 25.8 §15's global liveness check. Each time the fix was an oracle sharing no machinery with the mechanism under test, and P14 and P15 are that pair here.

17. Measured Behaviour — Coalescing

DERIVED. 120,000 steps, ~0.6 completions/step.

policypacketsinterruptspackets/interrupt
threshold 1 (one per packet)71,99171,9911.0
threshold 16, no timer71,9914,49916.0
threshold 16 + 200-step timer71,9914,49916.0
threshold 64 + 200-step timer71,9911,12464.0

Off-by-one check: threshold 16 with >= fires 4,499 times; with > it fires 4,234 — every batch one packet larger than configured, and the last packet of each burst deferred to the timer.

Rows 2 and 3 are identical because the traffic is continuous. The timer never expires when the threshold is always reached first. Its value is entirely in the tail of a burst, which this steady-load measurement does not exercise — a limitation of the measurement, stated rather than hidden.

18. Verification — Mutations

Thirty-four mutations. Every "Caught by" entry names a property from §10.

#MutationSymptomCaught by
1Single head/tail shared across queuesone queue's stall stops allP1
2Queue id wrapped instead of range-checkedtraffic delivered to an unrelated ringP2
3Queue id 0 aliased with "no queue"queue 0 unreachable or always selectedP2
4Disabled queue still servedframes delivered to a torn-down ringP3
5Pointers allowed past the configured depthdescriptor reads outside the ringP4
6Configuring a queue leaves stale pointersphantom descriptors after re-setupP5
7Occupancy incremented on arrival, not acceptancebuffer overruns silentlyP6, P8
8Occupancy underflows on a spurious drainthe buffer appears infinitely deepP7
9Frame accepted with the buffer fullpacket data overwritten in the bufferP8
10All drops recorded under one counter§5's three causes become indistinguishableP9
11Frame dropped for lack of a descriptorframes discarded that the buffer had room forP10
12Descriptor allocated while busytwo frames share one host bufferP11
13Descriptor recycled on the last data beat479,609 conservation violations (§16)P12
14Descriptor recycled on status issue, not acceptancesoftware reads a partly-written bufferP12
15Host buffer reused while hardware owns itpacket data overwritten mid-DMAP13
16In-flight count cached rather than derivedthe allocator believes slots existP14
17Completion credited to the wrong queue3,284 identity violations; conservation blind (§16)P15
18Transfer request re-derived while stalledthe DMA address changes under the engineP16
19Packet fetched before the descriptor returnsa read against an address not yet knownP17, P18
20Packet address taken from the previous descriptorthe wrong packet transmittedP18
21Head advanced twice per descriptorevery other TX descriptor skippedP19
22Arbiter grants two queues in one cycletwo transfers share one resourceP20
23Blocked queue held at the arbiter headone stalled queue stops every queueP21, P23
24Fixed priority replaces round robinthe highest queue index never progressesP22
25Vector table read live while the event stallsnotification sent to a reprogrammed vectorP25
26Event raised with the status writehost reads a stale descriptor statusP24
27Coalescing compares the old counter valuefires one packet late, every batchP27
28Coalescing uses > instead of >=4,234 interrupts instead of 4,499 (§17)P27
29Both triggers close the batch twicethe count is cleared twice; events lostP26
30Coalescing timer never resetsone interrupt then silenceP26
31Timer runs with an empty batchinterrupts with nothing to reportP26
32PCIe credit stall reported as a network flow-control eventthe wrong subsystem is investigated (§5)P9
33Testbench runs only at a drain rate that keeps upP8, P9, P10 vacuous (§15: zero drops in 2 of 4 rows)P28 (c1, c2)
34Testbench uses one queue onlyisolation and cross-talk unreachableP28 (c4)

Mutations 13 and 17 are the pair §16 exists for. Each is caught by exactly one oracle, and a verification plan running only one of them scores the other as clean.

19. Executable Counterexamples

Counterexample A — recycling on the last beat (violates P12, caught by conservation).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// The RX descriptor is returned to software when the final data beat moves,
// rather than when the status write has been accepted.
module ce_a_recycle_on_beat (
  input  logic clk, rst_n,
  input  logic        dma_last_beat,
  input  logic [4:0]  dma_slot,
  output logic        recycle_valid,
  output logic [4:0]  recycle_slot
);
  assign recycle_valid = dma_last_beat;      // <-- not status_accepted
  assign recycle_slot  = dma_slot;
endmodule
 
// Failing stimulus: the last beat is issued, and the status write is
// back-pressured for several cycles.
// Golden: the descriptor stays owned until status_accepted.
// This:   software sees the descriptor returned and reads the buffer while
//         the final write may still be in flight.
// P12 fails. P14's conservation oracle reports 479,609 violations (§16).
// Observable consequence: intermittent short or corrupted frames delivered
// to the network stack, at a rate that rises with host-memory latency.

Counterexample B — cross-queue completion (violates P15, invisible to conservation).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// The completion path derives the queue from the current arbiter selection
// rather than from the record made when the transfer was issued.
module ce_b_cross_queue_completion (
  input  logic [2:0] arbiter_qid_now,     // whatever the arbiter is serving
  input  logic       completion_valid,
  output logic [2:0] credit_qid
);
  assign credit_qid = arbiter_qid_now;    // <-- not the issuing queue
endmodule
 
// Failing stimulus: two queues with transfers in flight; a completion for
// queue 1 returns while the arbiter is serving queue 2.
// Golden: queue 1 is credited.
// This:   queue 2 is credited. Queue 1 leaks an owned descriptor and queue 2
//         gains one it never issued.
// P15 fails. P14's CONSERVATION oracle reports ZERO — the wrong ring's
// decrement and increment cancel, so every ring's sum stays correct (§16).
// §16 measured 3,284 such events. Observable consequence: one queue slowly
// runs out of descriptors while another accumulates phantom free entries.

Counterexample C — the merged drop counter (violates P9, and destroys §5's diagnosis).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// All discarded frames increment one counter.
module ce_c_merged_drops (
  input  logic clk, rst_n,
  input  logic pkt_arrive, accept,
  output logic [31:0] rx_drops
);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) rx_drops <= '0;
    else if (pkt_arrive && !accept) rx_drops <= rx_drops + 1'b1;
endmodule
 
// Failing stimulus: a run mixing buffer-full drops, no-descriptor stalls and
// frames arriving on a disabled queue.
// Golden: three separate counters naming three unrelated causes.
// This:   one number. A reading of 23,495 is consistent with a PCIe/host
//         bandwidth shortfall (§15 row 3), with software failing to post
//         descriptors, and with a queue that was never enabled.
// P9 fails.
// Observable consequence: §21 case 2 is unanswerable. The three causes have
// three different owners — the platform, the driver, and configuration —
// and no further analysis of a merged counter can separate them.

20. Misconceptions

"Line-rate networking means an equal PCIe byte rate." §14 measured a 64-byte frame costing 1.50× its length in PCIe traffic. The transaction count difference is larger still.

"Packet loss is a network problem." §15 measured 23,495 frames dropped with the MAC and the offered load unchanged — the host/PCIe path could not drain the buffer (§5).

"If PCIe error counters are clean, PCIe is not involved." Nothing about PCIe failed in §15's rows. It was not fast enough, or the host behind it was not. Absence of errors is not absence of a limit.

"RX drops means the buffer overflowed." It covers at least three causes: buffer full, no free descriptor, queue disabled (§5, counterexample C). They have three different owners.

"No free descriptors means the adapter should drop the frame." Not if buffer space exists — it can wait (P10, §8 audit A). Dropping frames the adapter had room for is mutation 11.

"A descriptor can be recycled when the data has moved." Only when the status has been accepted (P12). §16 measured 479,609 conservation violations from the alternative.

"Our ring conservation check would catch a completion routing bug." It provably would not (§16). The wrong ring's decrement and increment cancel; both rings satisfy the invariant while both are corrupted.

"One oracle is enough if the invariant is strong." §16 measured two faults, two oracles, and each oracle blind to the other's fault.

"A stalled queue only affects itself." Only if the arbiter skips it (P21, P23). An arbiter that holds a blocked queue at the head stops serving everyone — mutation 23.

"More queues means more throughput." Only with genuine isolation (§11). Shared state or a coupling arbiter makes additional queues a liability.

"Interrupt coalescing is a PCIe feature." It is device policy (§12). PCIe specifies nothing about how often a device interrupts.

"Coalescing thresholds are approximate." §17 measured > instead of >= changing the interrupt count from 4,499 to 4,234, with every batch one packet oversized and burst tails deferred.

"The coalescing timer reduces the interrupt rate." At steady load it changes nothing — rows 2 and 3 of §17 are identical. Its value is bounding latency in the tail of a burst.

"An interrupt means the descriptor status is ready." Only if sequenced after the status write (P24). Otherwise the host reads a stale ring entry.

"The MSI-X vector can be read when the interrupt is sent." It must be snapshotted at event ownership (P25, §8 audit C). A host rewrite while the request is stalled retargets a notification that already belongs to a different queue.

"TX descriptors posted but the wire is idle means a network problem." It usually means the descriptor fetch or the packet fetch is not progressing (§21 case 3) — two separate PCIe reads, either of which can stall.

"SmartNIC processing is part of the PCIe architecture." It is another consumer of the same descriptor and buffer resources (§1). Its internal behaviour is outside this chapter and is not a PCIe requirement.

21. Debugging

22. Understanding Check

Q1. RX packet buffers fill while the MAC continues receiving. How can a host-memory or PCIe bottleneck appear as network packet loss?

Because the adapter cannot tell the wire to wait (§2, §5). Frames arrive at whatever rate the far end sends; the packet buffer is the only slack; and when the PCIe/host path drains it more slowly than the network fills it, occupancy pins at capacity and further frames are discarded. §15 measured 23,495 drops at a drain rate 22% below the offered load, with the MAC, the network and the offered load unchanged in every row. The counter that increments is a receive drop in the network statistics — the same one a genuine network overrun increments — while every PCIe error counter reads zero, because nothing about PCIe failed. It simply was not fast enough.

Q2. Your ring conservation check reports clean and two queues are corrupting each other. Explain.

Conservation is structurally blind to cross-queue routing (§16, counterexample B). Crediting a completion to the wrong ring decrements and increments the same wrong ring, so that ring's free + owned is unchanged; the originating ring keeps its owned entry, so its sum is unchanged too. Both rings satisfy the invariant while both are corrupted. §16 measured 3,284 such events with zero conservation violations. Detecting it requires an identity oracle — does this completion return to the ring that issued it? — checked against a record the ring logic never reads (P15).

Q3. Sizing a PCIe path from average network throughput. What does that miss?

The fixed per-frame cost, which dominates at small frame sizes (§14). Every frame also costs a descriptor read and a status write; §14 measured a 64-byte frame costing 96 bytes of PCIe traffic — 1.50× — against 1.004× for a 9000-byte frame. A test at large frames and a deployment at small frames are different PCIe workloads at the same wire rate, and the transaction count difference is larger than the byte difference, which matters more for Tags and per-transaction overheads (22.4).

Q4. A frame arrives, buffer space exists, and no free host descriptor is available. Should the adapter drop it?

No — it can wait (P10, §8 audit A, mutation 11). Acceptance depends on buffer space; descriptor allocation is a separate, later decision. Dropping here throws away a frame the adapter had room for, and it misattributes the cause: the drop counter that should rise is DROP_NODESC, which names a software limit (the driver has not posted buffers), not a PCIe one. §9 Block 3 keeps the two decisions separate for exactly this reason, and §5's three-way attribution depends on it.

Q5. Why must an RX descriptor be recycled on status acceptance rather than on the last data beat?

Because the last beat does not mean the data is visible to software (P12, counterexample A). Recycling on the beat returns the buffer while the final write may still be in flight, so software reads a partly-written frame. §16 measured 479,609 conservation violations from that single change. The correct sequence is the same one every device in this module follows: durable status record first, ownership return second, notification third — and P24 covers the third step.

Q6. Interrupt coalescing at threshold 16 fires 4,234 times instead of 4,499. What is wrong?

The comparison is > instead of >= (§12, §17, P27, mutation 28). It waits for the 17th packet, so every batch is one packet larger than configured, and the last packet of each burst is deferred to the timer rather than closing the batch. This is 25.4 §7's next-value comparison rule in a new setting: the threshold must be evaluated on count + 1, not on the registered count.

Q7. One queue stalls and every queue stops. Where is the fault?

In the arbiter, not in the stalled queue (§11, P21, P23, mutation 23). A blocked queue must be skipped, not held at the head of the rotation. This is 25.8 §15's per-class liveness lesson applied to queues: a system where one blocked participant halts all the others has coupled things that were supposed to be independent, and the symptom — total stoppage from a single-queue problem — points at the shared machinery rather than at the queue that reported first.

Q8. Interrupt coalescing shows identical interrupt counts with and without the timer. Is the timer doing nothing?

Not nothing — nothing at this load (§17). Rows 2 and 3 are identical because the traffic is continuous, so the packet threshold is always reached before the timer expires. The timer's entire value is in the tail of a burst, where the last few packets would otherwise wait indefinitely for a threshold that no further traffic will reach. §17's steady-load measurement does not exercise that case, which is a stated limitation of the measurement rather than evidence about the timer — and testing it requires a stimulus with bursts that end.

23. What Comes Next

One chapter remains, and it removes the last assumption: that someone else wrote the DMA engine.

ChapterThe device, and what it owns
26.1 CPUsthe host — address ownership and translation
26.2 GPUscontrol path and data path as separate resources
26.3 SSD Controllersa device that fetches its own work; two "completions"
26.4 (this)a device driven by traffic it does not control
26.5 FPGA Cardsa device whose DMA engine you write yourself

This chapter's device could not apply back-pressure to its input. That produced §5's result — a PCIe-side limit surfacing as a network statistic — and it is the sharpest example in the module of why the bottleneck and the symptom can live in different subsystems.

§16 is the result to carry forward. Two ownership faults, two oracles, each blind to the other's fault. 26.5 has more ownership boundaries than any chapter in this module — descriptor, Tag, FIFO, clock domain, reset epoch, local memory — and the same principle applies at each: an invariant that holds under a fault is not evidence the fault is absent.