AMBA AXI · Module 8

Verification Challenges

Why outstanding, out-of-order, multi-master AXI traffic is the hardest part of the protocol to verify — state-space explosion, the out-of-order scoreboard problem, coverage of concurrency, and the constrained-random + assertion methodology that tames it.

Module 8 built up the concurrency model — outstanding depth, IDs, same-ID ordering, out-of-order completion, and how it all converges at the interconnect. This closing chapter asks the practical question: how do you verify all of that? The honest answer is that outstanding, out-of-order, multi-master AXI is the hardest part of the protocol to verify, because correctness now depends on interleavings and timing rather than single transactions. This chapter lays out why it's hard — state-space explosion, the out-of-order scoreboard, concurrency coverage — and the methodology (constrained-random stimulus, a byte-accurate reference model, protocol assertions, coverage closure) that makes it tractable.

1. Why It's Hard — State-Space Explosion

A single in-order transaction is easy to check: issue it, compare the result. Concurrency detonates that simplicity. The behaviors multiply along several axes at once:

Outstanding depth — how many transactions are in flight (1 … the max).
ID combinations — which transactions share IDs (ordered) vs differ (reorderable).
Completion interleavings — the orders in which different-ID responses can return.
Multiple masters — concurrent traffic from several managers through the interconnect.
Address relationships — overlapping vs independent addresses (hazards).

The number of distinct interleavings grows combinatorially — you cannot enumerate them. So verification shifts from "check every case" to "sample the space well and check every sample rigorously," which is a fundamentally different problem from in-order checking.

Figure 1 — the concurrency state space. Outstanding depth × ID combinations × completion interleavings × number of masters × address relationships multiply into a space far too large to enumerate. Verification must sample it (constrained-random) rather than exhaustively cover it — the core reason concurrent AXI is hard to verify.

2. The Out-of-Order Scoreboard Problem

The central checker — the scoreboard — is much harder than for in-order traffic. It must:

Match by ID, not position — pair each BID/RID response with its originating request by ID, since responses arrive in any order (8.4). A position-based scoreboard is itself wrong.
Track per-ID order — assert that same-ID responses arrive in issue order while allowing different-ID responses to interleave (8.3/8.4). That checker is written out in same-ID ordering §6; §5 below adds the lifecycle bookkeeping it does not cover.
Model memory byte-accurately — a reference memory updated by WSTRB so reads can be checked to the byte (Module 7), since narrow/unaligned/sparse writes are in the mix.
Recognize hazards — flag overlapping-address accesses across different IDs (undefined result) rather than asserting a specific value.

So the scoreboard is effectively a concurrent reference model of the whole ordering contract plus memory, not a simple expected-value table. Building it correctly is most of the verification effort — and a scoreboard that quietly assumes in-order completion will pass a buggy DUT (or fail a correct one) the moment reordering occurs.

Figure 2 — the out-of-order scoreboard. Requests and responses feed a model that matches by ID (never position), checks same-ID issue order while allowing different-ID interleave, maintains a byte-accurate reference memory (WSTRB-updated), and flags same-address different-ID hazards. It's a concurrent model of the ordering contract plus memory — the hardest component to build.

3. The Methodology — Sample, Check, Cover

Because the space can't be enumerated, the industry-standard approach is constrained-random verification with coverage closure, layered with assertions and a reference model — the structure a UVM testbench provides:

Constrained-random stimulus — generate legal but varied traffic: random outstanding depths, ID assignments, burst types, addresses, and (critically) active response reordering from slave/interconnect models, across multiple masters.
Protocol assertions / VIP — continuously check the rules (handshake stability, same-ID ordering, 4 KB boundary, exclusive access, legal strobes) on every transaction — catching violations the moment they occur.
Reference model / scoreboard — check the results (Section 2).
Functional coverage — measure which parts of the state space were actually exercised (depths reached, ID mixes, interleavings, hazards, corners) and drive stimulus until coverage closes.
Directed tests — pin down known hard corners explicitly (max depth, ID collisions, same-ID across slaves, deadlock scenarios) that random may hit rarely.

The loop is: randomize → check (assertions + scoreboard) → measure coverage → refine constraints toward uncovered corners. Coverage is what turns "we ran a lot of random traffic" into "we exercised the space."

Figure 3 — the verification methodology. Constrained-random stimulus (multi-master, actively reordering) drives the DUT; protocol assertions check the rules continuously while a byte-accurate scoreboard checks results; functional coverage measures what was exercised and feeds back to refine stimulus; directed tests nail known corners. Sample-check-cover replaces enumerate.

4. The Corner Cases That Bite

Certain scenarios are where concurrency bugs concentrate — and where random stimulus needs help (directed tests / targeted constraints) because it hits them rarely:

Maximum outstanding depth — fill every buffer to capacity, then one more (must back-pressure, not drop).
ID collisions across masters — multiple managers using the same ID value through the interconnect (tests ID extension/routing).
Same-ID across slaves — the serialization case (assert order preserved; characterize the throughput cost).
Maximally-reordered responses — last-issued different-ID transaction completes first (stresses scoreboard matching).
Address hazards — overlapping accesses across different IDs (must be flagged) and dependent read/write pairs (must be ordered by the master).
Deadlock / livelock — all channels and pairs saturated; assert forward progress — see handshake deadlock rules for the dependency graph and the bounded-liveness watchdog.
Backpressure interleavings — READY deasserted at adversarial moments on each channel.

Figure 4 — the concurrency corner-case checklist. Max outstanding depth, cross-master ID collisions, same-ID across slaves, maximally-reordered responses, address hazards, deadlock/livelock, and adversarial backpressure. These are where bugs concentrate and where directed tests supplement random stimulus, since random hits them rarely.

5. The Outstanding Tracker — the Bookkeeping, Written Out

Section 2 specifies a scoreboard and §7 calls it "the single biggest leverage point." Both describe it in prose. Here is the part that carries the weight — the outstanding bookkeeping: what is in flight, whether the depth contract is respected, and whether every request is answered exactly once.

This is deliberately not an ordering checker — same-ID ordering already owns that, and duplicating it would leave the harder half unwritten. What is missing there and needed here is the lifecycle: issue → track → respond → retire, plus the three failures that lifecycle exposes and ordering checks cannot see — a response that arrives twice, a request that is never answered, and a manager that exceeds its declared depth.

axi_outstanding_tracker.sv — lifecycle bookkeeping, bindable to any AXI4 port

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// ─────────────────────────────────────────────────────────────────────────────
//  Outstanding-transaction tracker (AMBA AXI4).
//     bind axi_master axi_outstanding_tracker #(.ID_W(4), .MAX_OUTSTANDING(16)) u_ot (.*);
//
//  Answers three questions an ordering checker cannot:
//    1. did a response arrive for something that was never issued (or already
//       retired) — a DUPLICATE;
//    2. did anything issued never come back — a MISSING response, only visible
//       at end of test;
//    3. did in-flight count ever exceed the depth the manager declared it would
//       not exceed — a DEPTH violation, which is a contract with the slave's
//       buffering, not a protocol rule.
// ─────────────────────────────────────────────────────────────────────────────
module axi_outstanding_tracker #(
  parameter int ID_W            = 4,
  parameter int MAX_OUTSTANDING = 16     // the manager's declared limit
) (
  input logic            aclk, aresetn,
  input logic            awvalid, awready, input logic [ID_W-1:0] awid,
  input logic            bvalid,  bready,  input logic [ID_W-1:0] bid,
  input logic            arvalid, arready, input logic [ID_W-1:0] arid,
  input logic            rvalid,  rready, rlast, input logic [ID_W-1:0] rid
);
 
  localparam int N_ID = 1 << ID_W;
 
  // Per-ID in-flight counts. Per-ID (not a single total) because AXI allows
  // many outstanding per ID, and a duplicate response is only detectable
  // against the count for THAT id.
  int unsigned wr_inflight [N_ID];
  int unsigned rd_inflight [N_ID];
  int unsigned wr_total, rd_total;          // aggregate, for the depth contract
  int unsigned wr_issued, wr_retired, rd_issued, rd_retired;
 
  always_ff @(posedge aclk) begin
    if (!aresetn) begin
      for (int i = 0; i < N_ID; i++) begin
        wr_inflight[i] <= 0;
        rd_inflight[i] <= 0;
      end
      wr_total <= 0; rd_total <= 0;
      wr_issued <= 0; wr_retired <= 0; rd_issued <= 0; rd_retired <= 0;
    end else begin
      // ── issue ────────────────────────────────────────────────────────────
      if (awvalid && awready) begin
        wr_inflight[awid] <= wr_inflight[awid] + 1;
        wr_total  <= wr_total + 1;
        wr_issued <= wr_issued + 1;
      end
      if (arvalid && arready) begin
        rd_inflight[arid] <= rd_inflight[arid] + 1;
        rd_total  <= rd_total + 1;
        rd_issued <= rd_issued + 1;
      end
 
      // ── retire ───────────────────────────────────────────────────────────
      //  A response for an ID with zero in flight is the DUPLICATE / phantom
      //  case: either the slave answered twice, or an interconnect remapped an
      //  ID and the response landed on the wrong one.
      if (bvalid && bready) begin
        if (wr_inflight[bid] == 0)
          $error("[%0t] B response for AWID=%0h with nothing in flight - duplicate or ID remap",
                 $time, bid);
        else begin
          wr_inflight[bid] <= wr_inflight[bid] - 1;
          wr_total   <= wr_total - 1;
          wr_retired <= wr_retired + 1;
        end
      end
      if (rvalid && rready && rlast) begin
        if (rd_inflight[rid] == 0)
          $error("[%0t] RLAST for ARID=%0h with nothing in flight - duplicate or ID remap",
                 $time, rid);
        else begin
          rd_inflight[rid] <= rd_inflight[rid] - 1;
          rd_total   <= rd_total - 1;
          rd_retired <= rd_retired + 1;
        end
      end
    end
  end
 
  // ── The depth contract. NOT an AXI rule: AXI places no limit on outstanding
  //    depth. This is the integration agreement between a manager's declared
  //    capability and a slave's buffering, and exceeding it overflows a real
  //    queue somewhere. Treated as a check because it is a contract, and
  //    labelled so nobody quotes it as protocol.
  a_write_depth: assert property (@(posedge aclk) disable iff (!aresetn)
      wr_total <= MAX_OUTSTANDING)
    else $error("[%0t] write outstanding depth %0d exceeded declared %0d",
                $time, wr_total, MAX_OUTSTANDING);
  a_read_depth:  assert property (@(posedge aclk) disable iff (!aresetn)
      rd_total <= MAX_OUTSTANDING)
    else $error("[%0t] read outstanding depth %0d exceeded declared %0d",
                $time, rd_total, MAX_OUTSTANDING);
 
  // ── Coverage of the depth axis from §1. Without these, a run that never got
  //    past one outstanding transaction still "passes" every check above.
  covergroup cg_depth @(posedge aclk);
    option.per_instance = 1;
    wr_depth: coverpoint wr_total {
      bins idle    = {0};
      bins shallow = {[1:3]};
      bins mid     = {[4:MAX_OUTSTANDING-1]};
      bins at_max  = {MAX_OUTSTANDING};        // the corner §4 says random under-hits
    }
    rd_depth: coverpoint rd_total {
      bins idle    = {0};
      bins shallow = {[1:3]};
      bins mid     = {[4:MAX_OUTSTANDING-1]};
      bins at_max  = {MAX_OUTSTANDING};
    }
    concurrent: cross wr_depth, rd_depth;      // reads and writes in flight together
  endgroup
  cg_depth cg = new();
 
  // ── The drain check. Every other check here is conditional; this is the one
  //    that fails on the ABSENCE of an event, which is why it must run at the
  //    end rather than continuously.
  final begin
    if (wr_issued != wr_retired)
      $error("DRAIN: %0d writes issued, %0d retired - %0d never answered",
             wr_issued, wr_retired, wr_issued - wr_retired);
    if (rd_issued != rd_retired)
      $error("DRAIN: %0d reads issued, %0d retired - %0d never answered",
             rd_issued, rd_retired, rd_issued - rd_retired);
    for (int i = 0; i < N_ID; i++) begin
      if (wr_inflight[i] != 0)
        $error("DRAIN: AWID=%0h still has %0d write(s) outstanding", i, wr_inflight[i]);
      if (rd_inflight[i] != 0)
        $error("DRAIN: ARID=%0h still has %0d read(s) outstanding",  i, rd_inflight[i]);
    end
    $display("[outstanding] writes %0d/%0d retired, reads %0d/%0d retired",
             wr_retired, wr_issued, rd_retired, rd_issued);
  end
 
endmodule

What each check catches, and what a failure means.

Check	Catches	A failure means
in-flight count is zero on response	Duplicate / phantom response	The subordinate answered twice, replayed, or an interconnect remapped an ID so the response retired the wrong counter
`a_write_depth` / `a_read_depth`	Depth-contract breach	The manager issued beyond what it declared — a queue overflows downstream. Not a protocol violation; AXI sets no depth limit
`final` drain block	Missing response	Something was issued and never answered: a drop, a hang, or a test that ended before the bus did
`cg_depth` cross	Untested concurrency	If `at_max` is empty, the corner §4 calls out as under-hit by random was never reached, and the depth checks above proved nothing

The cg_depth cross is the piece worth dwelling on. Section 1 argues the state space cannot be enumerated and must be sampled; a coverage model is what turns that argument into a measurement. Without it, a regression that never exceeded two outstanding transactions reports the same clean result as one that saturated every queue — and §3's "sample, check, cover" loop has no cover to close.

6. Common Misconceptions

Myths to unlearn

"If in-order traffic passes, the design is verified." No — concurrency bugs only appear under reordering, depth, and multi-master traffic. In-order testing leaves the hardest behaviors unexercised.
"A scoreboard that matches by arrival order is fine." It's wrong — out-of-order completion requires matching by ID. Position-based scoreboards mis-verify the moment reordering happens.
"Random traffic alone covers the space." Random is necessary but hits hard corners (max depth, ID collisions, deadlock) rarely — you need coverage to know what was exercised and directed tests for the corners.
"Protocol assertions or a scoreboard alone is enough." You need both — assertions check the rules continuously; the scoreboard checks the results. Each catches what the other misses.
"Out-of-order bugs that don't reproduce aren't real." They're timing-dependent, not absent — a latent assumption (in-order, hazard) that the current seed didn't expose. Reproducibility requires controlling seeds/timing, not dismissing the bug.

7. Debug Lab — A Response Arrives for a Transaction That Already Retired

Scoreboard reports more responses than requests, and the extras carry valid-looking data

DUPLICATE-RESPONSE-ID-REMAP

Observed Symptom

A multi-master regression reports intermittent scoreboard errors of the form "response for a transaction that is not outstanding." The extra responses carry plausible data — correct-looking RDATA, RRESP = OKAY. Single-master runs are clean. The rate rises with the number of masters and with outstanding depth.

Expected vs Actual

Expected. Each accepted AR/AW produces exactly one completion, and the per-ID in-flight count returns to zero.

Actual. Some IDs retire more completions than they were issued, while other IDs end the test still outstanding. The totals are close, but the per-ID books do not balance — which is the detail that identifies the failure.

Diagnostic

Three observations narrow it fast, and the third is decisive:

Is the aggregate count wrong, or only the per-ID counts? If wr_issued == wr_retired overall while individual wr_inflight[i] are non-zero and others went negative-by-detection, nothing was lost or invented — completions were attributed to the wrong ID.
Does it scale with master count rather than with traffic? A subordinate answering twice would scale with traffic. Scaling with the number of masters points at the interconnect, which is the only component that rewrites IDs.
Compare the ID at the manager port with the ID at the subordinate port. An interconnect extends AxID with master-index bits on the way out and strips them on the way back. If the widths or the strip logic disagree, a response returns under a different manager-visible ID than the request carried.

That last check is conclusive and takes one waveform: same transaction, two IDs.

Root Cause

The interconnect's ID remapping was asymmetric. Requests were tagged with master-index bits correctly; the return path stripped the wrong field width for one master, so its completions surfaced under an ID belonging to another master. The receiving manager saw a completion for an ID it had also legitimately used — hence "plausible data" — and the tracker's wr_inflight[bid] == 0 check fired on whichever manager happened to have nothing in flight for that ID at the time.

The reason this is intermittent is that it only becomes visible when the wrongly-credited ID happens to be idle. When both masters have that ID in flight, the books still balance and the corruption is silent — the response simply retires the wrong request, and the scoreboard compares the wrong pair.

Fix

The repair is in the interconnect's ID width and strip logic; the verification lesson is that the check must live at both ports. Bind the tracker at the manager port and at each subordinate port:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bind axi_master     axi_outstanding_tracker #(.ID_W(4))  u_ot_mgr (.*);
bind axi_slave_port axi_outstanding_tracker #(.ID_W(6))  u_ot_sub (.*);   // extended IDs

Each side independently balances its own books. An interconnect that conserves transactions keeps both balanced; one that remaps asymmetrically breaks exactly one — and which one tells you the direction of the fault.

Prevention

Treat conservation as a first-class property. Transactions in equals transactions out, per port and per ID, and the tracker's final drain block is the cheapest possible statement of it.

Then stress the axis that exposes remapping: the same ID value used concurrently by different masters — §4's "ID collisions across masters" corner. Random stimulus under-hits it because independent masters rarely choose colliding IDs unaided, so constrain for it explicitly. A design where ID collision never occurs in regression has never tested its ID extension logic, and that logic is exactly where this class of bug lives.

8. Debugging Insight

🔍 Debugging insight — concurrency bugs are intermittent and need controlled reproduction

Debugging concurrent-AXI failures differs from single-transaction debug:

A failure that reproduces only on some seeds? It's a timing/interleaving-dependent bug — a specific completion order or depth triggers it. Capture and replay the exact seed and freeze the response ordering; don't dismiss it as flaky. The interleaving that failed is the test case.
Scoreboard mismatch where data is "right" but paired wrong? Check whether the scoreboard (or DUT) matched by position instead of ID — the classic out-of-order checker bug. Verify the matching keys on BID/RID.
Intermittent wrong value at one address? A same-address different-ID hazard raced — the order, hence result, varied with timing. Reproduce by forcing the two completion orders and confirm both are "legal" per AXI (which means the test/DUT relied on an order it shouldn't).
Hang only under high load? A deadlock/backpressure corner — reproduce by saturating all channels and use the per-channel progress to find which channel can't drain.

The meta-lesson: make the failing interleaving deterministic (seed + forced ordering), then it debugs like any other case. The hard part of concurrent debug is reproduction, not analysis — invest the tooling to replay exact orderings.

9. Verification Insight

✅ Verification insight — the complete strategy in one place

Concurrent AXI verification is the integration of everything in Module 8, so the strategy combines all the per-chapter checks into one environment: an actively-reordering, multi-master constrained-random stimulus; protocol assertions/VIP for every rule (handshake stability, same-ID order, 4 KB, exclusive, strobes); a byte-accurate, ID-matching scoreboard that models the ordering contract and memory and flags hazards; and functional coverage of the concurrency space (depths, ID mixes, interleavings, corners) driving closure. Make runs reproducible (seed-controlled, with the ability to replay a specific completion ordering) so intermittent failures become deterministic test cases.

Prioritize the corners that random under-hits with directed tests: max outstanding depth with overflow, cross-master ID collisions, same-ID across slaves, maximally-reordered responses, overlapping-address hazards, and deadlock/backpressure saturation. The single biggest leverage point is the scoreboard design — get the ID-matching, per-ID ordering, byte-accurate memory, and hazard detection right, and it validates the entire model under any interleaving; get it wrong and the whole environment gives false confidence. Coverage closure on the concurrency axes is what lets you claim the (un-enumerable) space was actually exercised. This is the capstone of AXI verification — the rest of the protocol's rules feed into checks that run inside this concurrent harness.

10. Interview Questions

11. Summary

Verifying concurrent AXI is the hardest part of the protocol because correctness depends on interleavings and timing, and the state space — outstanding depth × ID combinations × completion interleavings × masters × address relationships — is too large to enumerate. So verification shifts to sample, check, cover: constrained-random, multi-master, actively-reordering stimulus; protocol assertions for the rules; a byte-accurate, ID-matching scoreboard for the results; and functional coverage to prove the space was exercised, with directed tests for the corners random under-hits (max depth, cross-master ID collisions, same-ID across slaves, maximal reordering, address hazards, deadlock, backpressure).

Two things carry disproportionate weight. The scoreboard is the highest-leverage component — get ID-matching, per-ID ordering, byte-accurate memory, and hazard detection right and it validates the whole model under any interleaving; get it wrong and the environment gives false confidence. And reproducibility (seed control + forced orderings) is what turns intermittent, timing-dependent failures into deterministic, debuggable, permanently-covered test cases. This closes Module 8: the concurrency model is only as trustworthy as the harness that exercises it. Next, Module 9 sharpens the ordering rules themselves — the precise read/write ordering and dependency guarantees that these harnesses must encode.

12. Where This Is Specified

The concurrency model this chapter verifies is normative in the Arm AMBA AXI Protocol Specification — transaction identifiers and ordering in §A5, and the channel handshake rules in §A3. Arm publishes it on the AMBA AXI documentation page.

The point worth confirming in the specification, because §5's tracker depends on it, is what the standard does not say: there is no maximum outstanding depth, and no bound on how long a response may take. Both are integration agreements. A verification environment absolutely should check them — a breach overflows a real queue — but they belong in a different category from the ordering and handshake rules, and conflating the two is how a performance finding gets mis-filed as a protocol bug.

The SVA and covergroups above are IEEE Std 1800 (SystemVerilog) — clause 16 for assertions, clause 19 for functional coverage. See concurrent assertions for the sampling semantics, and associative arrays for the ID-keyed scoreboard structure §2 describes.

Related lessons. The ordering half of the checker is same-ID ordering; the reordering it must tolerate is different-ID ordering; the response semantics being matched are BRESP and read data interleaving. For the scoreboard build-out see AXI scoreboards, for the rule checks AXI assertions, and for the stimulus side constrained-random AXI.

13. What Comes Next

You've completed the concurrency model and how to verify it; Module 9 sharpens the ordering and dependency rules:

9.1 — Read & Write Ordering (coming next) — the precise per-ID ordering guarantees for reads and writes, and the dependencies a master must enforce.

Previous: 8.5 — Interconnect Implications. Related: 8.4 — Different-ID & Out-of-Order Completion and 8.3 — Same-ID Ordering Rules — the rules this harness must check. For the broader protocol catalog, see the AMBA family overview doc.