Configuration Over CXL.io

Chapter 7.1 listed configuration as one of CXL.io's five jobs and moved on. This chapter stops there, because "the host reads and writes configuration space" hides every decision that makes a device work or brick.

A configuration register is not storage. It is a contract — and a single field with the wrong access type can make a device undiscoverable in a way that looks like a link problem.

1. The Engineering Problem — A Register Is Not a Variable

Software writes a value. It reads it back. It matches. Everything looks correct.

That test passes on a register file where reserved bits are writable, where a status field can be overwritten by software, where a command bit fires an action every cycle it stays set, where two addresses reach the same storage, and where a partial write destroys three fields the access never named. Read-back-matches is compatible with all five defects.

The reason is that a register has properties a variable does not: an owner, a reset value, an access type, a side effect, and a visibility. Software's write is one participant in a contract, and hardware is the other.

2. The One-Sentence Model

A configuration register is the hardware/software contract table — every field declares who owns it, what it resets to, what a write means, and what a write does — and configuration bugs are almost always a field whose declared contract and implemented behaviour disagree.

Call it the contract: not what the register holds, but what each side is allowed to do to it.

3. What This Chapter Owns

Deliberately not repeated: the PCIe configuration mechanism and header layout, which the PCIe track covers in depth. This chapter is about what happens inside the device when one of those accesses lands.

4. What a Field Declares

Five properties, and every configuration bug in this chapter is one of them being wrong.

The access type is where most of the damage lives, because there are only a few and they are easy to conflate.

W1C is the one that gets implemented as RW, and it is the one whose failure loses evidence rather than corrupting data. §9 shows why.

The five properties are not independent checks scattered through a design — they sit on one path, and each stage of that path owns one of them.

The response node is the one that looks optional and is not. A configuration access is non-posted in both directions, so every accepted access owes exactly one response — which is why RTL 5 counts them and why 7.5 makes the posted/non-posted split its subject.

5. Teaching-model boundary

6. RTL 1 — The Contract, in Logic

  // Teaching layout of one 32-bit control register:
  //   [7:0]   RW    software-owned control
  //   [15:8]  RO    hardware-owned status
  //   [16]    W1C   sticky event, set by hardware, cleared by writing 1
  //   [31:17] RSVD  must read zero and must not be writable
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rw_q <= 8'h00; ro_q <= 8'hA5; w1c_q <= 1'b0; rsvd_q <= 15'd0;
      clr_req_q <= 1'b0;
      ro_changed_err <= 1'b0; rsvd_set_err <= 1'b0; w1c_not_cleared_err <= 1'b0;
    end else begin
      // --- RW: byte-enabled, software-owned
      if (sw_write && sw_be[0]) rw_q <= sw_wdata[7:0];
 
      // --- RO: hardware-owned. A software write must not change it.
      if (RO_WRITABLE && sw_write && sw_be[1]) ro_q <= sw_wdata[15:8];
 
      // --- W1C: hardware SETS, software clears by writing 1.
      // Hardware set must win a same-cycle race, or an event is lost.
      if (hw_event)                                   w1c_q <= 1'b1;
      else if (sw_write && sw_be[2] && sw_wdata[16])  w1c_q <= W1C_AS_RW ? 1'b1 : 1'b0;
      else if (W1C_AS_RW && sw_write && sw_be[2])     w1c_q <= sw_wdata[16];
 
      // --- RSVD: must stay zero.
      if (RSVD_WRITABLE && sw_write && sw_be[3]) rsvd_q <= sw_wdata[31:17];
 
      // ---- diagnostics, stated about the OUTPUT
      if (ro_q != ro_reset)                       ro_changed_err      <= 1'b1;
      if (rsvd_q != 15'd0)                        rsvd_set_err        <= 1'b1;
      // The property is about the cycle AFTER the write: on the write cycle
      // itself w1c_q has not had an edge to change on, so checking it there
      // flags the correct behaviour. Register the request and check the result.
      clr_req_q <= sw_write && sw_be[2] && sw_wdata[16] && !hw_event;
      if (clr_req_q && w1c_q) w1c_not_cleared_err <= 1'b1;
    end
  end

State. Four independently-owned fields plus three sticky diagnostics. Each field's if has a different guard, and that asymmetry is the contract.

Synthesis. Thirty-two flops, four write-enable terms and three comparators. Trivial — which is exactly why register bugs ship: nothing about the logic is hard, so nothing about it gets reviewed.

DV. Every field type needs its own negative test: write a value to RO and confirm it does not move; write zero to W1C and confirm it does not clear; write ones to RSVD and confirm it stays zero. Read-back-matches tests none of those.

  after reset : rdata=0000a500
  write 0000003C : rdata=0000a53c
  write 0000FF00 : RO field=a5 (must stay A5) | RO-writable=ff
  write FFFE0000 : RSVD=0 (must stay 0) | RSVD-writable=7fff
  hardware event : W1C bit=1
  write 0 to W1C : bit=1 (must stay 1)
  write 1 to W1C : bit=0 (must be 0) | W1C-as-RW bit=1

Every one of those values is asserted, not printed. The RO-writable=ff and RSVD-writable=7fff columns are the deliberately-broken variants, which exist so the diagnostics have a reachable failing case.

7. RTL 2 — Decoding, and the Alias

  localparam logic [11:0] A_CTRL = 12'h010, A_STATUS = 12'h014, A_CAP = 12'h018;
 
  // The bug shape: decoding only the low 8 bits makes 0x110 alias 0x010.
  // Decode on the WORD address so alignment is a separate question -- a
  // decoder that folds the two together makes its own alignment check
  // unreachable, because a misaligned address can never equal an aligned one.
  assign eff = IGNORE_UPPER ? {4'h0, addr[7:2], 2'b00} : {addr[11:2], 2'b00};
 
  assign misaligned = acc_valid && (addr[1:0] != 2'b00);
 
  assign sel_ctrl   = acc_valid && (eff == A_CTRL)   && (ALLOW_UNALIGNED || !misaligned);
  assign sel_status = acc_valid && (eff == A_STATUS) && (ALLOW_UNALIGNED || !misaligned);
  assign sel_cap    = acc_valid && (eff == A_CAP)    && (ALLOW_UNALIGNED || !misaligned);
 
  assign hit         = sel_ctrl || sel_status || sel_cap;
  assign unsupported = acc_valid && !hit && !misaligned;

  addr 010 : ctrl=1 status=0 cap=0 hit=1
  addr 110 : correct hit=0 unsupported=1 | alias-decoder hit=1
  addr 012 : misaligned=1 hit=0 | allow-unaligned hit=1

Address 0x110 is the whole lesson. A decoder comparing only the low byte sees 0x10 and selects the control register. Software using the documented address never notices; anything that walks the space — enumeration, a memory test, a dump tool — writes to a register it did not intend to touch.

An alias is worse than an unmapped address because an unmapped address is reported. unsupported exists so that a walk over the space produces a refusal rather than a silent hit somewhere unexpected.

8. RTL 3 and 4 — Partial Writes and One-Cycle Commands

  always_comb begin
    merged = data_q;
    if (IGNORE_BE) merged = wdata;
    else begin
      if (be[0]) merged[7:0]   = wdata[7:0];
      if (be[1]) merged[15:8]  = wdata[15:8];
      if (be[2]) merged[23:16] = wdata[23:16];
      if (be[3]) merged[31:24] = wdata[31:24];
    end
  end

  full write : data=aabbccdd
  write byte0 only : data=aabbcc11 (upper bytes must survive)
  ignore-BE variant : data=00000011

A register holding four independent fields is normal, and software writing one of them is normal. A design that writes the whole word on every access destroys the other three — and the corruption is invisible whenever software happens to write the full word, which is most of the time in a directed test.

Then the command register:

  // The command bit is stored so software can read back what it asked for.
  // The ACTION is a one-cycle pulse derived from the write, not from the bit.
  assign fire   = sw_write && sw_cmd_bit;
  assign action = LEVEL_NOT_PULSE ? cmd_q : fire;

  one command write : action pulses=1 writes=1
  five cycles later : action pulses=1 | level-variant pulses=5

One write, one action. The level variant fires once per cycle for as long as the bit stays set — five actions from one command, and in a real device that is five DMA kicks, five resets, or five of whatever the command starts.

The invariant is that actions never outnumber commands, which is checkable with two counters and no knowledge of what the command does.

9. Waveform — Four Accesses Against the Real RTL

Three things are worth reading off it.

The write at cycle 3 appears in rdata at cycle 4, not cycle 3 — a register updates on the edge, so a read in the same cycle as a write returns the old value. That is correct and it is a routine source of confusion in bring-up.

The W1C clear at cycle 6 leaves rdata at 0000a53c, not 0000a500. The clear enabled only byte 2, so the RW field's 3c survives. A clear that enabled all four bytes would have zeroed it — correct behaviour, and a good demonstration of why byte enables exist.

action is high for exactly one cycle at cycle 8, and err is high for exactly one cycle at cycle 10. Both are one-cycle events by construction, and both are counted so the invariant is checkable.

10. RTL 5 and 6 — Responses, and the Race

A configuration write is non-posted — it requires a completion. So the device owes exactly one response per accepted access.

  assign accepted  = acc_valid && acc_ready && (pending_q < DEPTH[3:0]);
  assign rsp_c     = NEVER_RESPOND ? 1'b0 : (svc_done && (pending_q != 4'd0));
 
      // Responses may never outnumber accepted accesses.
      if (n_rsp_q > n_acc_q)                 double_response_err <= 1'b1;
      if (svc_done && (pending_q == 4'd0))   response_without_access_err <= 1'b1;

  3 accesses accepted : pending=3
  3 responses : pending=0 accesses=3 responses=3
  accesses and responses matched an oracle            : ok
  9 accesses into depth 4 : pending=4
  response with nothing pending : detected=1 pending=0

Two responses to one access is data corruption in software's view; zero is a hang. The pending_q != 0 guard is what makes the second response a detected error rather than a decrement of a zero counter — mutation M10 removes it and the count wraps to fifteen.

Then the race that loses evidence:

  always_comb begin
    if (CLEAR_WINS) begin
      if      (sw_clear) next_sticky = 1'b0;
      else if (hw_set)   next_sticky = 1'b1;
      else               next_sticky = sticky_q;
    end else begin
      if      (hw_set)   next_sticky = 1'b1;
      else if (sw_clear) next_sticky = 1'b0;
      else               next_sticky = sticky_q;
    end
  end

  simultaneous set+clear : set-wins sticky=1 | clear-wins sticky=0
  events=1 observable=1 | clear-wins events=1 observable=0

An event that happened cannot be un-happened by a clear that was issued before it arrived. Software's clear refers to the events it has already read; a hardware event arriving in the same cycle is new information, and losing it means nothing anywhere records that it occurred.

The n_set_q versus n_observed_q pair is what makes the loss measurable rather than arguable: events that happened, and events software could have seen. On the correct design they are equal. Chapter 7.3 builds the whole observability story on this distinction.

11. Assertions

Concurrent SVA execution: NOT SUPPORTED BY Icarus Verilog — the only simulator available here. The properties below are bind-ready and were not executed; each maps to a procedural stand-in and a mutation.

// SAFETY -------------------------------------------------------------------
// V1 — a read-only field never changes, whatever software writes.
a_ro_stable: assert property (@(posedge clk) disable iff (!rst_n)
  $stable(ro_q));
 
// V2 — reserved bits read zero, always.
a_rsvd_zero: assert property (@(posedge clk) disable iff (!rst_n)
  rsvd_q == '0);
 
// V3 — writing one to a W1C field clears it by the next cycle, unless
//      hardware set it in the same cycle.
a_w1c_clears: assert property (@(posedge clk) disable iff (!rst_n)
  (sw_write && sw_be[2] && sw_wdata[16] && !hw_event) |=> !w1c_q);
 
// V4 — writing zero to a W1C field never clears it.
a_w1c_zero_noop: assert property (@(posedge clk) disable iff (!rst_n)
  (sw_write && sw_be[2] && !sw_wdata[16] && w1c_q) |=> w1c_q);
 
// V5 — a byte whose enable is low is unchanged by the write.
a_be_respected: assert property (@(posedge clk) disable iff (!rst_n)
  (wr && !be[0]) |=> $stable(data_q[7:0]));
 
// V6 — at most one register is selected.
a_decode_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({sel_ctrl, sel_status, sel_cap}));
 
// V7 — a hit implies the word address is a mapped offset.
a_no_alias: assert property (@(posedge clk) disable iff (!rst_n)
  hit |-> ({addr[11:2],2'b00} inside {A_CTRL, A_STATUS, A_CAP}));
 
// V8 — an access reaches a register, or is reported unsupported or misaligned.
a_access_total: assert property (@(posedge clk) disable iff (!rst_n)
  acc_valid |-> (hit || unsupported || misaligned));
 
// V9 — a command write produces exactly one action pulse.
a_one_pulse: assert property (@(posedge clk) disable iff (!rst_n)
  action |=> !action);
 
// V10 — responses never outnumber accepted accesses.
a_no_double_response: assert property (@(posedge clk) disable iff (!rst_n)
  n_rsp_q <= n_acc_q);
 
// V11 — a hardware set wins a same-cycle software clear.
a_set_wins: assert property (@(posedge clk) disable iff (!rst_n)
  hw_set |=> sticky_q);
 
// LIVENESS -----------------------------------------------------------------
// V12 — an accepted configuration access eventually produces a response.
//       ENVIRONMENT ASSUMPTION: the service side completes. A configuration
//       write is non-posted, so violating this is a software hang rather than
//       a silent loss — which is why the NEVER_RESPOND variant exists.
a_eventual_response: assert property (@(posedge clk) disable iff (!rst_n)
  accepted |-> s_eventually rsp_valid);

V1 is deliberately written as $stable rather than as an implication over writes. A read-only field must not change for any reason — not merely "not because software wrote it" — and the stronger form catches a hardware path that was never supposed to exist.

V4 is the pair to V3 and the one that is usually missing. V3 alone is satisfied by a field that clears on every write, which is exactly the W1C-as-RW bug.

12. Mutation Testing

Thirteen mutations. Clean code restored after each.

13/13 killed, 0 escaped

One escaped on the first run. M11 accepts accesses past capacity, and the original test only issued three into a depth-four tracker — so capacity was never reached. Nine accesses killed it.

Two defects were found in my own RTL before mutation testing began, and both are recorded in the callouts above: a W1C checker that fired on correct behaviour, and a decoder that folded alignment into its address comparison and thereby made its own alignment checker unreachable. Neither would have been caught by mutation — the first because it failed on the good design, the second because an unreachable checker produces an equivalent mutant.

13. Verification Plan

Rows 5 and 9 are the ones that are usually missing. Write-zero-does-not-clear is what separates W1C from RW, and a plan with only the write-one case passes the buggy design.

14. Debug Lab

// Sticky event bit.
if (sw_write && sw_be[2]) w1c_q <= sw_wdata[16];

  write 0 to W1C : bit=1 (correct: stays set)
  W1C-as-RW variant : bit=0 after the same write

if (hw_event)                                  w1c_q <= 1'b1;
else if (sw_write && sw_be[2] && sw_wdata[16]) w1c_q <= 1'b0;   // only a ONE clears

// Compare the offset within the register block.
assign eff = {4'h0, addr[7:0]};

  addr 110 : correct hit=0 unsupported=1 | alias-decoder hit=1

assign eff = {addr[11:2], 2'b00};       // full word address
if (hit && ({addr[11:2],2'b00} inside {A_CTRL, A_STATUS, A_CAP}) == 0)
  alias_err <= 1'b1;

// The command bit drives the action.
assign action = cmd_q;

  one command write : action pulses=1 (correct)
  five cycles later : level-variant pulses=5

assign fire   = sw_write && sw_cmd_bit;   // the EVENT
assign action = fire;                     // one cycle, by construction
if (n_action_q > n_write_q) multi_pulse_err <= 1'b1;

// Store the written word.
if (wr) data_q <= wdata;

  write byte0 only : correct=aabbcc11 | ignore-BE=00000011

if (be[0]) merged[7:0]   = wdata[7:0];
if (be[1]) merged[15:8]  = wdata[15:8];
if (be[2]) merged[23:16] = wdata[23:16];
if (be[3]) merged[31:24] = wdata[31:24];

if      (sw_clear) sticky_q <= 1'b0;
else if (hw_set)   sticky_q <= 1'b1;

  simultaneous set+clear : set-wins sticky=1 | clear-wins sticky=0
  events=1 observable=1  | clear-wins events=1 observable=0

if      (hw_set)   sticky_q <= 1'b1;    // the event wins
else if (sw_clear) sticky_q <= 1'b0;
if (hw_set && !next_sticky) event_lost_err <= 1'b1;

// Service completed; send the response.
assign rsp_valid = svc_done;

  response with nothing pending : detected=1 pending=0   (correct)
  mutated design : pending decrements from 0 and wraps to 15

assign rsp_c = svc_done && (pending_q != 4'd0);
if (n_rsp_q > n_acc_q)               double_response_err <= 1'b1;
if (svc_done && (pending_q == 4'd0)) response_without_access_err <= 1'b1;

15. Design Review

• Who owns each field, and what resets it?
• Can software race hardware on any bit? Which side wins, and why that side?
• Is any field with a side effect driven from the stored bit rather than the write?
• Can two addresses reach the same register?
• What happens to a malformed or misaligned access — hit, refusal, or nothing?
• Does a partial write preserve the bytes it did not enable?
• Can a configuration access produce zero responses, or two?
• Which registers survive reset, and is that intentional for every one of them?
• Is any register crossing a clock domain, and what synchronises it?

16. How This Appears in Real Engineering

Register bugs ship because the logic is trivial. Nothing in a register file is architecturally hard, so nothing gets reviewed with the attention given to a datapath — and the failure modes are subtle, load-dependent and easy to blame on software.

W1C-as-RW is the most common single defect in this class. It survives because the obvious test — write one, observe clear — passes on the broken design. Only the write-zero case separates them.

Aliasing is found by tools, not by drivers. Drivers use documented offsets. Diagnostic dumps, memory tests and enumeration walk the space, which is why the corruption first appears when someone tries to debug something else.

The hardware/software race scales the wrong way. More polling should mean better observability and instead means more lost events, so the symptom actively misleads the person investigating it.

17. Common Misconceptions

18. Interview Reasoning

19. Exercises

1. Implement. Write a register with an RW field, an RO field and a W1C field, then write the three negative tests that distinguish it from a design where all three are RW. State which of your tests the buggy design passes.

2. Calculate. A 12-bit configuration space is decoded on the low 8 bits. How many distinct addresses alias onto each mapped register? Now decode on the low 10 bits and recompute. State the general relationship.

3. RTL task. Add a fourth access type: a field hardware may update at any time and software may also write, with a defined winner. State the new invariant and which existing assertion must be weakened.

4. DV task. Write the coverage cross for register access, and explain why crossing address with read/write alone leaves the byte-enable and W1C bugs uncovered.

5. Debug task. Hardware telemetry reports more errors than software sees, and the gap widens as polling frequency increases. Give your investigation order and name the single counter pair that settles it.

6. Design review. A colleague proposes clear-on-read for the status register "to save an access". Give the strongest version of the argument, then name what it costs and which specific tool it breaks.

20. Summary

A configuration register is a contract, and configuration bugs are a contract and an implementation disagreeing.

• Every field declares an owner, reset value, access type, side effect and visibility. Getting one wrong is the whole defect class.
• W1C is not RW. Writing zero must do nothing, or read-modify-write on neighbouring fields destroys evidence.
• A command is an event, not a level — one write, exactly one action, checkable with two counters.
• A decoder must compare the whole address, and alignment is a separate question — folding them makes the alignment check unreachable.
• Partial writes must preserve the bytes they did not enable, and full-word tests cannot see the difference.
• A configuration write is non-posted: exactly one response per accepted access, never zero, never two.
• Hardware set wins a same-cycle software clear, and the loss is only measurable with an events-versus-observable counter pair.
• Verification lesson: read-back-matches proves almost nothing. The coverage is in the negative tests.

Chapter 7.3 takes the sticky bit from §10 and asks the harder question: how do you report errors and events to software without losing evidence, and without drowning it?