Fabric Managers

3.3 introduced the fabric: switches, multiple hosts, and a controller called the fabric manager.

Module 12 built the pool — which host gets which capacity, and the policy that decides.

This chapter builds the manager itself: the authority it has, the authority it does not have, and the machinery a configuration change actually needs when the fabric it is changing is already carrying traffic.

1. The Engineering Problem — The Thing That Configures The Fabric Is Not On It

A fabric manager is a control plane. That phrase is used loosely enough to be worthless, so here is what it has to mean in silicon.

It owns connectivity and nothing else. It binds a device to a host, unbinds it, sets a route, enumerates what is present. It does not decide who holds a cache line in what state — it has no view of any line. A manager that can command a coherency state is a manager that can corrupt memory from outside the coherency protocol, and section 5 builds the boundary as a checkable expression rather than a paragraph.

It changes a fabric that is already running. Rebinding a device is not a register write. Traffic is in flight, and the reconfiguration must not land underneath it. Section 6 builds the quiesce sequence and measures what it costs.

Its path is its own. The whole point of out-of-band is that the manager's commands do not queue behind the traffic they are about to stop. A shared path produces the specific deadlock where the command to drain the fabric is waiting for the fabric to drain. Section 8 builds both and shows the difference.

A change either happens or it does not. Half a configuration is a fabric where some elements route by the new map and some by the old. Sections 9 and 12 build the transaction and the version agreement that make "half" impossible to observe.

Exactly one manager owns it. Two is not redundancy, it is two authorities issuing conflicting configurations to elements that cannot tell them apart. Section 10 builds the ownership.

And losing it must not stop traffic. A management fault that becomes a data outage is a control plane that was never really separate. Section 11 builds the coupling and the version that does not have it.

This chapter against 12.1, stated precisely. Module 12 owns which host should get what. This chapter owns the machinery that applies that decision to a live fabric safely. If a section here could be moved into Module 12 without loss, it is in the wrong chapter.

2. The One-Sentence Model

A fabric manager is an out-of-band owner of connectivity that changes a running fabric as a transaction — quiesce it, validate the change, apply it to every element, resume — and every failure in this chapter is one of those five words missing.

Call it own, quiesce, validate, apply, resume. Every defect below is a change applied without ownership, without quiescing, without validation, to only some elements, or without ever resuming.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

The RTL below is a teaching model. It is not a CXL fabric manager and does not implement any FM command set.

What is faithful: the authority boundary, the quiesce-before-change ordering, the separation of the management path from the data path, the propose–validate–commit–rollback shape, single ownership, and the version agreement across elements. Those are structural, and a real implementation has all of them under different names.

What is not: command encodings, transport, timing, the number of elements, and every width. A real fabric has hundreds of elements and a real quiesce takes far longer than the models below.

Every model is parameterised so that the correct behaviour and a specific plausible failure are the same source under a different parameter, instantiated together, driven by one stimulus stream. The difference between them is measured in the transcript, not asserted in prose.

5. RTL 1 — What The Manager May Command

Connectivity is the manager's. Coherency is not. The manager has no view of any cache line, so a command that names one is a command it could not have computed.

module fm_authority #(parameter int FM_TOUCHES_COHERENCY = 0) (
  input  logic clk, rst_n,
  input  logic       cmd,
  input  logic [2:0] cmd_kind,
  output logic       allowed, refused,
  output logic       overreach_err,
  output logic [7:0] n_allowed, n_refused, n_overreach
);
  localparam logic [2:0] C_BIND=3'd0, C_UNBIND=3'd1, C_ROUTE=3'd2,
                         C_ENUM=3'd3, C_LINESTATE=3'd4, C_PERM=3'd5;
  logic in_scope;
  assign in_scope = (cmd_kind == C_BIND)  || (cmd_kind == C_UNBIND)
                 || (cmd_kind == C_ROUTE) || (cmd_kind == C_ENUM);
  assign allowed = cmd && (in_scope || (FM_TOUCHES_COHERENCY != 0));
  assign refused = cmd && !allowed;
  assign overreach_err = allowed && !in_scope;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_allowed <= 8'd0; n_refused <= 8'd0; n_overreach <= 8'd0;
    end else if (cmd) begin
      if (allowed) n_allowed <= n_allowed + 8'd1;
      else         n_refused <= n_refused + 8'd1;
      if (overreach_err) n_overreach <= n_overreach + 8'd1;
    end
  end
endmodule

And the oracle it is checked against, written with no reference to that expression:

  function integer o_scope(input integer k);
    begin
      o_scope = 0;
      if (k==0) o_scope = 1;   // bind
      if (k==1) o_scope = 1;   // unbind
      if (k==2) o_scope = 1;   // route
      if (k==3) o_scope = 1;   // enumerate
      // 4 set-line-state and 5 grant-permission are coherency, not connectivity
    end
  endfunction

Six command kinds are driven at both builds. Four are connectivity and four are allowed. Two — setting a line's state, granting a permission — are coherency, and the correct build refuses both:

  authority: allowed=4 refused=2 of 6 driven | overreach correct=0 permissive=1

The check is written against an oracle — an explicit list of what connectivity means, written as integers with no reference to the design's expression — so that a mutation which rewrites the expression is caught by something that does not share its structure.

overreach_err is the interesting output. In the correct build it is unreachable: allowed implies in_scope. It is not dead code, it is the monitor for the permissive build, which accepts "set this line to Modified" and thereby reaches into a coherency domain it cannot see. That is the boundary this section exists to draw.

6. RTL 2 — Changing A Fabric That Is Carrying Traffic

A rebind under live traffic is a transaction landing on a route that is being rewritten underneath it. The fix is ordering, and the ordering has a name at every step.

module quiesce_gate #(parameter int SKIP_DRAIN = 0) (
  input  logic clk, rst_n,
  input  logic       req_change, quiesce_ack, change_done,
  input  logic [3:0] in_flight,
  output logic [2:0] phase,
  output logic       may_change,
  output logic       changed_live_err,
  output logic [7:0] drain_cycles, max_drain, n_changes
);
  localparam logic [2:0] IDLE=3'd0, QUIESCE=3'd1, DRAIN=3'd2,
                         CHANGE=3'd3, RESUME=3'd4;
  logic [2:0] ph_q;
  logic [7:0] drn_q;
  assign phase        = ph_q;
  assign drain_cycles = drn_q;
  // The whole ordering, in one expression: CHANGE is the only phase in which
  // anything may change. SKIP_DRAIN adds DRAIN, and that is the bug.
  assign may_change = (ph_q == CHANGE)
                      || ((SKIP_DRAIN != 0) && (ph_q == DRAIN));
  assign changed_live_err = may_change && (in_flight != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ph_q <= IDLE; drn_q <= 8'd0; max_drain <= 8'd0; n_changes <= 8'd0;
    end else begin
      case (ph_q)
        IDLE:    if (req_change) begin ph_q <= QUIESCE; drn_q <= 8'd0; end
        // Asking the fabric to stop is not the fabric having stopped.
        QUIESCE: if (quiesce_ack) ph_q <= DRAIN;
        DRAIN:   begin
                   if (in_flight == 4'd0) ph_q <= CHANGE;
                   else begin
                     drn_q <= drn_q + 8'd1;
                     // The peak is latched, so a later shorter drain cannot
                     // lower the worst case an operator has to plan around.
                     if (drn_q + 8'd1 > max_drain) max_drain <= drn_q + 8'd1;
                   end
                 end
        CHANGE:  if (change_done) begin
                   ph_q <= RESUME; n_changes <= n_changes + 8'd1;
                 end
        RESUME:  ph_q <= IDLE;
        default: ph_q <= IDLE;
      endcase
    end
  end
endmodule

Three separate guards, and each is a different mistake if it is missing:

• QUIESCE waits for an acknowledgement. The manager asking the fabric to stop is not the fabric having stopped. The transcript holds the phase at 1 across three cycles to prove the request alone does not advance it.
• DRAIN waits for in_flight to reach zero. The drain is as long as the traffic takes, and the model latches the longest one it has seen.
• CHANGE is the only phase in which anything may change. SKIP_DRAIN is the build that changes during DRAIN, and it is caught with three transactions still in flight.

  quiesce  : drain=3 latched=3 | changed-live correct=0 skip-drain=1 held quiescing=1

The peak is latched, not sampled: a later, shorter drain must not lower it. The testbench drives exactly that — a second change with one transaction in flight — and asserts the latched value is unchanged. A "maximum" that a quiet interval can reduce is a maximum of the last interval, not of the run.

7. Waveform — Ten Cycles Of A Change

Transcribed from the printed trace of the parameterised pair. Both builds see one stimulus stream.

Read the two error rows against each other. live_err stays flat for the whole run because the correct build's may_change is false everywhere in_flight is non-zero. skip_err fires for exactly the three cycles the fabric spent draining, and stops at cycle 7 not because the faulty build got better but because the traffic finally left.

8. RTL 3 — Out Of Band Means Its Own Path

This is the section that justifies the phrase. The manager's readiness must not depend on the data path's occupancy.

module fm_oob_path #(parameter int SHARED_PATH = 0) (
  input  logic clk, rst_n,
  input  logic       fm_send, data_send,
  input  logic       fm_taken, data_taken,
  output logic       fm_ready, data_ready,
  output logic       fm_blocked_err,
  output logic [3:0] fm_depth, data_depth,
  output logic [7:0] n_fm, n_data, fm_wait, max_fm_wait
);
  localparam int DEPTH = 4;
  logic [3:0] f_q, d_q;
  logic [7:0] wait_q;
  assign fm_depth   = f_q;
  assign data_depth = d_q;
  assign fm_wait    = wait_q;
  // Out of band means exactly this: the FM's readiness does not depend on the
  // data path's occupancy. SHARED_PATH couples them.
  assign fm_ready   = (f_q < DEPTH[3:0])
                      && ((SHARED_PATH == 0) || (d_q < DEPTH[3:0]));
  assign data_ready = (d_q < DEPTH[3:0]);
  assign fm_blocked_err = fm_send && !fm_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      f_q <= 4'd0; d_q <= 4'd0; wait_q <= 8'd0;
      n_fm <= 8'd0; n_data <= 8'd0; max_fm_wait <= 8'd0;
    end else begin
      if (fm_send && fm_ready) begin
        f_q <= f_q + 4'd1; n_fm <= n_fm + 8'd1; wait_q <= 8'd0;
      end else if (fm_send && !fm_ready) begin
        wait_q <= wait_q + 8'd1;
        if (wait_q + 8'd1 > max_fm_wait) max_fm_wait <= wait_q + 8'd1;
      // Guarded: an unguarded decrement wraps the depth to fifteen and the
      // fabric believes it has messages nobody sent.
      end else if (fm_taken && f_q != 4'd0) f_q <= f_q - 4'd1;
 
      if (data_send && data_ready) begin
        d_q <= d_q + 4'd1; n_data <= n_data + 8'd1;
      end else if (data_taken && d_q != 4'd0) d_q <= d_q - 4'd1;
    end
  end
endmodule

The stimulus fills the data path completely, then sends one management command:

  oob path : fm_ready with data full correct=1 shared=0 | blocked=1 shared wait latched=3

That single line is the whole argument. With the data path full, the out-of-band build accepts the command; the shared build refuses it and starts counting a wait. Now name the command: it is the command to quiesce the fabric. The shared build has arranged for the instruction to stop the traffic to be queued behind the traffic. The fabric drains only when the traffic finishes on its own, which is to say the manager has no control over it at the one moment control was the point.

The FM path is not unbounded, and the model says so. Its queue has a depth, a fifth message is refused rather than silently dropped, the refusal is counted as a wait, and an accepted message clears the wait counter without disturbing the latched worst case:

  fm queue : depth 4 accepted=4, fifth refused, longest wait=3

Taking from an empty queue is guarded. Without the guard the depth wraps to fifteen and the fabric believes it has management messages to deliver that were never sent — a fault that looks like corruption and is actually an unguarded decrement.

9. RTL 4 — A Configuration Change Is A Transaction

Not a write. Propose, validate, commit — or roll back leaving the fabric exactly as it was.

module config_txn #(parameter int COMMIT_WITHOUT_VALIDATE = 0,
                    parameter int EAGER_APPLY = 0) (
  input  logic clk, rst_n,
  input  logic       propose,
  input  logic [3:0] new_cfg,
  input  logic       validated, invalid, commit_ack, abort_req,
  output logic [3:0] active_cfg,
  output logic [2:0] stage,        // 0 idle 1 proposed 2 validated 3 committing
  output logic       committed, rolled_back,
  output logic       unvalidated_commit_err,
  output logic       half_applied_err,
  output logic [7:0] n_committed, n_rolled_back
);
  localparam logic [2:0] IDLE=3'd0, PROP=3'd1, VALID=3'd2, COMMIT=3'd3;
  logic [2:0] st_q;
  logic [3:0] cfg_q, pend_q;
  logic       applied_q;
  assign active_cfg = cfg_q;
  assign stage      = st_q;
  assign committed   = (st_q == COMMIT) && commit_ack;
  assign rolled_back = (st_q != IDLE) && (invalid || abort_req);
  assign unvalidated_commit_err = (st_q == COMMIT) && (COMMIT_WITHOUT_VALIDATE != 0);
  assign half_applied_err = (st_q == IDLE) && applied_q && (cfg_q != pend_q);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      st_q <= IDLE; cfg_q <= 4'd0; pend_q <= 4'd0; applied_q <= 1'b0;
      n_committed <= 8'd0; n_rolled_back <= 8'd0;
    end else begin
      case (st_q)
        IDLE:  if (propose) begin
                 pend_q <= new_cfg;
                 // EAGER_APPLY writes the proposal out to the elements now,
                 // before anything has validated or committed it.
                 if (EAGER_APPLY != 0) applied_q <= 1'b1;
                 st_q <= (COMMIT_WITHOUT_VALIDATE != 0) ? COMMIT : PROP;
               end
        PROP:  begin
                 if (invalid || abort_req) begin
                   // A rollback must rewind what the elements hold, not just
                   // the manager's own record of it.
                   st_q <= IDLE;
                   if (EAGER_APPLY == 0) pend_q <= cfg_q;
                   n_rolled_back <= n_rolled_back + 8'd1;
                 end else if (validated) st_q <= VALID;
               end
        VALID: begin
                 if (abort_req) begin
                   st_q <= IDLE; pend_q <= cfg_q;
                   n_rolled_back <= n_rolled_back + 8'd1;
                 end else st_q <= COMMIT;
               end
        COMMIT: if (commit_ack) begin
                  cfg_q <= pend_q; applied_q <= 1'b1; st_q <= IDLE;
                  n_committed <= n_committed + 8'd1;
                end
        default: st_q <= IDLE;
      endcase
    end
  end
endmodule

Four properties, each with its own stimulus:

• The proposal does not change the fabric. active_cfg stays at its old value through PROPOSED and VALIDATED, and the transcript holds the stage at 1 for three cycles to prove validation is required rather than merely usual.
• The commit needs the fabric's acknowledgement. In COMMIT with commit_ack low, committed is 0 and active_cfg is unchanged. The acknowledgement is what signals the commit, not arrival in the state.
• A rollback restores. Both the failed-validation path and the abort-after-validation path leave active_cfg at 7 — the configuration the fabric started on.
• An abort with nothing open rolls nothing back. rolled_back is gated on being inside a transaction, and driving an abort from IDLE does not increment the rollback count.

The fourth output is the invariant that spans all of it. half_applied_err says: outside a transaction, what the manager records as active and what the elements were last told must be the same thing. In the correct build it never fires, which raises the question of how you know the check works at all.

The answer is EAGER_APPLY — a build that writes the proposal out to the elements as soon as it is proposed, before anything has validated it, and whose rollback therefore rewinds its own record and nothing else:

  invariant: half-applied correct=0 eager build=1 | idle abort rolled back=0

The eager build's rollback leaves the elements holding configuration 3 while its own record says the fabric is on 7. That is the state no complete transaction produces, and it is exactly what "half applied" means when it stops being a phrase.

10. RTL 5 — Exactly One Manager

Two managers is not redundancy. It is two authorities issuing conflicting configurations to elements that cannot tell them apart.

module fm_ownership #(parameter int FAULT_INJECT = 0) (
  input  logic clk, rst_n,
  input  logic       claim, release_own, heartbeat,
  input  logic [1:0] mgr_id,
  input  logic       cfg_cmd,
  input  logic [1:0] cmd_from,
  output logic       owned,
  output logic [1:0] owner,
  output logic       accept_cmd,
  output logic       two_manager_err, foreign_cmd_err, unowned_cmd_err,
  output logic [7:0] n_claims, n_accepted, n_rejected
);
  logic       own_q;
  logic [1:0] id_q;
  assign owned = own_q;
  assign owner = id_q;
  assign accept_cmd = cfg_cmd && own_q
                      && ((FAULT_INJECT != 0) || (cmd_from == id_q));
  assign two_manager_err = claim && own_q && (mgr_id != id_q);
  assign foreign_cmd_err = cfg_cmd && own_q && (cmd_from != id_q);
  assign unowned_cmd_err = cfg_cmd && !own_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      own_q <= 1'b0; id_q <= 2'd0;
      n_claims <= 8'd0; n_accepted <= 8'd0; n_rejected <= 8'd0;
    end else begin
      // Only the owner can hand the fabric away, and a claim cannot overwrite
      // an existing owner -- it is reported instead.
      if (release_own && own_q && (mgr_id == id_q)) own_q <= 1'b0;
      else if (claim && !own_q) begin
        own_q <= 1'b1; id_q <= mgr_id; n_claims <= n_claims + 8'd1;
      end
      if (accept_cmd)                  n_accepted <= n_accepted + 8'd1;
      else if (cfg_cmd && !accept_cmd) n_rejected <= n_rejected + 8'd1;
    end
  end
endmodule

Three distinct violations, and the model keeps them distinct because they have different causes and different fixes:

  ownership: two_manager=1 foreign=1 unowned=1 | faulty build accepted a foreign cmd=1

Note what gates acceptance in the correct build. It is ownership, not the sender's identity. The testbench drives a command from sender 0 at a fabric nobody owns — and sender 0 happens to match the reset owner id, so a design that checked only the identity would accept it. The correct build refuses, because the fabric has no owner to have sent it.

Releasing is guarded the same way: a manager that does not own the fabric cannot hand it away. The stimulus drives a release from manager 2 while manager 1 owns it, confirms ownership is unchanged, then drives the real owner's release and confirms it takes.

FAULT_INJECT is the build that accepts from anyone. Under identical stimulus it accepts the foreign command the correct build refused, which is the moment two managers begin configuring one fabric without either of them knowing.

11. RTL 6 — Losing The Manager

The manager is a control plane. Losing it must stop reconfiguration and must not stop traffic. A design that couples the two has turned a management fault into a data outage.

module fm_liveness #(parameter int FM_GATES_TRAFFIC = 0) (
  input  logic clk, rst_n,
  input  logic       fm_alive,
  input  logic       traffic_req, cfg_req,
  output logic       traffic_ok, cfg_ok,
  output logic       outage_err,   // traffic stopped by a management fault
  output logic [7:0] n_traffic, n_traffic_lost, n_cfg_blocked,
                     down_cycles, max_down
);
  logic [7:0] dn_q;
  assign down_cycles = dn_q;
  assign traffic_ok = traffic_req && (fm_alive || (FM_GATES_TRAFFIC == 0));
  assign cfg_ok     = cfg_req && fm_alive;
  assign outage_err = traffic_req && !traffic_ok;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      dn_q <= 8'd0; max_down <= 8'd0;
      n_traffic <= 8'd0; n_traffic_lost <= 8'd0; n_cfg_blocked <= 8'd0;
    end else begin
      if (traffic_ok)           n_traffic      <= n_traffic + 8'd1;
      if (outage_err)           n_traffic_lost <= n_traffic_lost + 8'd1;
      if (cfg_req && !fm_alive) n_cfg_blocked  <= n_cfg_blocked + 8'd1;
      if (!fm_alive) begin
        dn_q <= dn_q + 8'd1;
        // How long the fabric ran with nobody able to change it.
        if (dn_q + 8'd1 > max_down) max_down <= dn_q + 8'd1;
      end else dn_q <= 8'd0;
    end
  end
endmodule

Two lines, and they say opposite things on purpose. traffic_ok does not mention fm_alive in the correct build. cfg_ok mentions nothing else.

  liveness : outage correct=0 gating build=1 | down latched=5 cfg blocked=5

Across a five-cycle management outage the correct build serves every traffic request and blocks every configuration attempt. The gating build loses all five. The outage length is latched — a later, shorter outage does not reduce it — because what an operator needs is the worst interval the fabric ever ran unmanaged, not the last one.

Both counters matter and they answer different questions. n_cfg_blocked is how much management work was lost, which is recoverable: the manager comes back and the work is redone. n_traffic_lost is how much user traffic was lost, which is not.

12. RTL 7 — Every Element On The Same Version

An element still running an old configuration is not merely stale. It is routing traffic by a map the rest of the fabric has replaced.

module config_version #(parameter int NO_VERSION_CHECK = 0) (
  input  logic clk, rst_n,
  input  logic       push,
  input  logic [3:0] new_ver,
  input  logic       elem_ack,
  input  logic [1:0] elem_id,
  input  logic       traffic,
  input  logic [1:0] traffic_elem,
  output logic [3:0] cur_ver, acked_mask,
  output logic       all_acked,
  output logic       stale_element_err,
  output logic       split_fabric_err,
  output logic [7:0] n_pushes, n_stale, skew_cycles, max_skew
);
  logic [3:0] ver_q, ack_q;
  logic [7:0] skew_q;
  assign cur_ver     = ver_q;
  assign acked_mask  = ack_q;
  assign skew_cycles = skew_q;
  assign all_acked   = (ack_q == 4'b1111);
  assign stale_element_err = traffic && !ack_q[traffic_elem]
                             && (NO_VERSION_CHECK == 0);
  assign split_fabric_err = !all_acked && (ack_q != 4'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ver_q <= 4'd0; ack_q <= 4'b1111; skew_q <= 8'd0;
      n_pushes <= 8'd0; n_stale <= 8'd0; max_skew <= 8'd0;
    end else begin
      if (push) begin
        // A new version invalidates every acknowledgement, including the ones
        // for the version it replaces.
        ver_q <= new_ver; ack_q <= 4'd0; skew_q <= 8'd0;
        n_pushes <= n_pushes + 8'd1;
      end else begin
        if (elem_ack) ack_q[elem_id] <= 1'b1;
        if (!all_acked) begin
          skew_q <= skew_q + 8'd1;
          // The interval in which the fabric ran two configurations at once.
          if (skew_q + 8'd1 > max_skew) max_skew <= skew_q + 8'd1;
        end
      end
      if (stale_element_err) n_stale <= n_stale + 8'd1;
    end
  end
endmodule

Two different failures, and the distinction is the point of the section:

• Stale is per-element and per-access: traffic reached an element that has not taken the current version. It is judged at the element the traffic is going to — the testbench drives traffic to an un-acked element while a different element is fully current, precisely so that a check reading the wrong element fails.
• Split is a property of the fabric as a whole: some elements have taken the new version and some have not. It is asserted while two of four have acknowledged, and it clears when the fourth does.

  version : split=1 stale correct=1 no-check=0 | skew latched=6 pushes=1

The skew is the measurement that turns this from a boolean into something an architect can use. It is the number of cycles the fabric spent with elements disagreeing, and its peak is latched. A fabric with a large worst-case skew is a fabric where a configuration change is observable as a routing inconsistency for that long, whether or not any traffic happened to hit it.

NO_VERSION_CHECK is the build that reports nothing. Under the same stimulus that raises one stale access in the correct build, it raises none — traffic routed by a replaced map, silently.

13. RTL 8 — Auditing Every Change

A manager with authority and no audit is a single point from which every element in the fabric can be made wrong at once.

  always_comb begin
    legal = 1'b0;
    case (op)
      O_BIND:   legal = !dev_bound_mask[dev];
      O_UNBIND: legal =  dev_bound_mask[dev] && !dev_in_use;
      O_ROUTE:  legal =  dev_bound_mask[dev];
      default:  legal = 1'b0;
    endcase
  end
  assign illegal_cfg_err = apply_cfg && !legal;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_audited <= 8'd0; n_illegal <= 8'd0; run_q <= 8'd0; worst_run <= 8'd0;
    end else if (apply_cfg) begin
      n_audited <= n_audited + 8'd1;
      if (!legal) begin
        n_illegal <= n_illegal + 8'd1;
        run_q <= run_q + 8'd1;
        // A run is a manager working from a stale view of the fabric; an
        // isolated one is a race. The distinction changes the response.
        if (run_q + 8'd1 > worst_run) worst_run <= run_q + 8'd1;
      end else run_q <= 8'd0;
    end
  end
endmodule

Three rules, and each is a real mistake rather than a decoration:

• Binding a device that is already bound. Two hosts believing they own one device — the fabric-scale form of 13.3's two-owner failure.
• Unbinding a device that is not bound. A manager operating from a view of the fabric that no longer matches it.
• Unbinding a device with traffic still on it. The transactions in flight have nowhere to land.

The rules are checked against an oracle written as plain integers, swept across every operation and every device, before any change is applied.

  audit   : illegal=3 of 3 audited, worst run=3 | bind-bound=1 unbind-in-use=1

worst_run is the interesting statistic, and it exists because an isolated illegal change and a run of them mean different things. One is a race. Three in a row is a manager working from a stale view of the fabric, and it will keep producing them until its view is refreshed. The run counter therefore resets on a legal change and the peak is latched — the testbench drives three illegal, one legal, then two more illegal, and asserts the worst run is still 3 rather than 5.

14. RTL 9 — What Managing The Fabric Costs

Every model above measures something. This one turns the measurements into the two ratios an architect actually asks for.

  assign attempts    = {1'b0, n_changes} + {1'b0, n_rollbacks};
  assign weighted_rb = {16'd0, n_rollbacks} * 32'd100;
  assign rollback_pct  = (attempts == 17'd0) ? 8'd0
                       : (weighted_rb / {15'd0, attempts});
  assign weighted_q  = {16'd0, total_quiesce} * 32'd100;
  assign mean_change   = (n_changes == 16'd0) ? 8'd0
                       : ({16'd0, total_change}  / {16'd0, n_changes});
  assign mean_quiesce  = (n_changes == 16'd0) ? 8'd0
                       : ({16'd0, total_quiesce} / {16'd0, n_changes});
  // The share of fabric time spent quiesced: the cost the manager imposes on
  // everyone else, which the change count alone does not show.
  assign quiesce_share_pct = (n_traffic == 16'd0) ? 8'd0
                           : (weighted_q / {16'd0, n_traffic});
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_changes <= 16'd0; n_rollbacks <= 16'd0; total_change <= 16'd0;
      total_quiesce <= 16'd0; n_traffic <= 16'd0;
    end else begin
      if (change_done) begin
        n_changes     <= n_changes + 16'd1;
        total_change  <= total_change  + {8'd0, change_cycles};
        total_quiesce <= total_quiesce + {8'd0, quiesce_cycles};
      end
      if (rolled_back)   n_rollbacks <= n_rollbacks + 16'd1;
      if (traffic_cycle) n_traffic   <= n_traffic + 16'd1;
    end
  end
endmodule

  cost    : changes=2 rollbacks=2 (50%) mean change=10 quiesce=3 | fabric quiesced 30% of the time

Two numbers, and they answer questions the raw counts do not.

The rollback rate is a property of the manager. Half the attempts rolled back means the manager is proposing configurations that do not validate — its view of the fabric disagrees with the fabric's.

The quiesce share is a property imposed on everyone else. Thirty percent of the fabric's traffic cycles were spent stopped. The change count alone hides this completely: two changes sounds cheap until you see what the fabric paid for them.

Three width disciplines are load-bearing here and each is a real defect if it is missing. Percentages are computed in 32-bit intermediates because a 16-bit count multiplied by 100 overflows. attempts is 17 bits because two 16-bit counts can sum past 16. And both ratios guard the empty sample by returning 0, not 100 — an unrun fabric has not rolled back everything, and the testbench asserts that before any attempt is driven.

The guard is for a genuinely empty sample and nothing wider. A second cost model driven with exactly one traffic cycle and one quiesce cycle reports a 100 percent quiesce share, which is correct and meaningful: the fabric was stopped for all of the time it was asked to carry anything.

15. RTL 10 — The Manager Assembled

Own, quiesce, validate, apply, resume — with the one invariant that spans all five.

module fm_top #(parameter int NO_RESUME = 0) (
  input  logic clk, rst_n,
  input  logic       own, req, validated, drained, applied,
  output logic [2:0] stage,   // 0 idle 1 quiesce 2 validate 3 apply 4 resume
  output logic       fabric_open,
  output logic       stuck_quiesced_err,
  output logic [7:0] n_done, quiesced_cycles, max_quiesced
);
  localparam logic [2:0] IDLE=3'd0, QUIESCE=3'd1, VALIDATE=3'd2,
                         APPLY=3'd3, RESUME=3'd4;
  logic [2:0] st_q;
  logic [7:0] qc_q;
  assign stage           = st_q;
  assign quiesced_cycles = qc_q;
  // Closed for the whole transaction, reopened at the end. NO_RESUME never
  // reopens it: a fabric that stops permanently on a change that succeeded.
  assign fabric_open = (st_q == IDLE);
  assign stuck_quiesced_err = (st_q == RESUME) && (NO_RESUME != 0);
 
      case (st_q)
        IDLE:     if (own && req) begin st_q <= QUIESCE; qc_q <= 8'd0; end
        QUIESCE:  if (drained)   st_q <= VALIDATE;
        VALIDATE: if (validated) st_q <= APPLY;
        APPLY:    if (applied)   st_q <= RESUME;
        RESUME:   if (NO_RESUME == 0) begin
                    st_q <= IDLE; n_done <= n_done + 8'd1;
                  end
        default:  st_q <= IDLE;
      endcase

And the interval measurement that runs underneath the whole case statement:

      if (st_q != IDLE) begin
        qc_q <= qc_q + 8'd1;
        if (qc_q + 8'd1 > max_quiesced) max_quiesced <= qc_q + 8'd1;
      end

Every guard in that case statement is a section of this chapter compressed to one condition, and the testbench holds the machine in each state to prove the guard is doing work rather than being incidentally satisfied. A change is requested with own low and the machine does not move. It sits in QUIESCE until drained, in VALIDATE until validated, in APPLY until applied.

  assembled: open during a change=0 closed cycles latched=10 | no-resume stuck=1
             unowned request ignored (stage=0) | single-cycle quiesce share=100%

fabric_open is the invariant. The fabric is closed for the whole transaction and reopened at the end, and the interval is measured rather than assumed — ten cycles, latched.

NO_RESUME is the build that never reopens it. Everything about that build succeeds: it owned the fabric, quiesced correctly, validated, applied. And its fabric is closed permanently, with n_done at zero. That is the failure worth ending on, because it is the one that looks least like a failure at every step until the last one.

16. Quantitative Reasoning

Every number below is printed by the models above, not asserted here.

Three of these deserve a sentence each.

Zero traffic lost against five. Same stimulus, same outage, one parameter apart. That is the entire argument for keeping the control plane out of the data path, and it is a measurement rather than a principle.

Fifty percent rollback with a thirty percent quiesce share. These are independent and both are bad in different ways. A manager could have a zero percent rollback rate and still stop the fabric for a third of its life.

A worst run of 3, not 5. Six illegal changes were driven in total. If the run counter did not reset on the legal change between them, the worst run would read 5 and an operator would conclude the manager's view had been stale for twice as long as it was.

17. Assertions

Icarus Verilog 13.0 does not support concurrent SVA, so every property below is an immediate check inside the testbench. Each is written so that x fails rather than passing vacuously — cond !== 1'b1 rather than !cond — and each is sampled after a settle so that a continuous assignment read in the delta its driver changes cannot produce a false result.

18. Mutation Testing

A passing testbench proves nothing until you have broken the design and watched it fail. 101 mutations were injected into the three source files, one at a time, each of which had to make the baseline print RESULT: FAIL.

101 of 101 were killed.

The first run killed 84 and left 17 survivors, and the classification is more instructive than the final number.

Fifteen of the seventeen were the testbench's fault, not the design's. Among them: the FM queue was never filled, so its depth limit and its wait counter were untested; a command was never sent to an unowned fabric from a sender whose id matched the reset owner, so the ownership gate was masked by the identity check; the assembled machine was never held in VALIDATE or APPLY, so two of its three guards were never shown to be load-bearing.

The two genuinely unreachable ones were both half_applied_err. In the correct build the invariant cannot fire — that is what makes it an invariant. Making the mutations observable required two changes: applied_q became sticky, so the invariant applies forever after the fabric is first configured rather than being cleared by the next proposal; and an EAGER_APPLY build was added that writes the proposal out before validating it. With both, removing the rollback's restore makes the correct build fail, and hardcoding the monitor to zero makes the eager build's violation disappear.

A representative sample of the 101:

19. Verification Strategy

The chapter's models are verified by five techniques, and each exists because the one before it has a specific blind spot.

Parameterised twin builds. One source under a parameter, two instances, one stimulus stream. FM_TOUCHES_COHERENCY, SKIP_DRAIN, SHARED_PATH, COMMIT_WITHOUT_VALIDATE, EAGER_APPLY, FAULT_INJECT, FM_GATES_TRAFFIC, NO_VERSION_CHECK, NO_RESUME. The difference between correct and broken is then a measured number in the transcript rather than a claim in prose, and neither build can be accused of having had an easier stimulus.

Independent oracles. The authority list, the audit rules and the version agreement are each checked against a function written in plain integers with no reference to the design's expression. A mutation that rewrites the design's case statement is caught by something that does not share its structure.

Latched peaks, sampled after the event. Every maximum in this chapter — drain, FM wait, outage, skew, illegal run, closed interval — is latched, and every one is re-tested with a shorter later event to prove the peak does not fall. A peak that a quiet interval can lower is the last interval, not the worst.

Deliberately-broken builds for unreachable monitors. Section 18 covers the two that needed it.

Delta discipline. Every sample of a combinational output is preceded by a settle. This is not fastidiousness. In a $display a missing settle corrupts a transcript; in a check it produces a false failure, which is worse, because it sends you hunting a design bug that does not exist.

20. Synthesis and Implementation Reality

The models synthesise, and what they cost is worth stating because it bears on where a real manager's logic lives.

The phase machines are trivial. Five states, one-hot or binary, a handful of flops. quiesce_gate and fm_top are each smaller than a single FIFO pointer pair. Nothing in this chapter's control is expensive.

The counters dominate, and they are diagnostic. Every latched peak and every ratio is observability, not function. On a real element they belong behind a configuration bit so they can be compiled out or clock-gated, and their widths need to survive a long run rather than a testbench — the 8-bit counters here would be 32 or 48 bits in silicon, and the ratio intermediates scale with them.

The divisions are not synthesisable as written. rollback_pct and quiesce_share_pct use /, which is fine for a model and wrong for an element. In silicon these are computed by firmware reading the raw counters, or by a small sequential divider that runs once per sampling interval. The reason they appear as combinational divisions here is that the chapter is about what to measure; how to compute it is an implementation decision that should not be made in the teaching model.

The audit is combinational and it is on the configuration path, not the data path. That is the whole reason it can afford to be thorough. A rule set that would be unaffordable per-packet is free per-configuration-change, because configuration changes are rare by construction — and section 14 measures exactly how rare.

The management path is physically separate. In the models that is fm_oob_path's two independent queues. In a real system it is a different interface entirely, and section 8's transcript is the argument for why the separation has to be physical rather than a priority bit on a shared path. A priority bit does not help when the shared resource is already full.

21. Silicon Observability

What a real fabric element should expose, taken directly from what the models count.

Three of these are the ones to alarm on rather than merely log.

n_traffic_lost non-zero means the control plane and the data plane are coupled somewhere, and no amount of management-side reliability work will fix it.

worst_run greater than one means the manager has been operating from a stale view of the fabric, and the fix is refreshing its view rather than retrying the change.

max_skew growing across runs means elements are taking longer and longer to accept configurations, which is the early form of an element that will eventually stop accepting them.

22. Debug Lab

Four failures, each reproducible in the models, each with a distinguishing signature. In every case the discipline is the same: find the counter that separates the candidate causes before touching the design.

22.1 The fabric will not drain

Symptom. A configuration change was requested twenty seconds ago. The manager reports it as in progress. Traffic is still flowing.

The wrong move. Increase the drain timeout.

The signature. Read max_fm_wait on the management path.

If it is zero, the quiesce command reached the fabric and the fabric is genuinely still draining — something is not finishing, and the question is what. Look at in_flight and find which element is not retiring.

If it is non-zero and growing, the quiesce command has not been delivered at all. It is queued behind the traffic it was sent to stop. That is section 8's failure, and it is a wiring problem, not a timeout problem. A longer timeout makes it take longer to discover.

The two look identical from the manager's side — a change that will not complete — and the counter separates them in one read.

22.2 Two hosts see the same device

Symptom. A device is responding to two hosts. Memory corruption follows.

The candidates, and the counter that picks one.

The fourth row is the one worth internalising. A double-binding looks like a manager fault, and if all three management counters are clean it is not one, and every hour spent in the fabric manager's logs is wasted.

22.3 Traffic stopped when the manager restarted

Symptom. The fabric manager was restarted for a routine update. Traffic stopped for the duration.

The reading. n_traffic_lost non-zero, max_down equal to the restart duration.

The diagnosis. This is FM_GATES_TRAFFIC. Something on the data path is consulting the manager — a route lookup that requires the manager to answer, a credit that only the manager can return, a keep-alive that gates forwarding. The fabric is not tolerating the loss of its control plane because the control plane is on the data path.

Why it is worth flagging even when it is brief. A three-second restart is survivable and a three-second unplanned outage may not be, and they are the same fault. The counter does not distinguish between a planned restart and a crash, and neither does the fabric.

22.4 Some traffic is being routed by the old configuration

Symptom. A configuration change completed successfully. A small fraction of traffic is still following the previous route.

The reading. n_stale non-zero, max_skew large, acked_mask not all ones.

The diagnosis. The change committed at the manager and did not reach every element. The fabric is split: some elements route by the new map, some by the old, and the traffic that fails is the traffic that happened to cross the boundary.

The two sub-cases, and the one that matters. If acked_mask is now complete and max_skew is merely large, the change eventually landed everywhere and the failures were transient — the fabric's skew window is longer than the manager assumed, and the fix is not declaring a change complete until every element has acknowledged.

If acked_mask is still incomplete, an element has not taken the configuration at all and is not going to. That is not a skew problem. That element is either wedged or unreachable, and every second of continued traffic is traffic routed by a map the rest of the fabric replaced.

The distinction is one read of all_acked and it changes the response completely: wait, or evacuate.

23. Design Review

Ten questions for a fabric-manager design, each with the specific failure it is looking for.

1. Can the manager command anything a coherency agent would command? If yes, there is a path to corrupting memory from outside the coherency protocol. The boundary must be an expression that can be checked, not a convention.

2. Does a configuration change wait for an acknowledgement that the fabric stopped, or only for the request to be sent? The second is section 6's SKIP_DRAIN.

3. Does the management path share any resource with the data path? Queues, credits, buffers, arbitration. If yes, the command to stop the fabric can queue behind the traffic it is stopping.

4. Is a configuration change atomic from every element's point of view? Not "does the manager treat it as a transaction" — can any element ever observe a partial one? Section 12's acked_mask is what makes the answer measurable.

5. What happens when two managers both believe they own the fabric? "It cannot happen" is not an answer; the question is what the elements do when it does.

6. Does anything on the data path consult the manager? Section 22.3. The answer is usually yes and usually unintentional.

7. Is every configuration change audited against the fabric's present state? And is the audit's view of that state fresh? An audit against a stale view passes illegal changes with complete confidence.

8. What is the worst-case interval the fabric is quiesced, and who measured it? If nobody has, it is unbounded.

9. Can a rollback leave anything applied? Section 9. This is the invariant that needs a deliberately-broken build to test, which is why it is usually untested.

10. If the manager never comes back, what happens? Traffic must continue. Reconfiguration must not. Anything else is a control plane that was never separate.

24. How This Appears In Real Engineering

The out-of-band path is a hardware decision made early and regretted late. Whether the management path shares silicon with the data path is settled in the floorplan, and section 8's failure is not discoverable until the fabric is under load. Teams find it during the first congestion event, which is exactly when they need the manager and cannot reach it.

The quiesce window is the number the operations team asks for. Everything else in this chapter is engineering; max_quiesced is the number that goes into a maintenance-window plan. A team that cannot state it will be asked to guess, and the guess will be optimistic.

Split-brain handling is written after the first incident. The ownership model in section 10 is not difficult, and it is very commonly absent, because a single manager works perfectly until the day it is failed over. two_manager_err costs a flop and a comparator and it is the difference between a diagnosed incident and an unexplained one.

The audit is the thing that gets cut. It is on the configuration path, it is cheap, it catches the class of bug that is hardest to diagnose from the outside, and it is routinely omitted because the manager is trusted. The manager is one piece of software with a view of the fabric that can be stale.

Version skew is the failure that survives testing. A configuration change works in every test because the tests are small and the elements acknowledge quickly. At scale the skew window grows, and the failure appears as a small fraction of misrouted traffic that correlates with nothing except recent configuration changes.

25. Common Misconceptions

"The fabric manager controls the fabric, so it controls coherency too." No. It has no view of any cache line. It owns which devices are connected to which hosts; the coherency protocol owns what those hosts may do with the memory afterwards. Section 5 builds the boundary.

"Out-of-band just means a separate API." No. It means a separate path. An API on a shared transport is in-band with extra steps, and section 8's transcript shows exactly what it costs at the moment it matters.

"Quiescing means telling the fabric to stop." Telling is not stopping. Section 6 holds the phase at QUIESCE across three cycles to make the distinction concrete, and the drain afterwards is where the time actually goes.

"A rollback is easy — just don't commit." Only if nothing was applied before the commit. The EAGER_APPLY build did everything the correct build did except wait, and its rollback rewound its own record while the elements kept the configuration it abandoned.

"Two managers is redundancy." Two managers is two authorities. Redundancy is two managers of which exactly one owns the fabric at any instant, which requires the ownership machinery in section 10 rather than merely a second manager.

"If the manager dies, the fabric is down." Only if it was built that way. Section 11 measures both versions under one stimulus: zero traffic lost against five, one parameter apart.

"A change that the manager committed is a change that happened." It happened at the manager. Section 12 is about the interval before it happens everywhere else, and that interval is measurable, latched, and larger than teams expect.

"The audit is redundant because the manager already validates." The manager validates against its own view of the fabric. The audit validates against the fabric's present state. worst_run exists precisely to distinguish a race from a manager whose view has gone stale.

26. Interview Reasoning

Q1. What does a fabric manager actually own? Connectivity: binding, unbinding, routing, enumeration. Not coherency — it has no view of any cache line, so a command naming one is a command it could not have computed. The boundary should be an expression the hardware checks, not a convention the software honours.

Q2. Why does out-of-band have to mean a separate path rather than a separate protocol? Because the failure it prevents is congestion. With the data path full, a management command on a shared path waits, and the command that waits is the one that would have stopped the traffic. A separate protocol on a shared transport still queues.

Q3. A configuration change has been in progress for a long time. What do you read first? The management path's wait counter. Non-zero means the quiesce command never arrived and you have a coupled-path problem. Zero means the command arrived and the fabric genuinely has not drained, which is a different investigation entirely.

Q4. Why is quiescing two steps rather than one? Requesting the stop and the fabric having stopped are different events, and the traffic already in flight has to finish after the second one. Skipping either produces a reconfiguration landing underneath live transactions.

Q5. What does the drain time depend on? The traffic in the fabric, not the change. That is why it is measured and latched rather than assumed — the manager cannot bound it, only observe it.

Q6. Why must a maximum be latched rather than sampled? A sampled maximum reports the most recent interval. The worst case is what bounds a maintenance window, and a quiet period must not be able to lower it.

Q7. Why is a configuration change a transaction rather than a write? Because it touches multiple elements and can fail part-way. A write that fails part-way leaves a fabric where some elements route by the new map and some by the old, and no element can tell which case it is in.

Q8. What is the invariant that spans a configuration transaction? Outside a transaction, what the manager records as active and what the elements were last told are the same thing. It cannot fire in a correct design, which is why testing it requires a build that violates it.

Q9. How do you test a monitor that never fires? Build something that makes it fire. If you cannot construct that build, the monitor is not checkable and you have written a comment. Section 18 needed exactly this twice.

Q10. Why did applied_q have to become sticky? Because a per-transaction flag cleared at each proposal makes the invariant unobservable — every violation is erased by the next proposal before anything can check it. Sticky means the invariant applies forever after the fabric is first configured.

Q11. Two hosts are talking to one device. Is that a manager fault? Not necessarily. Read two_manager_err, foreign_cmd_err and n_illegal. If all three are clean the fabric was configured correctly and the problem is in the coherency layer, and every hour in the manager's logs is wasted.

Q12. What is the difference between two_manager_err and foreign_cmd_err? The first is a second manager claiming an owned fabric — a failover that did not wait. The second is a command arriving from a non-owner — split brain already in progress. Different causes, different fixes, so they are counted separately.

Q13. Why gate command acceptance on ownership rather than on the sender's identity? Because an identity check passes for a sender that happens to match a reset value, at a fabric nobody owns. Ownership is the property that matters; identity only matters once ownership exists.

Q14. Should a non-owner be able to release the fabric? No. Otherwise any manager can hand away a fabric it does not hold, which is a one-command path to an unowned fabric with configurations in flight.

Q15. What must happen when the fabric manager is lost? Reconfiguration stops. Traffic does not. Anything else means something on the data path consults the manager.

Q16. How do you find that coupling? Take the manager away and count traffic. The models do exactly this: five cycles down, zero lost in the correct build, five lost in the gating build, one parameter apart.

Q17. Blocked configuration attempts and lost traffic are both counted during an outage. Why separately? Blocked configuration is recoverable — the manager returns and the work is redone. Lost traffic is not. Merging them into one health number hides the distinction that determines how serious the incident is.

Q18. What is version skew? The interval during which elements are running different configurations. It starts when the manager pushes and ends when the last element acknowledges, and its peak is the length of time a configuration change is observable as a routing inconsistency.

Q19. Stale and split — what is the difference? Stale is per-access: traffic reached an element that has not taken the current version. Split is a property of the fabric: elements disagree. A fabric can be split with zero stale accesses, and that is luck rather than correctness.

Q20. Why judge staleness at the destination element? Because that is the element whose map routes the traffic. A check that reads some other element's acknowledgement can be fully satisfied while the traffic is misrouted, and the mutation that makes exactly that substitution survived until the stimulus drove the two elements to differ.

Q21. Why audit changes if the manager already validates them? The manager validates against its own view of the fabric. The audit validates against the fabric's present state. When the two disagree, the manager is confident and wrong, and only the audit notices.

Q22. What are three rules worth auditing? Binding a device that is already bound; unbinding a device that is not bound; unbinding a device with traffic still on it. Each is a real, plausible mistake with a distinct consequence.

Q23. What does a run of illegal changes tell you that a single one does not? A single one is a race. A run is a manager working from a stale view, and it will keep producing them until its view is refreshed. That is why the run resets on a legal change and its peak is latched.

Q24. Why must an empty-sample guard return zero rather than a hundred? Because a fabric that has attempted nothing has not rolled back everything. A guard returning 100 reports a catastrophic manager on a fabric that has done no work, and it is asserted before any attempt is driven.

Q25. What is the quiesce share and why is it not the change count? It is the fraction of the fabric's traffic time spent stopped. Two changes sounds cheap; thirty percent of traffic time is the cost those two changes imposed on everyone else, and the count alone hides it entirely.

Q26. Where should the percentage divisions live in silicon? Not in the element. Firmware reads the raw counters, or a small sequential divider runs once per sampling interval. Combinational division appears in the model because the chapter is about what to measure.

Q27. What is the failure mode of a manager that does everything right and never resumes? A fabric closed permanently after a configuration change that succeeded at every step. It owned the fabric, quiesced, validated, applied — and n_done is zero and no traffic moves. It is the failure that looks least like one until the last step.

Q28. Why parameterise a broken build into the same source rather than writing a separate broken module? So that the correct and broken versions cannot differ in anything except the parameter, and so that one stimulus stream drives both. The difference in the transcript is then a measurement rather than a claim, and neither build can have had an easier test.

Q29. Your audit counter reads four illegal changes and you drove three. Where do you look first? At the stimulus, not the counter. A combinational output read in the same delta its driver changes returns the previous value — and here the loop variable had not settled, so the first change was audited against the sweep's final device rather than the one written. Confirm the stimulus that reached the design was the stimulus you wrote before you debug the logic.

Q30. If you could add one register to a fabric element, which? max_skew. It is the only one that predicts a failure rather than reporting one: a skew growing across runs is an element on its way to not accepting configurations at all, and it is visible long before any traffic is misrouted.

27. Exercises

1. Add a STALE_VIEW parameter to fm_audit that makes dev_bound_mask one change out of date. Show that it produces a run of illegal changes rather than isolated ones, and that worst_run distinguishes it from a race.

2. Extend quiesce_gate with a drain timeout. Decide what happens when it expires — abandon the change, or change anyway — and build both. Measure which one leaves the fabric in a worse state.

3. Give fm_oob_path a priority bit on a shared path instead of a separate path. Show that under a full data path it does not help, and explain in one sentence why.

4. Add a second manager to fm_ownership with a proper handover: the old owner releases, the new one claims. Show that two_manager_err stays clear throughout, then remove the wait and show it fire.

5. Make config_version push a second version before the first has been fully acknowledged. Decide whether that is legal, and if it is, work out what acked_mask must mean.

6. Add a quiesce_share_pct alarm threshold to fm_cost and drive a stimulus that crosses it. Justify the threshold you chose from the numbers in section 16.

7. Break fm_top so it resumes before applying. Find the smallest change to the testbench that catches it, and check whether the existing assertions already do.

8. Take the 101-mutation suite and remove one testbench check at a time. Find the checks that no mutation depends on, and decide whether each is redundant or covering something the suite does not reach.

28. Summary

A fabric manager is an out-of-band owner of connectivity that changes a running fabric as a transaction.

Each word is a section, and each has a measured failure attached:

• Out-of-band: with the data path full, the correct build delivers a management command and the shared build queues it behind the traffic it was sent to stop.
• Owner: acceptance is gated on ownership, not identity; a second claim, a foreign command and an unowned command are three different faults counted separately.
• Connectivity: four command kinds allowed, two refused, checked against an independent oracle — the manager has no view of any cache line.
• Running: three guards before anything changes, and the skip-drain build caught reconfiguring with three transactions in flight.
• Transaction: propose, validate, commit or roll back; the eager build's rollback leaves the elements holding a configuration its own record says was never committed.

And two things the fabric must survive: losing the manager — zero traffic lost against five, one parameter apart — and a change that reaches only some elements, which is what max_skew measures and what a completed-at-the-manager change hides.

101 mutations, 101 killed. The seventeen that survived the first run were fifteen testbench gaps and two invariants that no correct build can reach, and the second kind is the one worth remembering: a monitor that says nothing was left behind will survive every mutation you write until you build something that leaves something behind.

15.2 takes the manager as given and asks what shape the fabric it manages should be.