Device Architecture

24.1 built the link interface. 24.2 built the transaction pipeline. 24.3 built the memory controller. Each of them passes its own tests. This chapter is about why that is not a device.

The three blocks do not run at the same frequency, do not come out of reset at the same moment, do not have the same datapath width, and are not adjacent on the die. Every one of those four is a boundary, and every boundary is a place where two correct blocks produce an incorrect system.

The integration failures are also the ones that are hardest to find, because every block-level test still passes after they are introduced — which is section 14, and it is the whole chapter.

1. The Engineering Problem — Four Boundaries And What Crosses Them

Blocks run at different frequencies. A signal crossing needs two synchroniser stages, which cost two cycles and are the only thing standing between the design and metastability. Section 5.

Blocks come out of reset at different moments. A block released before the block it depends on issues its first transaction to a peer that is not there. Section 6.

Blocks have different datapath widths. A 256-bit internal path onto a 64-bit interface is four beats and a quarter of the wide side's utilisation. Section 7.

And blocks are not adjacent on the die. Eight millimetres at two per cycle is four cycles of wire before any logic runs, on a path budgeted at eight. Section 11.

Then there is the counter nobody builds. Eight of twenty debug counters missing at three days each is 24 days added to a silicon debug — and it is decided before tape-out, when it looks like area. Section 12.

This chapter against 24.3, stated precisely. That one owns what one block does internally. This one owns what happens between blocks — which is why every model here is about a boundary rather than about a function, and why section 14's weak definition is a block-level sign-off.

2. The One-Sentence Model

A device works when every block passes its own tests, every domain crossing is synchronised, blocks release from reset in dependency order, the rate through each width boundary is known, the distance between blocks is in the pipeline, and the counters exist before tape-out — and every defect below is a device made of correct blocks.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one boundary. A real device top level is a clock and reset controller, a set of synchronisers, several width converters, a register interconnect, a power-management unit, a debug fabric and a floorplan, and none of that is reproduced. What is reproduced is the arithmetic each demands, and the shape of the mistake when it is skipped.

Three simplifications are worth stating. Section 5 counts synchroniser stages rather than computing a mean time between failures, which is the number that actually justifies two stages. Section 11 treats wire delay as linear in distance, which it is not below a certain length. Section 12 assigns a fixed debug cost per missing counter, which is a first-order stand-in for something highly variable. In each case the conclusion is the same and the model is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a block-level assumption carried across a boundary: one clock domain, one reset, one width, fast registers, free power transitions, adjacent blocks. Each is true inside a block and false between two, which is exactly why block-level verification cannot find any of them.

Figure 1 — Three blocks on the left that each pass, and the device on the right that does not. Everything in the middle column is unowned by any block-level testbench, which is why it is where the integration bugs live.

5. RTL 1 — A Signal Crossing Domains Needs Synchronising

// RTL 1 - crossing a clock domain. A signal that changes in one domain and is
// sampled in another needs synchronising, and the stages cost latency.
module clock_domain_crossing #(parameter int ASSUME_SYNCHRONOUS = 0) (
  input  logic clk, rst_n,
  input  logic       check_it,
  input  logic       crossing,            // source and destination differ
  input  logic [7:0] sync_stages,
  output logic [7:0] latency_cycles,
  output logic       safe,
  output logic [7:0] n_crossings, n_unsafe,
  output logic       sync_omitted_err
);
  // A synchroniser costs one destination cycle per stage; a design that assumes
  // one domain pays nothing and samples a changing signal directly.
  assign latency_cycles = (ASSUME_SYNCHRONOUS != 0) ? 8'd0
                        : (crossing ? sync_stages : 8'd0);
  assign safe = !crossing || (sync_stages >= 8'd2);
  // A real domain crossing sampled with no synchroniser in the path.
  assign sync_omitted_err = check_it && crossing && (sync_stages != 8'd0)
                            && (latency_cycles == 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_crossings <= 8'd0; n_unsafe <= 8'd0;
    end else if (check_it) begin
      n_crossings <= n_crossings + 8'd1;
      if (!safe) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Six crossings examined.

Two unsafe when the crossing is modelled — and the same two in the one-domain model, because the stage count is visible to both.

The last column is deliberately not a difference, and it is worth saying why. The safety judgement reads the declared stage count, so both builds agree; what the builds disagree about is the latency the synchroniser costs, which is what a timing budget needs and what the one-domain model reports as zero. A design can know a crossing is unsafe and still budget it as free.

Row six is the exemption that keeps the check honest. A signal that stays in one domain needs no synchroniser, so having none is correct — safe is !crossing || stages >= 2, and section 17 records that this case was demanded by a surviving mutation.

Two stages is a convention rather than a derivation here. The number that justifies it is a mean time between failures computed from the clock rates and the flop's metastability window; this model counts stages and takes two as the answer, which is the boundary this chapter draws around itself.

Two cycles is the cost and it is charged on every crossing. A design with synchronisers on both directions of a handshake pays four before any data moves — which is why a crossing is a place to move data in bulk rather than a place to exchange single signals.

6. RTL 2 — Blocks Release From Reset In Dependency Order

// RTL 2 - reset ordering. A block cannot come out of reset before the block it
// depends on, or its first transaction meets a peer that is not there yet.
module reset_ordering #(parameter int RELEASE_ALL_AT_ONCE = 0) (
  input  logic clk, rst_n,
  input  logic       release_it,
  input  logic       dependency_ready, has_dependency,
  input  logic [7:0] stage_index, dependency_stage,
  output logic       released, released_early, order_ok,
  output logic [7:0] n_releases, n_early,
  output logic       reset_order_err
);
  // A block is released when its dependency is ready, or immediately if the
  // design releases everything together.
  assign released = (RELEASE_ALL_AT_ONCE != 0) ? release_it
                  : (release_it && (!has_dependency || dependency_ready));
  assign order_ok = !has_dependency || (stage_index > dependency_stage);
  assign released_early = released && has_dependency && !dependency_ready;
  // A block released before the block it depends on.
  assign reset_order_err = release_it && released_early;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_releases <= 8'd0; n_early <= 8'd0;
    end else if (release_it) begin
      n_releases <= n_releases + 8'd1;
      if (released_early) n_early <= n_early + 8'd1;
    end
  end
endmodule

Five release attempts.

None released early when ordering is respected; two when everything releases together.

Row four is the distinction this model draws and it matters operationally. A block whose dependency happens to be ready is released correctly even though the stage numbering says it should not have been — the plan is wrong and the outcome is right. order_ok and released_early are separate outputs because they are separate defects with separate owners: one is a reset-controller programming error and the other is a schedule that has not yet failed.

Row five is the plan error that is easiest to make. Two blocks assigned the same reset stage have no ordering between them at all; whichever the controller happens to release first wins, and it will be consistent in simulation and possibly not in silicon.

Row three is the case that keeps the release honest. A block with no dependency releases immediately, and a design that made every block wait for something would never start — released is !has_dependency || dependency_ready, and section 17 records that dropping the first term was one of the mutations.

7. RTL 3 — A Wide Path Meeting A Narrow One Takes Beats

// RTL 3 - width conversion. A wide internal datapath meeting a narrow external
// one takes several beats per transfer, and the ratio is the rate.
module width_conversion #(parameter int ASSUME_SAME_WIDTH = 0) (
  input  logic clk, rst_n,
  input  logic       convert,
  input  logic [15:0] internal_bits, external_bits,
  input  logic [7:0]  cycles_per_beat,
  output logic [7:0]  beats, cycles, utilisation_pct,
  output logic        matched,
  output logic [7:0]  n_conversions, n_throttled,
  output logic        width_ignored_err
);
  logic [31:0] b_q, c_q, u_q;
  assign b_q = (ASSUME_SAME_WIDTH != 0) ? 32'd1
             : ((external_bits == 16'd0) ? 32'd0
                : (({16'd0, internal_bits} + {16'd0, external_bits} - 32'd1)
                   / {16'd0, external_bits}));
  assign beats = (b_q > 32'd255) ? 8'hFF : b_q[7:0];
  assign c_q = {24'd0, beats} * {24'd0, cycles_per_beat};
  assign cycles = (c_q > 32'd255) ? 8'hFF : c_q[7:0];
  // One beat per cycle is full rate; more beats means the wide side waits.
  assign u_q = (cycles == 8'd0) ? 32'd0
             : (({24'd0, cycles_per_beat} * 32'd100) / {24'd0, cycles});
  assign utilisation_pct = (u_q > 32'd255) ? 8'hFF : u_q[7:0];
  assign matched = (beats <= 8'd1);
  // A wide transfer costed as one beat on a narrow interface.
  assign width_ignored_err = convert && (external_bits != 16'd0)
                             && (internal_bits > external_bits) && (beats == 8'd1);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_conversions <= 8'd0; n_throttled <= 8'd0;
    end else if (convert) begin
      n_conversions <= n_conversions + 8'd1;
      if (!matched) n_throttled <= n_throttled + 8'd1;
    end
  end
endmodule

Six conversions. A 256-bit internal datapath.

Three throttled when the beats are counted; none when they are not.

A quarter of the wide side is the rate, and it is the number every bandwidth calculation upstream of the boundary needs. 24.2 §5's pipeline issues one transaction per cycle into something that accepts a quarter of one — and the pipeline's own testbench, which ends at its output port, cannot see it.

Row four separates the two ways a boundary throttles. Four beats at two cycles each is eight cycles, but the utilisation is still 25% — the extra cycles are the interface being slow, not the width being narrow. A single "cycles" number conflates them, and the fix for each is different: a wider interface against a faster one.

Row six is the direction that costs nothing. A narrow internal path onto a wide external one takes one beat, because a beat carries the whole transfer — width conversion is only a throttle in one direction, and which direction depends on where the block boundary falls.

8. RTL 4 — A Blocking Register Read Costs The Whole Path

// RTL 4 - the register path. Configuration space is reached over a slow path,
// and a datapath that blocks on it inherits that path's latency.
module register_access_path #(parameter int REGISTERS_ARE_FAST = 0) (
  input  logic clk, rst_n,
  input  logic       access,
  input  logic       blocking,
  input  logic [7:0] register_latency, datapath_budget,
  output logic [7:0] stall_cycles, effective_latency,
  output logic       within_budget,
  output logic [7:0] n_accesses, n_over,
  output logic       register_cost_ignored_err
);
  logic [15:0] s_q;
  // A blocking register read stalls the datapath for the whole path latency.
  assign s_q = (REGISTERS_ARE_FAST != 0) ? 16'd0
             : (blocking ? {8'd0, register_latency} : 16'd0);
  assign stall_cycles = (s_q > 16'd255) ? 8'hFF : s_q[7:0];
  assign effective_latency = stall_cycles;
  assign within_budget = (effective_latency <= datapath_budget);
  // A blocking access over a slow path that cost the datapath nothing.
  assign register_cost_ignored_err = access && blocking
                                     && (register_latency != 8'd0)
                                     && (stall_cycles == 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_accesses <= 8'd0; n_over <= 8'd0;
    end else if (access) begin
      n_accesses <= n_accesses + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five register accesses. A sixteen-cycle datapath budget.

Two over budget when the path is counted; none when it is not.

Configuration space sits behind a narrow, slow path and the datapath sits on the critical one, and joining them with a blocking read imports the first's latency into the second. Forty cycles of stall on a datapath budgeted at sixteen is not a register problem; it is a boundary that was crossed synchronously when it should not have been.

Row two is the fix and it is architectural rather than optimisational. A non-blocking access — post the read, continue, collect the result later — costs nothing on the datapath and costs a completion path to build. The choice is made when the boundary is drawn, and reversing it later is a redesign.

Row five is what a badly placed register block does. Two hundred cycles is a path that crosses a domain, an interconnect and an arbiter; the datapath that blocks on it stops for two hundred cycles per access, and the register in question is usually being polled.

9. RTL 5 — A Power State Costs Its Transition Every Time

// RTL 5 - low-power states. Entering and leaving a low-power state costs time,
// so a residency shorter than the transition costs more than it saves.
module power_state #(parameter int TRANSITIONS_ARE_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] residency_us, entry_us, exit_us, idle_power_saved,
  output logic [15:0] transition_us, net_saved_us, overhead_pct,
  output logic        worth_entering,
  output logic [7:0]  n_evaluations, n_wasteful,
  output logic        transition_ignored_err
);
  logic [31:0] t_q, o_q;
  // A state change costs the entry and the exit, both of them every time.
  assign t_q = (TRANSITIONS_ARE_FREE != 0) ? 32'd0
             : ({16'd0, entry_us} + {16'd0, exit_us});
  assign transition_us = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign net_saved_us = (residency_us > transition_us)
                        ? (residency_us - transition_us) : 16'd0;
  assign o_q = (residency_us == 16'd0) ? 32'd65535
             : (({16'd0, transition_us} * 32'd100) / {16'd0, residency_us});
  assign overhead_pct = (o_q > 32'd65535) ? 16'hFFFF : o_q[15:0];
  assign worth_entering = (overhead_pct <= 16'd20);
  // A state change that happened and cost nothing.
  assign transition_ignored_err = evaluate && (entry_us != 16'd0)
                                  && (transition_us == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evaluations <= 8'd0; n_wasteful <= 8'd0;
    end else if (evaluate) begin
      n_evaluations <= n_evaluations + 8'd1;
      if (!worth_entering) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Six evaluations. Six microseconds to enter and four to leave — ten either way.

Three not worth entering when the transition is costed; one when it is not.

The transition is fixed and the residency is not, so a power state has a minimum useful idle period. At ten microseconds either way, fifty microseconds is the floor — and a device whose idle gaps are twenty microseconds enters the state, pays the transition, and saves half of what it spent.

That is 23.2 §9's arithmetic in a different unit, and the pairing is worth noticing: a composed machine's bind time against its instance lifetime, and a power state's transition against its residency, are the same fixed-cost-against-variable-duration calculation at eight orders of magnitude apart.

Row four is the state that costs more than it saves. A five-microsecond gap against a ten-microsecond transition is a net loss, and a policy that enters on any idle detection will do it repeatedly — which is a power regression caused by a power feature.

10. RTL 6 — Two Clocks Need An Elastic Buffer

// RTL 6 - two clocks. The link and the memory run at unrelated rates, and the
// elastic buffer between them has to cover the difference over a burst.
module clock_ratio #(parameter int ONE_CLOCK = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] source_mhz, sink_mhz, burst_beats, buffer_entries,
  output logic [15:0] rate_gap_pct, entries_needed,
  output logic        sufficient,
  output logic [7:0]  n_sizings, n_short,
  output logic        ratio_ignored_err
);
  logic [31:0] g_q, n_q;
  // The faster side outruns the slower one over a burst, and the buffer holds
  // the difference. One clock means no difference at all.
  assign g_q = (ONE_CLOCK != 0) ? 32'd0
             : ((source_mhz > sink_mhz)
                ? ((({16'd0, source_mhz} - {16'd0, sink_mhz}) * 32'd100)
                   / {16'd0, source_mhz}) : 32'd0);
  assign rate_gap_pct = (g_q > 32'd65535) ? 16'hFFFF : g_q[15:0];
  assign n_q = ({16'd0, burst_beats} * {16'd0, rate_gap_pct}) / 32'd100;
  assign entries_needed = (n_q > 32'd65535) ? 16'hFFFF : n_q[15:0];
  assign sufficient = (buffer_entries >= entries_needed);
  // Two clocks with a real gap, sized as if they were one.
  assign ratio_ignored_err = size_it && (source_mhz > sink_mhz)
                             && (rate_gap_pct == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_short <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!sufficient) n_short <= n_short + 8'd1;
    end
  end
endmodule

Seven sizings.

Two short when the ratio is counted; none when it is not.

The buffer holds the difference between two rates over a burst, and both terms move. Row five is the same clocks with twice the burst needing twice the buffer — the elastic buffer is sized by the traffic shape as much as by the frequencies, which is why it is the hardest number in this chapter to fix after the fact.

Row four is the direction that costs nothing. A sink faster than the source never falls behind, so no elasticity is needed in that direction — which means a boundary needs a buffer on one side and not the other, and a symmetric design wastes half of it.

Row seven is a stopped source, driven because a clock that is not running is the state during bring-up and after a power-state entry. The gap is zero because nothing is being produced, which is correct and is not the same as matched clocks.

11. RTL 7 — The Distance Across The Die Costs Cycles

// RTL 7 - the distance across the die. A signal that has to travel between
// blocks takes cycles, and a flat top level pretends it does not.
module top_level_timing #(parameter int IGNORE_INTERCONNECT = 0) (
  input  logic clk, rst_n,
  input  logic       route,
  input  logic [7:0] distance_mm, mm_per_cycle, logic_cycles, budget_cycles,
  output logic [7:0] wire_cycles, path_cycles, slack_cycles,
  output logic       meets_timing,
  output logic [7:0] n_routes, n_failing,
  output logic       interconnect_ignored_err
);
  logic [15:0] w_q, p_q;
  // Distance costs pipeline stages whether or not the top level has them.
  assign w_q = (IGNORE_INTERCONNECT != 0) ? 16'd0
             : ((mm_per_cycle == 8'd0) ? 16'd255
                : (({8'd0, distance_mm} + {8'd0, mm_per_cycle} - 16'd1)
                   / {8'd0, mm_per_cycle}));
  assign wire_cycles = (w_q > 16'd255) ? 8'hFF : w_q[7:0];
  assign p_q = {8'd0, wire_cycles} + {8'd0, logic_cycles};
  assign path_cycles = (p_q > 16'd255) ? 8'hFF : p_q[7:0];
  assign slack_cycles = (budget_cycles > path_cycles)
                        ? (budget_cycles - path_cycles) : 8'd0;
  assign meets_timing = (path_cycles <= budget_cycles);
  // A path across the die costed as if the blocks were adjacent.
  assign interconnect_ignored_err = route && (distance_mm != 8'd0)
                                    && (mm_per_cycle != 8'd0)
                                    && (wire_cycles == 8'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_routes <= 8'd0; n_failing <= 8'd0;
    end else if (route) begin
      n_routes <= n_routes + 8'd1;
      if (!meets_timing) n_failing <= n_failing + 8'd1;
    end
  end
endmodule

Six routes. Two millimetres per cycle, six cycles of logic, an eight-cycle budget.

Four failing when the distance is counted; none when it is not.

A flat top level is a floorplan assumption disguised as an RTL structure. Connecting two blocks with a wire and no register between them asserts that they are adjacent — and the assertion is made in RTL, checked at synthesis, and discovered at place-and-route, by which point moving the blocks is the only remaining option.

Row four is why the ceiling matters. Five millimetres at two per cycle is three cycles, not two and a half — a pipeline stage is not divisible, so the rounding is always up and always costs a full cycle. A floorplan that puts blocks 4.1 mm apart pays the same as one at 6 mm.

Row six is the case that decides a partitioning. Twenty millimetres is ten cycles of wire on a path budgeted at eight, and no amount of logic optimisation recovers it — the answer is either more pipeline stages, which changes the protocol between the blocks, or a different floorplan.

12. RTL 8 — The Counters Cost Area Before And Time After

// RTL 8 - visibility. Counters and trace cost area before tape-out and are the
// only thing that shortens a silicon debug afterwards.
module debug_visibility #(parameter int TRUST_THE_RTL = 0) (
  input  logic clk, rst_n,
  input  logic        estimate,
  input  logic [15:0] counters_present, counters_needed, base_debug_days,
  input  logic [7:0]  days_per_missing,
  output logic [15:0] missing, added_days, debug_days,
  output logic        acceptable,
  output logic [7:0]  n_estimates, n_costly,
  output logic        visibility_ignored_err
);
  logic [31:0] a_q, d_q;
  assign missing = (counters_needed > counters_present)
                   ? (counters_needed - counters_present) : 16'd0;
  // Every counter that is not there is time spent inferring what it would have
  // said. A design that trusts the RTL budgets none of it.
  assign a_q = (TRUST_THE_RTL != 0) ? 32'd0
             : ({16'd0, missing} * {24'd0, days_per_missing});
  assign added_days = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign d_q = {16'd0, base_debug_days} + {16'd0, added_days};
  assign debug_days = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  assign acceptable = (debug_days <= 16'd30);
  // Counters that are missing and cost nothing.
  assign visibility_ignored_err = estimate && (missing != 16'd0)
                                  && (days_per_missing != 8'd0)
                                  && (added_days == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_estimates <= 8'd0; n_costly <= 8'd0;
    end else if (estimate) begin
      n_estimates <= n_estimates + 8'd1;
      if (!acceptable) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six estimates. Twenty counters wanted, three days each to work around a missing one.

Two too costly when the counters are budgeted; none when they are not.

This is the model this batch has been building toward for four chapters. 24.1 §21 needed a stall-on-ready counter that did not exist. 24.2 §21 needed a poison-in against poison-out pair that did not exist. 24.3 §21 needed conflicts separated from misses and a per-bank histogram, neither of which existed. Three chapters, three debugs, and in every one the missing counter was the difference between an hour and a week.

Row six is a device with no visibility at all, and seventy days is not a hypothetical: a silicon bug with no counters is found by inference from external behaviour, which is bisection over a design nobody can observe. The counters would have cost a fraction of a percent of the die.

Row five is the exemption that keeps the check honest. A counter that costs nothing to be without is one nobody would use — and visibility_ignored_err requires a real cost per missing counter, because a long list of counters nobody consults is its own kind of waste.

13. RTL 9 — The Glue Is Not Free

// RTL 9 - the area of a device. The blocks are what the plan counts and the
// glue between them is what the floorplan discovers.
module area_budget #(parameter int IGNORE_INTEGRATION = 0) (
  input  logic clk, rst_n,
  input  logic        budget,
  input  logic [15:0] block_area, glue_pct, buffer_area, budget_area,
  output logic [15:0] glue_area, total_area, overrun_area,
  output logic        fits,
  output logic [7:0]  n_budgets, n_over,
  output logic        glue_ignored_err
);
  logic [31:0] g_q, t_q;
  // Interconnect, synchronisers, width converters and test logic are area the
  // block list does not contain.
  assign g_q = (IGNORE_INTEGRATION != 0) ? 32'd0
             : (({16'd0, block_area} * {16'd0, glue_pct}) / 32'd100);
  assign glue_area = (g_q > 32'd65535) ? 16'hFFFF : g_q[15:0];
  assign t_q = {16'd0, block_area} + {16'd0, glue_area} + {16'd0, buffer_area};
  assign total_area = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign overrun_area = (total_area > budget_area)
                        ? (total_area - budget_area) : 16'd0;
  assign fits = (total_area <= budget_area);
  // Integration area that was budgeted at nothing.
  assign glue_ignored_err = budget && (glue_pct != 16'd0) && (block_area != 16'd0)
                            && (glue_area == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_budgets <= 8'd0; n_over <= 8'd0;
    end else if (budget) begin
      n_budgets <= n_budgets + 8'd1;
      if (!fits) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five budgets. Ten thousand units of block area, a thousand of buffers, a thirteen-thousand budget.

One over budget when the glue is counted; none when it is not.

Twenty percent is the number the block list does not contain, and it is made of exactly the things this chapter models: synchronisers on every crossing, converters on every width change, pipeline registers for every millimetre, the register interconnect, and the test and debug logic from section 12. None of them belongs to a block, so none of them appears on a block owner's area estimate.

Row two is the difference between a device that fits and one that respins. Thirty percent instead of twenty is a thousand units over on a thirteen-thousand budget — and the extra ten points come from more crossings and more distance, both of which are consequences of the partitioning rather than of any block's implementation.

The buffers are counted separately for a reason. Section 10's elastic buffers and 24.5's queues are sized from traffic rather than scaled from block area, so they do not belong in the glue percentage — and a model that folded them in would hide the one part of the integration area that a designer can actually calculate.

Figure 3 — Two thousand units nobody owns. The block list is complete, correct, and eighteen percent short — and the gap is discovered at floorplan, which is the last place a device wants to discover an area problem.

14. RTL 10 — A CXL Device Assembled

// RTL 10 - a CXL device assembled. Everything that must hold before blocks that
// each pass their own tests make a device that works.
module device_model #(parameter int BLOCKS_INTEGRATE = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       blocks_pass,         // every block passes its own tests
  input  logic       domains_crossed,     // every crossing is synchronised
  input  logic       reset_ordered,       // blocks release in dependency order
  input  logic       widths_converted,    // the rate through each boundary is known
  input  logic       interconnect_timed,  // distance is in the pipeline
  input  logic       visibility_built,    // the counters exist before tape-out
  output logic       works,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_works,
  output logic       false_works_err
);
  assign fail_mask[0] = ~blocks_pass;
  assign fail_mask[1] = ~domains_crossed;
  assign fail_mask[2] = ~reset_ordered;
  assign fail_mask[3] = ~widths_converted;
  assign fail_mask[4] = ~interconnect_timed;
  assign fail_mask[5] = ~visibility_built;
  // The blocks-pass build is what a block-level sign-off reports.
  assign works = (BLOCKS_INTEGRATE != 0) ? blocks_pass : (fail_mask == 6'd0);
  assign false_works_err = evaluate && works && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_works <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (works) n_works <= n_works + 8'd1;
    end
  end
endmodule

One working configuration of six, and four false claims.

"Every block passed" is what a block-level sign-off reports and it is right about one of the six. The other five are all invisible to it by construction: a block-level testbench ends at the block's ports, and every defect in this chapter lives on the far side of one.

Row five deserves separate mention because its cost is deferred rather than immediate: a device with no counters works exactly as well as one with them, until the first silicon bug — at which point section 12's seventy days arrives, and nothing can be added.

Figure 4 — Reset first, because a block released early produces symptoms that look like anything at all. The order is by how badly each defect disguises itself: an early release corrupts state before any test starts, a metastable signal is intermittent, a throttle is merely slow, and a timing failure is at least reported by a tool.

15. Quantitative Reasoning

Domain crossings. Two synchroniser stages cost two destination cycles on every crossing, and one stage is not safe.

Reset ordering. A block released before its dependency is released early; an all-at-once release did it on two of five attempts.

Width conversion. 256 bits onto 64 is four beats and 25% of the wide side's utilisation; onto 96 it is three beats.

Register path. A blocking read over a 40-cycle path stalls the datapath forty cycles against a sixteen-cycle budget; a 200-cycle path stalls two hundred.

Power states. A ten-microsecond transition on a hundred-microsecond residency is 10%; on twenty it is 50%, and fifty microseconds is exactly the threshold.

Clock ratios. 1000 MHz into 800 is a 20% gap, needing 40 entries over a 200-beat burst and 80 over a 400-beat one.

Interconnect. Eight millimetres at two per cycle is four cycles of wire, taking a six-cycle logic path to ten against a budget of eight.

Visibility. Eight of twenty counters missing at three days each is 24 days added; none at all is sixty.

Area. Ten thousand units of blocks attract 2000 of glue at twenty percent, plus a thousand of buffers — 13,000 against a block list reporting 11,000.

The assembled model. Six properties, six configurations, one works. The block sign-off reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, every ceiling both on and off its boundary, and every floor past it — the three rules this batch inherits.

Domain crossings. Exactly two stages is driven at the safety threshold, and a same-domain signal with no synchroniser is asserted safe.

chk(dGs == 1'b1, "and is safe with no synchroniser, because there is nothing to cross");

Reset ordering. A block with no dependency is asserted to release immediately, and stage numbers that are equal are asserted as not an ordering.

Width conversion. A whole ratio and a partial one are both driven, and the narrow-onto-wide direction is asserted matched.

Register path. A stall exactly at the budget is driven, and a non-blocking access over a slow path is asserted as not an ignored cost.

Power states. An overhead of exactly twenty percent is constructed from a fifty-microsecond residency, and a residency shorter than the transition is asserted to floor at zero saving.

chk(pGn2 == 16'd0,  "nothing is saved, not a wrapped saving");
chk(pGo == 16'd200, "at a two hundred percent overhead");

Clock ratios. Two buffer sizes that exactly meet the requirement are constructed, a faster sink is asserted to need nothing, and a stopped source is driven.

Interconnect. A path exactly at the budget is driven, a distance that is not a whole number of cycles is driven, and the degenerate rate case is asserted on both builds — which is what killed one of the surviving mutations.

Visibility. A debug estimate of exactly thirty days is constructed, and more counters present than needed is asserted to floor at zero missing.

Area. A total exactly at the budget is driven, and a design with no integration overhead is asserted as not ignored glue.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 261 checks across two testbenches, 120 on the front five models and 141 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty-one mutations were injected one at a time. 61 injected, 61 killed, after two survivors.

One survivor needed a new case and one needed only an assertion, which is the distinction worth keeping.

Survivor 1 — a configuration outside the model's premise, again. Section 5's safe = !crossing || (stages >= 2) survived dropping the first term because every case with crossing = 0 also had two stages. A same-domain signal with no synchroniser is the normal state of most of a design, and driving it kills the mutation.

Survivor 2 — a case already driven and half observed. Section 11's interconnect_ignored_err survived dropping its rate guard because the only case with mm_per_cycle = 0 asserted the correct build alone. In the correct build the guard is redundant — the unknown-rate path returns 255, so wire_cycles == 0 is false either way — and in the broken build it is not, because that build reports zero wire cycles regardless. Adding one assertion on the second build killed it with no new stimulus.

That is the second time in three chapters that a survivor needed only the other build asserted, and the rule is now specific enough to check mechanically: on every degenerate case, assert both builds. The correct build's silence proves nothing about the broken one's.

The complete set was re-run after every stimulus change, per standing discipline, and all sixty-one held.

18. Verification Strategy

What a testbench for an integration model must cover.

Assert both builds on every degenerate case. One of two survivors here needed nothing else, and 24.3 produced the same shape. A guard that is redundant in the correct build and load-bearing in the broken one is invisible until the broken one is checked.

Drive the configuration the model is not named about. A same-domain signal in a domain-crossing model. A block with no dependency in a reset-ordering model. A narrow-onto-wide transfer in a width-conversion model. Three such exemptions here, each one the case that keeps a check from firing on correct hardware.

Separate the plan from the outcome. Section 6's order_ok and released_early are different outputs because they are different defects: a schedule that is wrong and has not yet failed is not the same as a release that failed, and a model with one output could not say which.

The cases where the block-level assumption is right. A signal that does not cross. A block with no dependency. Matched widths. A non-blocking register access. Adjacent blocks. Matched clocks. Six cases across nine models, each exempted explicitly, and each true inside a block.

Counters as a second signature. Ten models, ten pairs of totals, differing in nine — three throttled against none, two over budget against none, four failing against none. The tenth is section 5's n_unsafe, which is deliberately equal in both builds because the safety judgement reads a declared value rather than a computed one, and saying so is more useful than manufacturing a difference.

What a real integration needs that these models do not have. A mean time between failures for section 5, a wire-delay model that is not linear for section 11, and a debug-cost estimate that is not a constant for section 12. All three are abbreviations that preserve the conclusion, and section 26 exercises 1, 6 and 7 are where they come back.

19. Synthesis and Implementation Reality

Section 5's synchronisers are two flops and a constraint file entry, and the constraint is the part that goes wrong: a crossing with the flops and without the false-path declaration is optimised through by synthesis, which removes the synchroniser and leaves the RTL looking correct.

Section 6's reset controller is a small state machine holding a release order, and the order comes from a dependency graph nobody draws. The common failure is a graph that was right and a block that acquired a new dependency later.

Section 7's converters are a shift register and a counter each, cheap in gates and expensive in the protocol they impose: a wide side that must wait four beats needs backpressure, which is 24.1 §11's ready signal on a new boundary.

Section 8's register path crosses everything — a domain, an interconnect, an arbiter — which is why its latency is measured in tens of cycles rather than ones. The architectural fix is to make it non-blocking; the tempting fix is to make it faster, and that one does not scale.

Section 10's elastic buffers are the largest structures in this chapter, and their depth is the burst length times the rate gap, which means the traffic shape sizes them and the frequencies only scale them.

Section 11's pipeline stages change the protocol between blocks, because a registered path has a latency the receiving block must tolerate. Adding one late is not a floorplan change; it is an interface change, which is why it is discovered at the worst possible time.

20. Silicon Observability

"Counters present against counters in the debug plan" is the most self-referential entry in this batch and the most useful. A debug plan is written, the RTL is written, and nobody compares them until the silicon arrives — at which point section 12's arithmetic applies and cannot be undone. The comparison costs an afternoon and is the highest-leverage review in the chapter.

21. Debug Lab

Symptom. A CXL device's first silicon boots, enumerates and passes traffic. Then: memory bandwidth is 24% of the projection, one host in ten sees a corrupted descriptor at boot, the device hangs if a low-power state is enabled, and the team cannot say why any of it happens.

Step 1 — what can be measured? Counters present against the debug plan: the plan listed twenty and the RTL has twelve. Section 12, and the eight missing ones are the boundary counters — beats per transfer, elastic buffer occupancy, reset release order. The investigation starts with the finding that it will be slow.

Step 2 — the bandwidth. With no boundary counters, the only available measurement is end-to-end. Working back through 24.3's model, the DRAM side is delivering as expected in isolation. The deficit is between the pipeline and the controller — which is a width boundary.

Step 3 — measure it externally. A directed test with a known transfer size shows four beats per internal transfer where the plan assumed one. Section 7. The pipeline is 256 bits wide, the controller interface is 64, and no upstream bandwidth calculation carried the ratio. 24% is 25% within measurement error.

Step 4 — the corrupted descriptor. One host in ten, at boot only: a reset-order symptom. The reset controller releases the link interface and the pipeline in the same stage. Section 6, and the link interface's first configuration write arrives at a pipeline whose tag pool has not been initialised. It is one in ten because it depends on link training time.

Step 5 — the hang. Power-state residency: the policy enters on any idle gap, and the measured gap distribution has a median of eight microseconds against a ten-microsecond transition. Section 9 — but the hang, rather than the inefficiency, comes from the exit path crossing a domain whose synchroniser was optimised away because the false-path constraint was missing. Section 5, and section 19's mechanism exactly.

Step 6 — why did none of this appear in verification? Every block passed. The width ratio is on the far side of a port, the reset order is above the blocks, the power exit crosses between them, and the missing counters are nobody's block. Section 14, and four of its six properties failed at once.

The finding. Four integration defects, no block defects, and the first one found was the absence of the means to find the others. Sections 12, 7, 6, 9 and 5 in that order — which is the order the missing counters forced rather than the order of severity.

The fix. Add the false-path constraint and re-verify the crossing. Move the pipeline to a later reset stage. Raise the power-state entry threshold above the transition cost. Carry the width ratio into the bandwidth model — the silicon is correct and the projection was wrong. And add the eight counters, which cannot help this device and is the only change that helps the next one.

What made this hard. Nothing was broken. Every block was correct, every block-level test passed, and the device did four things wrong — three of which were decided in a top-level file that no block owner reviewed, and one of which was a missing line in a constraint file.

22. Design Review

1. Which signals cross clock domains, and does every one have two stages and a false-path constraint? The constraint is the half that gets forgotten. Sections 5 and 19.

2. What is the reset release order, and where is the dependency graph that produced it? A graph that was right and a block that gained a dependency. Section 6.

3. What is the width at every block boundary, and does the bandwidth model carry the ratio? 256 onto 64 is a quarter, and the pipeline's own testbench cannot see it. Section 7.

4. Are register accesses blocking or posted? Forty cycles of stall on a sixteen-cycle budget. Section 8.

5. What is the idle-gap distribution, against the power state's transition cost? A median gap below the transition means the state loses. Section 9.

6. How deep is each elastic buffer, and from what burst length? The traffic shape sizes it, not the frequencies. Section 10.

7. What is the floorplan distance on every inter-block path, and is it pipelined? Adding a stage late is an interface change. Sections 11 and 19.

8. Has the debug plan been compared against the RTL, line by line? An afternoon before tape-out against sixty days after. Sections 12 and 20.

9. What is the integration area as a fraction of the block area? Twenty percent that no block owner estimates. Section 13.

10. Which of the six properties does "every block passed" imply? Section 14 exists because the answer is the first one only.

23. How This Appears In Real Engineering

A block owner is accountable for a block and its ports, and every defect in this chapter is outside that boundary. That is not negligence; it is the partitioning working as designed, and it means the integration defects need an owner who is not a block owner.

An integration or top-level engineer owns all ten models and usually inherits them as a checklist rather than as a set of calculations. The difference matters at section 7 and section 10, where the answer is a number the block owners need and only the integrator can compute.

A physical-design engineer discovers sections 11 and 13 — the distance and the area — and discovers them after the RTL has committed to a partitioning. Section 19's point about pipeline stages being an interface change is why floorplan feedback belongs in the architecture phase rather than after it.

A post-silicon debug engineer inherits section 12, and inherits it as a fixed constraint. The counters that exist are the ones that exist, and section 21 is what a debug looks like when eight of twenty are missing: the first finding is that the investigation will be slow, and every subsequent finding is reached indirectly.

24. Common Misconceptions

"Every block passed, so the device works." One property of six. Section 14.

"The synchroniser is in the RTL." Without the constraint, synthesis optimises it away. Sections 5 and 19.

"Reset is reset." A block released before its dependency corrupts state before any test starts. Section 6.

"The datapath is 256 bits wide." Onto a 64-bit boundary it delivers a quarter of that. Section 7.

"It is only a register read." Forty cycles of datapath stall. Section 8.

"A low-power state saves power." Not if the idle gap is shorter than the transition. Section 9.

"The blocks are on the same die, so they are adjacent." Eight millimetres is four cycles before any logic. Section 11.

"We can add counters later." You cannot. Section 12.

"The area is the sum of the blocks." Plus twenty percent nobody owns. Section 13.

"Integration is an assembly step." It is where four of six properties are decided. Section 4.

25. Interview Reasoning

Q. Every block passes its own tests and the device does not work. Where do you start?

At the reset order, because an early release corrupts state before any other test begins and its symptoms look like anything at all. Then domain crossings, then width conversion, then interconnect timing — that ordering is by how badly each disguises itself, not by likelihood. All four are outside every block's port list, which is why block-level verification cannot find them.

Q. Your synchroniser is in the RTL and you still see metastability. Explain.

The false-path constraint is missing, so synthesis optimised through the two flops. The RTL looks correct and the netlist does not have a synchroniser in it — which is why the crossing is a constraint-file review as much as an RTL one, and why a design's crossing list should be a checked artefact rather than an assumption.

Q. The pipeline is 256 bits wide and the memory bandwidth is a quarter of the projection.

The boundary is 64 bits, so every internal transfer is four beats. The pipeline's own testbench ends at its port and cannot see the ratio, and no upstream bandwidth calculation carried it. The silicon is correct; the projection was made against the internal width rather than against the boundary's.

Q. How do you decide whether a low-power state is worth entering?

Compare the transition cost against the residency distribution, not against the mean idle time. A ten-microsecond transition needs about fifty microseconds of residency to be worth twenty percent overhead — and a policy that enters on any idle detection will repeatedly enter on gaps shorter than that, turning a power feature into a power regression.

Q. What does the space between blocks cost in area?

Around twenty percent of the block area, and none of it appears on a block owner's estimate: synchronisers on every crossing, converters on every width change, pipeline registers for every millimetre, the register interconnect and the debug logic. It is discovered at floorplan, which is the last place to discover an area problem.

Q. What is the highest-leverage review before tape-out?

Comparing the debug plan against the RTL, counter by counter. Eight missing counters at three days each is twenty-four days added to a silicon debug, and it cannot be recovered afterwards at any price. The comparison costs an afternoon, and section 21 is a device where nobody did it.

26. Exercises

1. Replace RTL 1's stage count with a mean time between failures computed from the two clock rates and a metastability window, and find the stages needed for a ten-year target.

2. Give RTL 2 a dependency graph rather than a single dependency, and find the minimum number of reset stages.

3. Extend RTL 3 to a boundary with backpressure and show what the wide side must do while it waits.

4. Model RTL 4's register path as posted rather than blocking and price the completion path it needs.

5. Drive RTL 5 with a real idle-gap distribution and find the entry threshold that maximises net saving.

6. Make RTL 7's wire delay non-linear below a threshold length and find where the linear model over-estimates.

7. Replace RTL 8's fixed days-per-counter with a distribution and compute the debug estimate as a percentile.

8. Combine RTL 6 and RTL 9: find the elastic buffer depth at which the area budget fails.

9. Model section 21 end to end: a missing constraint, a shared reset stage, an uncarried width ratio and eight missing counters.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the device it catches that the current mask calls working.

27. Summary

24.1, 24.2 and 24.3 built three blocks that each pass their own tests. This chapter is the finding that they do not make a device, and every defect in it lives on the far side of a block's port list.

Blocks run at different frequencies. A crossing needs two synchroniser stages costing two destination cycles, and a design can know a crossing is unsafe while budgeting it as free.

Blocks come out of reset at different moments. A block released before its dependency issues its first transaction to a peer that is not there — and the plan being wrong and the release failing are separate defects with separate owners.

Blocks have different datapath widths. 256 bits onto 64 is four beats and a quarter of the wide side, and the pipeline's own testbench ends at the port where that becomes true.

Configuration space is a different world from the datapath. A blocking read over a 40-cycle path stalls a datapath budgeted at sixteen, and the fix is architectural rather than optimisational.

A power state costs its transition every time it is entered. Ten microseconds against a hundred is 10% and against twenty is 50% — the same fixed-cost arithmetic as 23.2 §9, eight orders of magnitude apart.

Two clocks need an elastic buffer sized by the burst. A 20% gap over 200 beats needs 40 entries and over 400 needs 80 — the traffic shape sizes it and the frequencies only scale it.

And blocks are not adjacent on the die. Eight millimetres at two per cycle is four cycles of wire before any logic, and adding a pipeline stage late is an interface change rather than a floorplan one.

The counters cost area before and time after. Eight of twenty missing at three days each is 24 days added to a silicon debug, and none at all is sixty — and the comparison that would prevent it costs an afternoon. Three chapters of this module each ended on a debug that needed a counter nobody built.

The glue is not free. Ten thousand units of blocks attract 2000 of integration at twenty percent, plus buffers — 13,000 against a block list that reports 11,000 and is entirely correct.

One mutation survived on a configuration outside the model's premise and one on a case already driven and half observed — the second needed no new stimulus, only the other build asserted. That is twice in three chapters, and the rule is now mechanical: on every degenerate case, assert both builds.

Every block passing is one property of six. The sign-off an RTL team reports called five of six devices working when one was — and section 21 is first silicon that boots, enumerates, passes traffic, and does four things wrong, none of which is a block's fault.

24.5 — Buffering Strategies takes the one resource this chapter kept setting aside: the elastic buffers of section 10, the merge buffer of 24.2 §9, the scheduler queue of 24.3 §12 and the replay buffer of 24.2 §13 — all of them sized from traffic, all of them competing for the same SRAM.