“Packaging Is Separate from Architecture”

This one is not learned from a diagram or a datasheet. It is learned from an organisation chart — architecture in one group, packaging in another, with a handoff between them — and an org chart is a much harder thing to argue with than a mistaken inference.

1. The Belief

"I define the partition and the interfaces. Packaging figures out how to build it."

The sequencing is the error. Not the division of labour — that is sensible and necessary — but the assumption that architecture is upstream of packaging, so a partition can be finalised and then handed over.

The replacement model (§6):

Packaging sets both the price and the capacity of a boundary. Bump pitch and die edge length bound how many signals can cross; energy per bit bounds how much traffic you can afford to move; reach bounds where dies may sit. The partition you can build is a function of the package you chose — so the two are co-decided, and the arithmetic that connects them takes five minutes.

And this chapter's failure is different in kind from the rest of the module. 28.1 and 28.4 produce data corruption. This one produces a partition that cannot be built — discovered at floorplan, months after every downstream decision assumed it.

2. Why a Competent Engineer Believes It

Four reasons, and the first two are structural rather than intellectual.

On a monolithic SoC it is essentially true. Package selection there is driven by pin count, thermal envelope and cost, and none of those changes the internal architecture. An architect who has shipped several monolithic parts has genuinely never had packaging constrain a block boundary — because internal boundaries were wires, and wires are free.

The org chart enforces it. Architecture and packaging are usually different groups with different reporting lines, different tools and different review cadences. A handoff is the natural interaction, and handoffs are sequential by construction.

The tools do not model it. Architectural exploration happens in spreadsheets, performance models and block diagrams — none of which know a die's edge length or a substrate's bump pitch. A constraint invisible to your tools is a constraint you cannot accidentally discover.

And "a boundary is a boundary" is a good abstraction that stops being good. In architecture we treat an interface as a bandwidth number and a protocol. That abstraction is exactly right when boundaries are cheap and unlimited, which is the monolithic case — and it silently drops the two facts that matter here: how many wires can physically cross, and what each one costs to drive.

3. The Hidden Assumption

That the cost and capacity of a die-to-die crossing are constants the architecture can treat as given — that whatever bandwidth the partition needs, the package will supply.

Three sub-assumptions.

A — that interface width is unbounded. It is not. Signals cross on bumps, bumps sit on a die edge, and the edge has a finite length (§7).

B — that a crossing's energy is negligible. It is not, and at high bandwidth the interface can become a meaningful fraction of a die's power budget — which is a thermal problem, which is a packaging problem, which loops back into architecture.

C — that die placement is free. It is not: reach limits where dies may sit relative to each other, so a partition implies a floorplan, and some floorplans are not reachable.

4. The Root-Cause Tree

The misleading symptom deserves emphasis because it determines who gets blamed. The failure surfaces in the packaging group's work, so it is naturally described as a packaging shortfall — and the architecture that created it is never revisited. The same partition then appears in the next product.

5. Sourcing and Scope

6. Packaging Sets the Price and the Capacity of a Boundary

Three readings.

Row 1 is a hard constraint and the only one in this module that produces "cannot be built" rather than "performs badly." Energy and thermal budgets can be traded; a die edge cannot be extended. That asymmetry is why §7 is the chapter's centre.

Row 2 loops back into row 4, which is what makes packaging genuinely architectural rather than merely constraining. A wide, fast interface consumes power; power becomes heat; heat constrains what may be stacked — so an interface decision propagates into a physical-arrangement decision, which propagates back into the partition. Sequencing cannot resolve a loop.

And 26.2 §8 is this table's strongest confirmation (§5). When a crossing is expensive, you must cut where traffic is thin; when packaging makes it cheap, you can cut where the function boundary is. The same architecture is correct or incorrect depending on the package — which is exactly what "not separate" means.

7. The Minimal Counterexample — Beachfront Arithmetic

Two partitions. Architecturally identical: the same blocks, the same traffic, the same bandwidth across the boundary. One is buildable and one is not — and the only difference is which die edge the boundary lands on.

Step 1 — useful bits per second. 2 TB/s × 8 = 16 Tb/s of useful payload.

Step 2 — raw bits, after efficiency. 16 / 0.8 = 20 Tb/s on the wire.

Step 3 — signal count. 20 Tb/s ÷ 16 Gb/s = 1250 signals per direction. Bidirectional: 2500 signals.

Step 4 — total bumps including supplies. At 1:1, 5000 bumps.

Step 5 — bumps available per row on an edge. edge_length ÷ bump_pitch.

Now the two partitions.

Same logic. Same bandwidth. Same protocol. One partition is a floorplan problem and the other is impossible — and the architecture documents for the two are identical.

Now change only the packaging technology, holding the architecture fixed.

Four readings, and this table is the chapter.

Halving the bump pitch roughly halves the rows and quarters the depth, because both the count per row and the spacing improve. The same partition consumes a materially different fraction of the die depending on a packaging parameter the architect did not choose — and may not have been told.

The arithmetic is five steps and nobody does it early. It requires four numbers: target bandwidth, per-signal rate, efficiency, and bump pitch — three of which the architect already has. The missing one is the packaging parameter, which is precisely the input the belief says is somebody else's.

Partition B fails for a reason that has nothing to do with logic. The blocks are right, the traffic is right, the protocol is right. The boundary landed on a short edge. No amount of architectural cleverness above the physical layer changes it, and no review that examines only the block diagram can see it — the block diagram does not have edges.

And the actionable form is a ratio to carry into every partition review: required bumps ÷ available bumps. If it exceeds 1, the partition is not implementable as drawn. If it is close to 1, the interface is consuming a serious fraction of the die and that cost belongs in the trade (28.3 §6).

8. When the Belief Is Locally Useful

Three readings.

Row 1 is why the belief is held most confidently by the most experienced architects. A career of monolithic parts is a career in which this belief was correct — internal boundaries were free, and packaging genuinely was a downstream concern with no architectural feedback.

Row 3 is the distinction that keeps the correction from being arrogant, and it matters for how you raise it. The packaging group should own packaging. The claim is not that architects should design substrates — it is that four of their outputs are architectural inputs (§6), and a sequential handoff delivers them too late to be inputs at all.

And row 2 is the abstraction whose failure is hardest to notice, because the abstraction is otherwise excellent. An interface described as "2 TB/s, protocol X" is a complete description on-die and a partial one across a package — missing exactly the term that decides buildability.

9. Architectural Consequence

Three readings.

Row 4 is the one architects find most surprising. Die aspect ratio is an architectural parameter in a multi-die design, because it determines how much edge is available for a given area. A squarer die has less edge per unit area than an elongated one — so a floorplan choice made for other reasons can remove the beachfront a partition needed.

Row 5 is 26.1's hub topology seen from the packaging side. A documented product places twelve compute dies each with a dedicated link to one I/O die (26.1 §4, Class A). Twelve interfaces must terminate on that one die's available edges — a physical concentration that is invisible in a topology diagram and unavoidable in a floorplan.

And row 7 is the design hedge that the belief prevents. If width might be constrained by packaging, a design that can operate at reduced width has an option; one that hardcoded a width has a respin (§10).

10. RTL Consequence — a Hardcoded Width

The code an engineer writes when the required width is assumed available.

// WRONG. ILLUSTRATIVE. The width came from the bandwidth target and was then
// baked into the datapath. Note there is nothing sloppy here — the parameter
// exists, it is named, it is used. The defect is that everything else assumes
// its value, so it is a parameter that cannot actually be changed.
localparam int LINK_WIDTH = 64;
 
// BUG 1: the framing header is sized for exactly this width, by hand.
localparam int HDR_BITS = 16;                 // "64 lanes needs 16 header bits"
 
// BUG 2: the serialisation ratio is a literal. If width falls, this must change
//        and nothing says so.
localparam int SER_RATIO = 4;                 // 256-bit core bus / 64 lanes
 
logic [255:0]           core_data;
logic [LINK_WIDTH-1:0]  lane_data;
logic [1:0]             ser_phase_q;          // BUG 3: width literal — 2 bits
                                              //        only works for ratio 4
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) ser_phase_q <= '0;
  else if (tx_fire) ser_phase_q <= ser_phase_q + 2'd1;
end
 
// BUG 4: the slice is computed with hardcoded arithmetic rather than from the
//        parameters, so a width change silently mis-slices the core bus.
always_comb
  lane_data = core_data[ser_phase_q * 64 +: 64];

Architecture. A serialiser from a 256-bit core bus onto 64 lanes. LINK_WIDTH exists as a parameter and four other things depend on its value without deriving from it — so it is a constant wearing a parameter's name.

State. ser_phase_q, two bits — correct for a ratio of 4 and silently wrong for any other.

Event. Advances per transfer, wrapping at 4 by virtue of its width.

Contract. The receiver assumes a framing and slicing convention that is implied by these literals and written down nowhere. Change the width on one side and the two disagree about byte order with no error.

Failure — how it actually unfolds. This is not a cycle-level bug; it is a schedule failure, and the timeline is in weeks.

Root cause. A packaging output was consumed as a design constant. The parameter's existence created the appearance of flexibility, and every one of the four dependent literals removed it. Nobody made a decision to make width unchangeable; four people each wrote one line.

And the framing bug is the sharper of the two. If someone does force a width change and misses HDR_BITS or the slice arithmetic, the link trains, data flows, and bytes are silently reordered — a delivered-perfectly-to-the-wrong-place failure (21.6 §14) with no error anywhere.

DV/debug. The tell in review is 27.2 §15's I5 — a hardcoded width beside a parameter is a finding on sight. core_data[ser_phase_q * 64 +: 64] next to localparam LINK_WIDTH = 64 is the same number written twice, and the second one will not be updated.

11. Corrected — Width as a Derived Parameter

// CORRECT. ILLUSTRATIVE. Everything derives from two inputs, and an
// elaboration-time check catches the combinations that cannot work. The point
// is not elegance — it is that a late width change costs a parameter edit
// instead of a datapath redesign.
localparam int CORE_BITS  = 256;
localparam int LINK_WIDTH = 40;                       // set by BEACHFRONT (§7)
 
localparam int SER_RATIO  = CORE_BITS / LINK_WIDTH;   // derived, not literal
localparam int PHASE_W    = $clog2(SER_RATIO);
localparam int HDR_BITS   = $clog2(LINK_WIDTH) + HDR_OVERHEAD_BITS;
 
logic [CORE_BITS-1:0]   core_data;
logic [LINK_WIDTH-1:0]  lane_data;
logic [PHASE_W-1:0]     ser_phase_q;
 
// Elaboration-time checks. These fire at compile time on EVERY configuration,
// including a derivative product where someone changed one number — which is
// exactly the failure mode a temporal assertion catches far too late.
initial begin
  if (CORE_BITS % LINK_WIDTH != 0)
    $error("CORE_BITS (%0d) must be a multiple of LINK_WIDTH (%0d)",
           CORE_BITS, LINK_WIDTH);
  if (LINK_WIDTH > MAX_BEACHFRONT_LANES)
    $error("LINK_WIDTH (%0d) exceeds the beachfront budget (%0d lanes)",
           LINK_WIDTH, MAX_BEACHFRONT_LANES);
  // The bandwidth relationship, checked rather than assumed: does this width at
  // this rate actually meet the target the architecture promised?
  if (LINK_WIDTH * LANE_RATE_GBPS * FRAMING_EFF < TARGET_GBPS)
    $error("width %0d at %0d Gb/s yields %0d Gb/s useful, below target %0d",
           LINK_WIDTH, LANE_RATE_GBPS,
           (LINK_WIDTH * LANE_RATE_GBPS * FRAMING_EFF), TARGET_GBPS);
end
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) ser_phase_q <= '0;
  else if (tx_fire)
    ser_phase_q <= (ser_phase_q == PHASE_W'(SER_RATIO-1)) ? '0
                                                          : ser_phase_q + 1'b1;
end
 
always_comb
  lane_data = core_data[ser_phase_q * LINK_WIDTH +: LINK_WIDTH];
 
// MANDATORY. English: the serialisation phase never exceeds the derived ratio.
// Catches a phase counter whose width and whose wrap condition disagree — the
// residual form of §10's BUG 3 after a parameter change.
a_phase_in_range: assert property (
  @(posedge clk) disable iff (!rst_n)
    ser_phase_q < PHASE_W'(SER_RATIO)
);
 
// MANDATORY. English: a full core word is transmitted in exactly SER_RATIO
// transfers — no word is partially sent and no phase is skipped. Catches a
// mis-sliced datapath after a width change, which otherwise reorders bytes
// silently (§10).
a_word_complete_in_ratio: assert property (
  @(posedge clk) disable iff (!rst_n)
    (tx_fire && (ser_phase_q == '0))
      |-> ##[1:SER_RATIO] (tx_fire && (ser_phase_q == '0))
);

Architecture. Two inputs — core width and link width — and everything else derived. LINK_WIDTH is annotated with where the number came from, which is the documentation that makes it reviewable (27.2 §15 I1).

State. ser_phase_q at $clog2(SER_RATIO) bits, with an explicit wrap rather than relying on the counter's width to wrap correctly — because at a non-power-of-two ratio it will not.

Event. Phase advances per transfer and wraps at the derived ratio.

Contract. MAX_BEACHFRONT_LANES is a packaging input with a named owner (27.5 §17 H1) — it belongs in the contract matrix, not in a designer's head. The third check is the one that closes the loop: it verifies that the width the packaging permits still meets the bandwidth the architecture promised. If it does not, that is an architecture finding surfacing at compile time, which is the earliest anyone could possibly want it.

Failure. The residual risk is a non-integer ratio — CORE_BITS not divisible by LINK_WIDTH, which at width 40 and core 256 is exactly the case: 256/40 = 6.4. The first $error catches it at elaboration, and the architectural response is to choose a core width or a lane count that divides, or to add an explicit gearbox with its own buffering. The important part is that the choice is forced into the open rather than discovered by a mis-sliced bus.

DV/debug. The verification consequence is a width sweep, not a single configuration: the environment must elaborate and pass at every width the beachfront analysis leaves plausible. A parameter that has only ever been used at one value is a constant, whatever its declaration says.

12. Performance Consequence

A constrained width does not simply mean "less bandwidth" — it interacts with the concurrency window.

Illustrative (§5). Target 2 TB/s per direction; beachfront permits 40 lanes rather than 64.

Three readings.

The first-order effect is the 37 %, and it is the smaller problem. The larger one is that every downstream sizing decision assumed the original number — buffer depths, outstanding windows, arbitration weights and the performance model itself (27.4 §7).

And the concurrency interaction is the one that surprises (26.4 §14). Required outstanding is rate × round-trip latency. A narrower link at the same per-lane rate means each transfer takes more cycles, which lengthens the round trip — so the required outstanding window can rise even as the bandwidth falls. A design that sized its ID table for the wide configuration may be under-provisioned for the narrow one, and 27.4 §17's ceiling applies.

Which makes the honest architectural response a re-derivation rather than an acceptance. "We'll take 37 % less" assumes the rest of the design tolerates it. The correct move is to recompute the whole resource graph at the achievable width (26.3 §8) — because the binding constraint may have moved.

13. Thermal and Placement Consequence

The two packaging facts that constrain arrangement rather than width.

Three readings.

Row 1 is why 26.2 §5's arrangement is architectural rather than assembly detail. In a 3D stack, what sits above what determines what can be cooled — and that decision constrains which functions may be co-located, which is a partitioning decision.

Row 4 connects this chapter to 26.3's scheduler failure by a path nobody plans. Thermal throttling produces exactly the unequal-participant condition that an even work distribution amplifies into a package-wide collapse. A packaging-driven thermal limit therefore surfaces as a scheduling bug, and the two teams involved have no shared vocabulary for it.

And row 2 is the loop that defeats sequencing (§6). Interface width is chosen for bandwidth; width and rate determine interface power; power is heat; heat constrains arrangement; arrangement constrains the partition. You cannot resolve a loop by ordering it — you can only iterate, which is what co-decision means in practice.

14. Which Review Gate This Corrupts

And 27.5 §17 H8 is the row that makes the correction actionable without overreach. The review does not decide the electrical limits — it requires that a named specification and a named owner exist for them. That is precisely the input the belief leaves unassigned, and asking for the name costs nothing.

15. Whiteboard Question

"You've drawn a boundary carrying 2 TB/s per direction. Convince me it can be built."

A weak answer talks about protocol and bandwidth — it re-states the requirement rather than testing it.

A strong answer computes, out loud (§7): "2 TB/s useful is 16 Tb/s; at 80 % framing efficiency that's 20 Tb/s on the wire; at 16 Gb/s per signal that's 1250 signals per direction, so 2500 bidirectional, and roughly 5000 bumps with supplies. Now I need the edge and the pitch. On a 10 mm edge at 45 µm that's 222 bumps per row — about 23 rows, roughly a millimetre of die depth. Tight but plausible. On a 4 mm edge it's 57 rows and it isn't buildable."

Then it names the parameter it does not own: "and the number I need from packaging is the bump pitch and the usable edge. Those are inputs to this decision, not outputs of it — if I get them after I've drawn the boundary, I find out at floorplan."

And the strongest close states the consequence for the RTL (§10): "which is also why I'd parameterise the link width and derive the serialisation ratio from it, with an elaboration check against a beachfront budget. If the width changes late, I want a parameter edit, not a datapath redesign."

16. Interview Ladder

Weak candidate answer. "We define the interfaces and bandwidth; the packaging team implements it."

Why the interviewer probes. The answer is organisationally accurate and architecturally incomplete, and the follow-up tests whether the candidate has ever had a partition fail to fit.

Follow-up. "What if the die edge can't supply the bumps your interface needs?"

Strong revised answer. "Then the partition isn't implementable, and I should have known before drawing it. The check is short: required bandwidth over per-signal rate and framing efficiency gives a signal count; times supply overhead gives bumps; against edge length over bump pitch times the rows you can afford. Two architecturally identical partitions can differ by a factor of two or three in rows purely by which edge the boundary lands on — and halving the bump pitch roughly halves the rows."

The senior addition: "and it's a loop, not a sequence. Width drives interface power, power is heat, heat constrains stacking, stacking constrains the partition. You can't resolve a loop by ordering it — so packaging technology is an input chosen alongside the partition."

The addition that shows it reaches the RTL: "and I'd never hardcode the link width. Derive the serialisation ratio and the phase counter width from it, and put an elaboration-time check against a beachfront budget with a named owner. A late width change should cost a parameter edit, not a framing redesign and a full re-verification."

17. How to Remember the Correct Model

A boundary is not free, and the package sets its price. Bandwidth says what you want; beachfront says what you can have.

The operational form is one ratio, and it belongs in every partition review:

required bumps ÷ available bumps.

Above 1, the partition is not implementable as drawn. Near 1, the interface is consuming a serious fraction of the die and that cost belongs in the trade.

And the memory hook for why sequencing fails: width → power → heat → stacking → partition. That is a loop, and a handoff cannot resolve a loop — only iteration can, which is what co-decision means.

18. Understanding Check

19. Summary

Six things.

The error is sequencing, not the division of labour (§1, §8). Packaging should own packaging. Four of its outputs are architectural inputs, and a handoff delivers them too late to be inputs.

Beachfront is a hard constraint, and it is the counterexample (§7). Bandwidth ÷ (rate × efficiency) → signals → bumps, against edge ÷ pitch × rows. On a 10 mm edge, 23 rows; on a 4 mm edge, 57 and unbuildable — with identical architecture documents.

The packaging parameter moves the answer as much as the architecture does. Same partition, same edge: 45 µm needs 23 rows, 25 µm needs 13, 10 µm needs 5.

It is a loop, not a chain (§13): width → power → heat → stacking → partition. Sequencing cannot resolve a loop; only iteration can.

In RTL it is a width consumed as a constant (§10). Four dependent literals make a named parameter unchangeable — and a late width change costs a framing redesign and full re-verification instead of a parameter edit.

And a constrained width is not just less bandwidth (§12). The round trip lengthens, so the required outstanding window can rise as bandwidth falls — recompute the graph rather than accepting the shortfall.