Skip to content
VLSI Mentor

DDR · Module 32

Data-Centre Servers

When DDR is one tier among several it no longer owns the placement decision, and a policy that consumes its observed mean latency closes a delayed negative feedback loop that oscillates. The controller must export a decomposition rather than a number, because hysteresis cannot damp a swing its own action sets.

Every platform class in this module has had a memory system decided at design time and operated as built. This one does not.

A server platform's memory composition is chosen at deployment and changed during service — and when a second tier is present, DDR stops being the whole memory system and becomes one tier among several.

The dominating constraint is that DDR no longer owns the placement decision. Something above it decides which pages live where, and that decision consumes DDR's observed behaviour as an input — so the controller's job acquires a new obligation: to export a quantity a policy it does not own can safely act on.

And the naive export closes a loop.

A placement policy that consumes DDR's observed mean latency creates a delayed negative feedback loop: migrate hot pages in, DDR's load rises, its observed latency rises, the policy migrates them out, its load falls, and the policy migrates them back. Negative feedback with delay does not converge — it oscillates, and hysteresis alone does not stop it because the loop's amplitude is set by the migration granularity rather than by the decision threshold.

Axis A1 is identical for the fifth and last time. The same destructive read, the same thirteen obligations of 31.1 §5. What differs is that this controller is the only one in the module whose output is consumed as evidence by a decision-maker outside it.

1. The Shared Baseline, and This Chapter's Question

CURRICULUM-DERIVED from 32.1 §1: all five platform classes carry the same thirteen obligations of 31.1 §5, and the module's question is always the same three steps.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   1  WHICH CONSTRAINT
        DDR does not own the placement decision, and the decision
        that does own it CONSUMES DDR's observed behaviour (§2, §4)

   2  WHAT DECISION
        export a DECOMPOSITION rather than a number, because the
        naive export closes an oscillating loop (§6, §8)

   3  WHAT GRADE
        B for the DDR mechanisms, D for every figure, and the
        placement policy itself is outside this chapter (§9)

2. What Changes When DDR Is One Tier

Five things change on the DDR side, and only the last is new to this module.

When DDR is the whole memoryWhen DDR is one tier
Which addresses arriveall of themonly those placed here, by a policy
Who decides thatthe address map — 18.2a placement policy above the controller
When it is decidedat design and configuration timecontinuously, during service
What a page migration isnothing — there is nowhere to goa traffic event the controller sees as a burst
What the controller's output is used forperformanceperformance AND a placement decision

Row five is the chapter, and it is genuinely new. In thirty-one chapters the controller's telemetry has been consumed by engineers30.8's bottleneck analysis, 30.10's debug, 32.4 §7's provisioning evidence. Here it is consumed by a control loop that acts on it automatically, at a rate no engineer mediates — and that changes the requirements on it entirely.

Row four deserves its own note, because it is a real and often-missed cost. A migration is a burst of sequential traffic the controller did not request and cannot decline. CURRICULUM-DERIVED from 23.1 §8, the queueing component has no upper bound, so a migration burst inflates the latency of every request it queues behind. So the act of measuring-then-migrating perturbs the thing it measured, which is the seed of §6's loop.

3. The Boundary, Stated

This chapter is the most collision-prone in the module, so the boundary is stated as a table rather than left to inference. Every route below was verified present at the time of writing; the CXL track is under active development and a citation should be re-checked before it is relied on.

QuestionOwnerThis chapter's position
Why expand memory at all/cxl/why-memory-expansion-mattersassumed, never argued
What the semantics are/cxl/memory-expansion-semanticsnot modelled
What the device class is/cxl/type-3-devicesnot modelled
What the server looks like/cxl/memory-expansion-server-architecturesnot restated — §10's diagram shows only the DDR side
Pooling, sharing, allocation/cxl/shared-memory-pools, /cxl/pooled-datacenter-architecture, /cxl/pooling-benefits-and-challengesout of scope
The far tier's latency anatomy/cxl/cxl-latency-anatomy — a sum of named parts, tail against mean, per-segment budgetscited, never computed here
The instruction-to-port path, TLB reach, first-touch faults, NUMA placement/cxl/cxl-memory-access-costcited, never computed here
Real devices/cxl/real-industry-memory-devicesout of scope
What the DDR controller must EXPORT to a placement policythis chapter
The DDR-side cost of being one tierthis chapter

The last two rows are the whole of this chapter, and the reason they are unowned is structural: the CXL track models the far tier and the host path, and Module 23 models DDR's own latency — but nothing asks what DDR must publish so a policy can choose between them.

And one asymmetry is worth naming immediately. CURRICULUM-DERIVED from /cxl/cxl-latency-anatomy, which owns the tail against the mean for the far tier, and 23.1 §9, which owns why the average is the wrong statistic for DDR: both tiers' owners independently concluded that the mean is the wrong statisticso a policy comparing two means is comparing two numbers that both chapters say should not be used. §5 is that observation developed.

4. What the Placement Policy Needs

Work out what a placement decision actually requires, because the requirement is narrower than how fast is DDR and the difference matters.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   the policy's question, for a given page:

     "if this page lived on the near tier instead of the far tier,
      how much would its access cost change?"

   which decomposes into:

     1  the near tier's cost for THIS page's access pattern
     2  the far tier's cost for the same pattern
     3  the cost of MOVING it

   the DDR controller can speak to (1) and to part of (3).
   It can say nothing about (2), which is the CXL track's.

Now decompose (1), which is the DDR side's deliverable. CURRICULUM-DERIVED from 23.1, which owns the exhaustive decomposition into components that sum exactly to the total, and — the part that matters here — which components are irreducible and which are policy, plus the one component with no upper bound.

ComponentIrreducible?Does it move with load?Is it useful to a placement policy?
Row-state work — activate, prechargeirreducible for a given access patternnoyes — it is a property of the page's pattern
The configured device latencyirreduciblenoyes — a constant for this tier
Transfer occupancyirreduciblenoyes
Queueingpolicy23.1 §7yes, and it has no upper bound23.1 §8NO — see §5
Refresh interferencepolicy and conditionsomewhatmarginally

Row four is the one that must be excluded, and the reason is not that it is unimportant. Queueing is often the largest termCURRICULUM-DERIVED from 30.4 §4, whose latency budget puts it at nearly three times the configured device latency under load. It is excluded because it is a function of how much traffic is already here, and a policy deciding whether to send more traffic here must not be told the current queue depth as though it were a property of the destination.

A placement policy needs the part of DDR's cost that is a property of the PAGE. The queueing term is a property of the CURRENT LOAD, and exporting it as part of a single latency figure tells the policy that DDR is intrinsically slow when it is merely currently busy.

5. The Mean Is the Wrong Export

Two independent arguments, from the two chapters that own the two tiers.

Argument one — the statistic. CURRICULUM-DERIVED from 23.1 §9, which owns why the average is the wrong statistic and what to use instead, and from /cxl/cxl-latency-anatomy, which owns the tail against the mean for the far tier. Both owners rejected the mean for their own tier. So a policy comparing means is using, on both sides, the statistic both sides' owners say not to use.

Argument two — the composition. §4's table shows the mean blends a page-dependent term with a load-dependent term. DERIVED, grade D, with figures consistent with 30.4 §4's budget shape:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   GRADE D. Two pages, same tier, different access patterns.

     page P  : row-hit dominant  -> irreducible part 20 units
     page Q  : conflict dominant -> irreducible part 48 units
     current queueing term (shared, load-dependent) : 40 units

   what a MEAN export reports:
     P : 20 + 40 = 60
     Q : 48 + 40 = 88
     ratio Q/P = 1.47

   what the PAGE-DEPENDENT part is:
     P : 20,  Q : 48,  ratio = 2.40

   DERIVED: the blended figure UNDERSTATES the difference between
   the two pages by a factor of 1.63, because a large shared term
   is added to both. So the policy's ability to DISCRIMINATE
   between pages is degraded by exactly the term that has nothing
   to do with either page.

And it gets worse as the system gets busier, which is the wrong direction. DERIVED: at a queueing term of 200 units, P reports 220 and Q reports 248 — a ratio of 1.13. So under load, when placement matters most, a mean export makes every page look alike. CURRICULUM-DERIVED from 23.1 §8, the queueing component has no upper bound, so there is no load at which this stops getting worse.

Two conclusions, and the second is §6's premise.

The export must be decomposed, not merely a better statistic. A tail percentile of the blended figure has the same problem: it still contains the load-dependent term. So the fix is not export the p99 instead of the mean — it is export the components separately, which 23.1 already computes.

And a blended export does not merely mislead — it couples the policy to the load. That is a control-loop property rather than a measurement error, and it is the subject of §6.

6. The Loop

Trace the signal path, as 32.2 §3 did for the thermal case.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   the placement loop, when the advisory includes queueing

     policy sees DDR advisory latency LOW
         |
         | migrates hot pages TO the near tier
         v
     DDR's offered load RISES
         |
         | 23.1 §8: queueing has NO UPPER BOUND
         v
     DDR advisory latency RISES
         |
         | policy sees the near tier as expensive
         v
     migrates pages AWAY
         |
         | DDR's load FALLS
         v
     advisory latency FALLS        <-- closes here, back to the top

This is NEGATIVE feedback, and that is the interesting part. CURRICULUM-DERIVED from 32.2 §4, the thermal loop was positive feedback — hotter causes hotter — and its failure modes were amplification by 1/(1−G) and, past G = 1, runaway. Negative feedback is normally stabilising.

It is not stabilising here, because it is delayed.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   the delay, and where it comes from

     a migration is not instantaneous. It is a bulk data movement
     (§2 row four), so the load change LAGS the decision by the
     migration time T_m.

     and the measurement is windowed: the advisory reflects an
     average over W cycles, so the observation LAGS the load by W.

   total loop delay  T_d  =  T_m + W  (plus the policy's own period)

   a negative-feedback loop with gain G and delay T_d oscillates
   when the delay is long relative to the loop's own response
   time -- the classic result, and here BOTH terms are large:
   a bulk migration and a measurement window.

DERIVED, grade D, to show the shape rather than to predict a period:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   GRADE D.  migration time T_m = 10 ms, window W = 1 ms,
             policy period = 5 ms.

     T_d = 10 + 1 + 5 = 16 ms

   so the policy acts on information about a load state that is
   at least 16 ms old, and its action takes 10 ms to take effect.
   By the time the load has moved, the advisory it acted on
   describes a system that no longer exists.

   observable consequence: pages migrate in, the advisory rises,
   pages migrate out, the advisory falls, pages migrate in. The
   PERIOD is set by T_d and the AMPLITUDE by how many pages the
   policy moves per decision.

And the cost is not the oscillation itself — it is the migration traffic. DERIVED from §2 row four: each migration is a burst of traffic the controller did not request, so a thrashing policy imposes a continuous load that serves no application. CURRICULUM-DERIVED from 30.8 §2: this consumes rung 4 capacity to deliver zero rung-5 value — the purest form of that chapter's gap, because the wasted work is not even wrong, it is self-inflicted.

7. Why Hysteresis Is Necessary and Not Sufficient

CURRICULUM-DERIVED from 23.5, which owns that hysteresis is mandatory rather than an optimisation for an adaptive policy, and from 32.2 §7, which applied the same argument to a thermal throttle. So hysteresis is required here too, and this section is about why it does not finish the job.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   what hysteresis fixes and what it does not

   FIXES: small-signal chatter. A page near the decision boundary
          no longer flips on every window's noise.

   DOES NOT FIX: the delayed loop of §6. Hysteresis widens the
          band a signal must cross; it does not shorten T_d. A
          delayed loop whose swing EXCEEDS the band still
          oscillates, at the same period, with the band
          subtracted from the amplitude.

   and here the swing is large by construction:
     the advisory's swing = the queueing term's swing
     the queueing term's swing = a function of how many pages
       the policy just moved
     -> the loop's own action sets the swing that must be
        damped, so a fixed band cannot be sized against it

That last point is the structural reason hysteresis is insufficient, and it is worth stating as a general rule because it is not specific to memory:

Hysteresis damps a signal whose swing is independent of the controller's action. When the controller's own action determines the swing, the band must be sized against the action — and a band sized for the largest possible action is too wide to act at all.

Three things that do work, in increasing order of effectiveness.

Bound the action per decision. Migrate at most M pages per period, so the swing is bounded by M rather than by the policy's appetite. This is 17.4 §9's bounded-override discipline applied to a placement policy — the mechanism must be bounded, not merely present.

Shorten the delay. Reduce the measurement window W. But not below the point where the advisory becomes noise — and CURRICULUM-DERIVED from 32.2 §5, a window too short reports a burst as a sustained state, which is the same failure in a different domain. So W has a floor set by the traffic's own burstiness.

Or break the loop, which is the only one that removes the oscillation rather than damping it. Export the component that does not move with load — §4's irreducible terms — so the policy's input is no longer a function of the policy's output. DERIVED: with the queueing term excluded, dL_adv/dload = 0, so the loop gain is zero and the delay is irrelevant. A loop with zero gain does not oscillate at any delay.

That is the decision, and §8 states it.

8. Export a Decomposition, Not a Number

The DDR controller's obligation, in three parts.

Part 1 — export the irreducible components separately. CURRICULUM-DERIVED from 23.1, which already computes them and owns which components are irreducible and which are policy. The controller is not being asked to compute anything new — it is being asked to publish the decomposition it already has, rather than a sum.

Part 2 — export the load-dependent term too, but labelled, and never blended. The policy may legitimately want to know DDR is busy — that is a capacity question rather than a placement question. CURRICULUM-DERIVED from 23.2 §9's exhaustive-and-disjoint standard: the terms must sum to the total and be separately addressable, so a consumer can choose which to use and a reviewer can check the sum.

Part 3 — export it per access pattern, not per page. The controller does not know about pages. CURRICULUM-DERIVED from 18.2, what the controller sees is an address stream and its row-state consequences, so the honest export is keyed by an address region and describes the region's observed row-hit structure — and the policy maps regions to pages, which is its own business.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   the export, as a contract

     per address region, per window:
       irreducible_cost   -- row-state work + device latency +
                             transfer.  DOES NOT move with load.
       queueing_cost      -- labelled separately. MOVES with load.
       samples            -- so the consumer knows the confidence
       region_id          -- the controller's own regioning

     and one flag:
       advisory_stale     -- the window closed with too few samples
                             for this region to be usable

   the stale flag matters: a region with three samples has an
   irreducible cost estimate that is noise, and 30.8 §11's rule
   says an instrument that can report a meaningless value has
   not been checked.

And the honest limit of the whole export, which must be stated with it. The controller can describe only its own tier. CURRICULUM-DERIVED from /cxl/cxl-latency-anatomy and /cxl/cxl-memory-access-cost, the far tier's cost and the instruction-to-port path belong to those chapters — so the placement decision is a comparison the DDR controller contributes one side of and cannot make. A controller that exported a recommendation rather than a measurement would be making a decision with half the inputs.

9. The Evidence Situation

§Scope's grades, and this platform class has the best evidence base in the module — which is itself the finding.

What you might wantBest grade availableWhy
DDR's latency decomposition into named componentsA/CURRICULUM-DERIVED23.1 §3 — components that sum exactly, on a verified configuration
Which components are irreducibleA/CURRICULUM-DERIVED23.1 §7
That queueing has no upper boundA/CURRICULUM-DERIVED23.1 §8
That the average is the wrong statisticA/CURRICULUM-DERIVED23.1 §9
The current generation's feature setA/BModule 25
The far tier's latency anatomyanother track's A/cxl/cxl-latency-anatomy — cite, do not copy
A named platform's tier compositionnoneit is a deployment choice, changed in service — §2
A named platform's placement policynoneit is operating-system and firmware, and it changes
The migration time T_mC or Dmeasurable on a specific system; not a documented property

Two observations, and the second is the module's closing evidence lesson.

The DDR side is well documented and the composition is not — which is 18.4 §5's pattern for the fifth time in this module, and here it is sharper than anywhere else: the composition is not merely configured, it is changed during service. So there is no configuration to read once. CURRICULUM-DERIVED from 18.4 §5: asking for the tier composition is a category error, and the reason is stronger than for an address map — the answer has a validity period.

And that is why §8's export exists rather than a documented table. A platform whose composition changes in service cannot be characterised in advance, so the only durable artifact is a mechanism that reports the current state. CURRICULUM-DERIVED from 32.4 §7: instrumentation is a design deliverable — and here it is the deliverable, because there is no static answer to deliver instead.

10. The DDR Side, as Blocks

A block diagram showing only the DDR side of a tiered server memory system, because the far tier and the host path belong to the CXL track and are represented as a single boundary block. On the left, the placement policy sits outside the controller and is not modelled here; it issues migrations and consumes the advisory. Migration traffic enters the controller's queue as ordinary traffic the controller did not request and cannot decline. Application traffic enters the same queue. The queue feeds the legality filter and the scheduler, identical to every other platform because the thirteen shared obligations do not vary, and then the commit point and the DRAM device. The latency decomposer observes the completed accesses and separates the irreducible components, which are row-state work, device latency and transfer occupancy, from the queueing component, which moves with load and has no upper bound. The tier advisory export publishes those components separately and never blended, keyed by address region, with a staleness flag for regions with too few samples. The advisory returns to the placement policy, and that return path is the feedback edge this chapter analyses: if the advisory includes the queueing term, the policy's input becomes a function of the policy's own output and the loop oscillates. The churn monitor observes migration traffic and reports the migration rate and the fraction of migrated regions that migrate back, which is the observable that distinguishes a converging policy from a thrashing one. The far tier is drawn as a single boundary block labelled as the CXL track's, because this chapter cites its latency anatomy and models none of it.Placement policyOUTSIDE thecontrollerApplicationtrafficwhat was asked forMigrationtrafficnot requested, notdeclinableTier advisorydecomposed, neverblendedQueue + legalityidentical everywhereChurn monitormigrate-backfractionLatencydecomposer23.1's componentsCommit pointstate may changeDRAM devicethirteen obligationsFar tierboundarythe CXL track'smigratebulkreqcandcmddoneadvisoryevents12

The advisory edge back to the policy is the feedback path, and it is the only closed loop in the diagram. If the advisory carries the queueing term, the policy's input is a function of the policy's own output — §6's loop — and if it carries only the irreducible terms, the loop's gain is zero and the delay stops mattering.

And the far tier is one block, deliberately. CURRICULUM-DERIVED from §3's boundary: /cxl/cxl-latency-anatomy owns its anatomy and /cxl/cxl-memory-access-cost the instruction-to-port path. A diagram that expanded it would be restating another track's chapter, and §8's honest limit is that this controller contributes one side of a comparison it cannot make.

11. RTL — The Tier Advisory Export

The platform-profile block for this chapter. TIERED is the platform property, and the generate boundary shows that a single-tier platform needs no advisory at all — which is why the preceding four chapters' controllers did not have one.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// tier_advisory_export -- the platform-profile block of §11.
//
// CLASSIFICATION: synthesisable telemetry. Drives nothing directly,
// but its output is CONSUMED BY A CONTROL LOOP (§2 row five), which
// is why its contract is stricter than a debug counter's.
//
// WHAT IT IS: §8's three-part contract. It publishes 23.1's
// decomposition per address region, separating the irreducible terms
// from the load-dependent one, and NEVER sums them.
//
// WHY IT EXISTS HERE: §5 proves a blended export degrades the
// policy's ability to discriminate between pages, and worsens under
// load. §6 proves it closes a delayed negative-feedback loop. §7
// proves hysteresis cannot damp a loop whose swing the controller's
// own action sets. So the fix is structural: publish the component
// with zero load-dependence.
//
// HOW TO RUN IT: run a mixed workload and read irr_cost and
// queue_cost per region under two different load levels.
// EXPECTED RESULT: irr_cost is stable across load levels for the
// same access pattern; queue_cost is not.
// EXPECTED TRACE: irr_cost must not change when offered load
// changes and the access pattern does not.
//
// SYNTHESIS: per-region accumulators plus a sample counter. No
// divider: every cost is published as a sum and a count.
//
// LIMITATIONS: it describes THIS TIER only. The far tier's cost is
// /cxl/cxl-latency-anatomy's and the instruction-to-port path is
// /cxl/cxl-memory-access-cost's, so this block publishes a
// MEASUREMENT and never a RECOMMENDATION -- §8's honest limit. It
// also regions by address, not by page: 18.2 owns what the
// controller can see, and the policy maps regions to pages.
// ---------------------------------------------------------------------
module tier_advisory_export #(
  // The platform property. A single-tier platform has no placement
  // policy to advise, so the mechanism is ABSENT rather than unused.
  parameter bit TIERED       = 1'b1,

  parameter int NUM_REGIONS  = 16,
  parameter int WIN          = 1048576,
  // §8's stale threshold: a region with too few samples has an
  // estimate that is noise, and 30.8 §11's rule says an instrument
  // that can report a meaningless value has not been checked.
  parameter int MIN_SAMPLES  = 64,
  parameter int LAT_W        = 20,
  // COUNT, not INDEX: an accumulator must represent MIN_SAMPLES and
  // beyond at full latency, so it needs LAT_W + $clog2(WIN + 1)
  // bits. One short and it wraps on a hot region -- publishing a LOW
  // irreducible cost for exactly the region the policy is about to
  // move, which is the wrong direction.
  parameter int ACC_W        = LAT_W + $clog2(WIN + 1),
  parameter int SMP_W        = $clog2(WIN + 1),
  parameter int RID_W        = $clog2(NUM_REGIONS > 1 ? NUM_REGIONS : 2)
)(
  input  logic                clk,
  input  logic                rst_n,

  // One completed access, with 23.1's components already separated.
  // This block does NOT decompose -- 23.1 §10 owns the decomposer
  // and 27.3's independence discipline says a consumer must not
  // recompute its producer's result.
  input  logic                acc_done,
  input  logic [RID_W-1:0]    acc_region,
  input  logic [LAT_W-1:0]    acc_irreducible,
  input  logic [LAT_W-1:0]    acc_queueing,

  input  logic                win_tick,

  // The export. Published as SUMS and a COUNT, never as a mean --
  // §5's argument, and 27.5's refusal to produce a lone figure.
  output logic [ACC_W-1:0]    irr_sum   [NUM_REGIONS],
  output logic [ACC_W-1:0]    queue_sum [NUM_REGIONS],
  output logic [SMP_W-1:0]    samples   [NUM_REGIONS],
  output logic [NUM_REGIONS-1:0] advisory_stale,
  output logic                result_valid,
  output logic                acc_saturated
);
  initial begin
    if (NUM_REGIONS < 1) $fatal(1, "tier_advisory_export: NUM_REGIONS >= 1");
    if (MIN_SAMPLES < 1) $fatal(1, "tier_advisory_export: MIN_SAMPLES >= 1");
    if (WIN < MIN_SAMPLES)
      $fatal(1, "tier_advisory_export: WIN < MIN_SAMPLES -- no region can ever be non-stale");
  end

  generate
  if (TIERED) begin : g_tiered

    logic [ACC_W-1:0] a_irr [NUM_REGIONS];
    logic [ACC_W-1:0] a_q   [NUM_REGIONS];
    logic [SMP_W-1:0] a_n   [NUM_REGIONS];

    always_ff @(posedge clk) begin
      if (!rst_n) begin
        for (int r = 0; r < NUM_REGIONS; r++) begin
          a_irr[r] <= '0;  a_q[r] <= '0;  a_n[r] <= '0;
          irr_sum[r] <= '0;  queue_sum[r] <= '0;  samples[r] <= '0;
        end
        advisory_stale <= '1;      // nothing measured yet: ALL stale
        result_valid   <= 1'b0;
        acc_saturated  <= 1'b0;
      end else if (win_tick) begin
        for (int r = 0; r < NUM_REGIONS; r++) begin
          irr_sum[r]   <= a_irr[r];
          queue_sum[r] <= a_q[r];
          samples[r]   <= a_n[r];
          // §8's stale flag, per region. A region with too few
          // samples is REPORTED AS UNUSABLE rather than published
          // with a noisy estimate -- because the consumer is a
          // control loop (§2 row five) and it will act on whatever
          // it is given.
          advisory_stale[r] <= (a_n[r] < MIN_SAMPLES[SMP_W-1:0]);
          a_irr[r] <= '0;  a_q[r] <= '0;  a_n[r] <= '0;
        end
        result_valid <= 1'b1;
      end else begin
        result_valid <= 1'b0;
        if (acc_done) begin
          // The two components are accumulated SEPARATELY and are
          // never added. That separation is the whole block: §5
          // shows a blended figure loses discrimination under load,
          // and §6 shows it closes the loop.
          if (a_irr[acc_region] > ({ACC_W{1'b1}} - acc_irreducible))
            acc_saturated <= 1'b1;
          else
            a_irr[acc_region] <= a_irr[acc_region] + acc_irreducible;

          if (a_q[acc_region] > ({ACC_W{1'b1}} - acc_queueing))
            acc_saturated <= 1'b1;
          else
            a_q[acc_region] <= a_q[acc_region] + acc_queueing;

          if (a_n[acc_region] != {SMP_W{1'b1}})
            a_n[acc_region] <= a_n[acc_region] + 1'b1;
          else
            acc_saturated <= 1'b1;
        end
      end
    end

  end else begin : g_single_tier

    // A single-tier platform has no placement policy to advise. The
    // export is ABSENT, and every region is permanently stale --
    // which is the honest encoding of "there is no advisory here"
    // rather than publishing zeros a consumer might act on.
    always_ff @(posedge clk) begin
      if (!rst_n || 1'b1) begin
        for (int r = 0; r < NUM_REGIONS; r++) begin
          irr_sum[r] <= '0;  queue_sum[r] <= '0;  samples[r] <= '0;
        end
        advisory_stale <= '1;
        result_valid   <= 1'b0;
        acc_saturated  <= 1'b0;
      end
    end

  end
  endgenerate
endmodule

advisory_stale initialised to all-ones is the block's most important line. A control loop acts on whatever it is given — §2 row five — so a region that has not been measured must be reported as unusable, not as zero cost. CURRICULUM-DERIVED from 30.7 §9: an unmeasured value is unknown, and unknown is not a default. A zero-cost advisory would attract every page in the system to a region nobody has measured.

And the two accumulators are never added anywhere in the block. That is enforced by structure rather than by comment, which is the point: the blended figure §14's defect produces cannot be produced here without adding a line that does not exist.

12. RTL — Measuring Churn

§6's oscillation has an observable, and it is not the advisory. The advisory oscillating is the symptom; the cost is the migration traffic, so the measurement is on the migrations.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// migration_churn_monitor -- verification/telemetry, CORRECT as written.
//
// CLASSIFICATION: synthesisable telemetry. Drives nothing. Grade D.
//
// WHAT IT DOES: counts migrations per window and, crucially, the
// fraction that migrate BACK within a bounded number of windows --
// which is the observable that distinguishes a converging policy
// from a thrashing one.
//
// WHY IT EXISTS HERE: §6 establishes that the cost of the loop is
// the migration traffic, not the oscillation itself, and §7 that
// hysteresis cannot damp it. So the question a platform must answer
// is "is the policy converging?", and a migration COUNT cannot
// answer it -- a converging policy also migrates. Only the
// migrate-back fraction separates them.
//
// HOW TO RUN IT: run the real policy and read back_fraction as a
// pair over several windows.
// EXPECTED RESULT: a converging policy's back_fraction falls toward
// zero; a thrashing one's stays high indefinitely.
//
// SYNTHESIS: one direction bit and one recency counter per region,
// plus two counters. No divider.
//
// LIMITATIONS: it detects a region returning to a tier it recently
// left. It cannot attribute the cause -- a legitimately changing
// working set also migrates back, and distinguishing that from
// §6's loop requires correlating with the advisory's own swing,
// which §18's row six does. Stating that rather than implying
// attribution is 27.3's independence discipline.
// ---------------------------------------------------------------------
module migration_churn_monitor #(
  parameter int NUM_REGIONS = 16,
  parameter int WIN         = 1048576,
  // How many windows count as "recently". §6's loop period is set by
  // the total delay T_d, so this is a PLATFORM property and on a
  // real system it is a configuration input -- 32.1 §14's lesson.
  parameter int RECENT_WINS = 4,
  parameter int CNT_W       = $clog2(WIN + 1),
  parameter int REC_W       = $clog2(RECENT_WINS + 1),
  parameter int RID_W       = $clog2(NUM_REGIONS > 1 ? NUM_REGIONS : 2)
)(
  input  logic               clk,
  input  logic               rst_n,

  input  logic               mig_event,
  input  logic [RID_W-1:0]   mig_region,
  input  logic               mig_to_near,     // 1 = toward DDR
  input  logic               win_tick,

  output logic [CNT_W-1:0]   r_migrations,
  output logic [CNT_W-1:0]   r_back_migrations,
  output logic [CNT_W-1:0]   r_bytes_moved,
  output logic               result_valid
);
  // Per region: the direction of its last migration and how many
  // windows ago. A "back migration" is a migration in the OPPOSITE
  // direction to the last one, within RECENT_WINS.
  logic             last_dir [NUM_REGIONS];
  logic             has_last [NUM_REGIONS];
  logic [REC_W-1:0] age      [NUM_REGIONS];
  logic [CNT_W-1:0] c_mig, c_back, c_bytes;

  initial begin
    if (RECENT_WINS < 1) $fatal(1, "migration_churn_monitor: RECENT_WINS >= 1");
  end

  always_ff @(posedge clk) begin
    if (!rst_n) begin
      for (int r = 0; r < NUM_REGIONS; r++) begin
        last_dir[r] <= 1'b0;  has_last[r] <= 1'b0;  age[r] <= '0;
      end
      c_mig <= '0;  c_back <= '0;  c_bytes <= '0;
      r_migrations <= '0;  r_back_migrations <= '0;  r_bytes_moved <= '0;
      result_valid <= 1'b0;
    end else if (win_tick) begin
      // Published as a PAIR -- migrations and back-migrations -- so
      // the consumer divides and owns the rounding. 27.5 refuses a
      // lone percentage and 30.8 §11 requires the denominator to be
      // visible; a lone "churn score" would hide that a small
      // back-fraction over three migrations means nothing.
      r_migrations      <= c_mig;
      r_back_migrations <= c_back;
      r_bytes_moved     <= c_bytes;
      result_valid      <= 1'b1;
      c_mig <= '0;  c_back <= '0;  c_bytes <= '0;
      // Age every region's memory of its last migration, saturating
      // so a long-settled region does not wrap back into "recent".
      for (int r = 0; r < NUM_REGIONS; r++) begin
        if (has_last[r]) begin
          if (age[r] >= RECENT_WINS[REC_W-1:0]) has_last[r] <= 1'b0;
          else                                  age[r] <= age[r] + 1'b1;
        end
      end
    end else begin
      result_valid <= 1'b0;
      if (mig_event) begin
        if (c_mig != {CNT_W{1'b1}}) c_mig <= c_mig + 1'b1;
        // A migration in the opposite direction to a recent one is a
        // BACK migration -- the observable §6's loop produces and a
        // converging policy does not.
        if (has_last[mig_region] && (last_dir[mig_region] != mig_to_near)) begin
          if (c_back != {CNT_W{1'b1}}) c_back <= c_back + 1'b1;
        end
        last_dir[mig_region] <= mig_to_near;
        has_last[mig_region] <= 1'b1;
        age[mig_region]      <= '0;
        // §2 row four: a migration is traffic the controller did not
        // request. Counting the BYTES is what turns the oscillation
        // into a number a capacity argument can use.
        if (c_bytes != {CNT_W{1'b1}}) c_bytes <= c_bytes + 1'b1;
      end
    end
  end
endmodule

r_bytes_moved is the output that turns §6's oscillation into a cost rather than an anomaly. CURRICULUM-DERIVED from 30.8 §2: migration traffic consumes rung 4 capacity and delivers zero rung-5 value, so the bytes moved is directly deductible from the bandwidth the applications could have had. A platform reporting a high migration byte count is reporting bandwidth it spent on itself.

And the pair-not-ratio discipline matters more here than usual. A back-migration fraction of one-third means nothing over three migrations and a great deal over three thousand — so publishing the numerator and the denominator, per 27.5's refusal to produce a lone percentage, is what lets a reader tell a settling policy from a thrashing one.

13. RTL — Bounding a Burst You Cannot Decline

§7's first remedy was to bound the action per decision, and nothing has implemented it. The controller can, and the reason it can is a result from this module's first chapter.

CURRICULUM-DERIVED from 32.1 §3, whose two axes were deadline and droppability. Migration traffic has no deadline and is not droppable — which is exactly the writeback class's position in that table, and 32.1 §3 says the correct response to pressure for that class is defer, never discard.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   migration traffic, placed on 32.1 §3's two axes

     deadline    : NO. Nothing stalls waiting for a migration to
                   complete -- the page is readable at its old
                   location until the move commits.
     droppable   : NO. Discarding it loses the policy's decision
                   and may lose data if the move is destructive.

   -> the writeback class exactly, and 32.1 §3's response for that
      class is DEFER FREELY, NEVER DISCARD.

   so the controller CAN rate-limit migration traffic without
   violating anything, and that is §7 remedy one made available.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// migration_rate_limiter -- CORRECT as written. Grade-D bounds.
//
// CLASSIFICATION: synthesisable. It DELAYS and never drops.
//
// WHAT IT IS: §7's first remedy. It bounds the migration bytes
// admitted per window, so §6's loop swing is bounded by the limiter
// rather than by the policy's appetite -- which is what §7 says
// hysteresis cannot achieve.
//
// WHY IT EXISTS HERE: §7 establishes that a hysteresis band cannot
// be sized against a swing the controller's own action determines.
// Bounding the action bounds the swing, and 32.1 §3's droppability
// axis is what makes bounding LEGAL: migration traffic has no
// deadline, so deferring it violates nothing.
//
// HOW TO RUN IT: offer a migration burst far above the budget.
// EXPECTED RESULT: it is admitted at the budgeted rate over several
// windows, and NOTHING is dropped -- mig_deferred rises and
// mig_dropped does not exist as a concept in this block.
// EXPECTED TRACE: admitted bytes per window never exceed BUDGET,
// and the backlog drains monotonically once the offer stops.
//
// SYNTHESIS: one budget counter, one backlog counter. No memory --
// the migration data itself is elsewhere.
//
// LIMITATIONS: it bounds the RATE. It does not bound the TOTAL, so a
// policy that migrates continuously still consumes the budget
// continuously -- §12's churn monitor is what detects that, and this
// block cannot. Nor does it prioritise between migrations: 17.4 owns
// arbitration. Both boundaries are STATED and neither is the bug.
// ---------------------------------------------------------------------
module migration_rate_limiter #(
  // Grade D. The share of a window's bandwidth migrations may use.
  // On a real platform this is a CONFIGURATION input, per 32.1 §14:
  // a budget is a platform policy and a platform policy compiled in
  // is grade D consumed as grade A.
  parameter int WIN            = 65536,
  parameter int BUDGET_PCT     = 5,
  // COUNT, not INDEX: the backlog must represent a full window's
  // offer, so it needs $clog2(WIN + 1) bits. Sized to the BUDGET it
  // would saturate exactly when a burst exceeds the budget -- the
  // counter failing inside the case it exists to handle, and
  // silently losing a deferred migration.
  parameter int CNT_W          = $clog2(WIN + 1),
  parameter int BUDGET         = (WIN * BUDGET_PCT) / 100
)(
  input  logic               clk,
  input  logic               rst_n,

  input  logic               mig_offer,       // a migration beat offered
  input  logic               app_beat,        // an application beat
  input  logic               win_tick,

  output logic               mig_admit,
  output logic               mig_deferred,
  output logic [CNT_W-1:0]   backlog,
  output logic [CNT_W-1:0]   r_admitted,
  output logic [CNT_W-1:0]   r_offered,
  output logic               backlog_overflow,
  output logic               result_valid
);
  logic [CNT_W-1:0] spent, offered, backlog_q;

  initial begin
    if (BUDGET_PCT < 1 || BUDGET_PCT > 100)
      $fatal(1, "migration_rate_limiter: BUDGET_PCT must be 1..100");
    if (BUDGET < 1)
      $fatal(1, "migration_rate_limiter: BUDGET rounds to zero -- migrations could never complete");
    // §7's remedy must be BOUNDED IN BOTH DIRECTIONS. A budget of
    // 100% is indistinguishable from having no limiter, so the
    // mechanism would be present in the source and absent in the
    // silicon -- 31.2 §7's hazard.
    if (BUDGET_PCT == 100)
      $warning("migration_rate_limiter: BUDGET_PCT = 100 disables the limit (§7)");
  end

  // The admission decision. A migration beat is admitted while the
  // window's budget remains; otherwise it is DEFERRED into the
  // backlog. It is never dropped -- 32.1 §3's response for a
  // non-droppable class with no deadline.
  assign mig_admit    = mig_offer && (spent < BUDGET[CNT_W-1:0]);
  assign mig_deferred = mig_offer && !mig_admit;

  always_ff @(posedge clk) begin
    if (!rst_n) begin
      spent <= '0;  offered <= '0;  backlog_q <= '0;
      r_admitted <= '0;  r_offered <= '0;
      backlog_overflow <= 1'b0;  result_valid <= 1'b0;
    end else if (win_tick) begin
      r_admitted   <= spent;
      r_offered    <= offered;
      result_valid <= 1'b1;
      spent        <= '0;
      offered      <= '0;
      // The backlog SURVIVES the window boundary, because a deferred
      // migration is still owed. Clearing it here would be a silent
      // drop of a non-droppable class.
    end else begin
      result_valid <= 1'b0;
      if (mig_offer) begin
        if (offered != {CNT_W{1'b1}}) offered <= offered + 1'b1;
        if (mig_admit) begin
          if (spent != {CNT_W{1'b1}}) spent <= spent + 1'b1;
        end else begin
          // Deferred: it joins the backlog and is owed. Overflow is
          // REPORTED rather than absorbed, because an absorbed
          // backlog entry is a dropped migration and this class must
          // not be dropped.
          if (backlog_q == {CNT_W{1'b1}}) backlog_overflow <= 1'b1;
          else                            backlog_q <= backlog_q + 1'b1;
        end
      end
      // Drain the backlog with any budget the current offer left
      // unused, so a burst is spread across windows rather than lost.
      if (!mig_offer && (backlog_q != '0) && (spent < BUDGET[CNT_W-1:0])) begin
        backlog_q <= backlog_q - 1'b1;
        if (spent != {CNT_W{1'b1}}) spent <= spent + 1'b1;
      end
    end
  end

  assign backlog = backlog_q;
endmodule

The backlog surviving the window boundary is the block's load-bearing line. Clearing it at win_tick would be arithmetically tidy and would silently drop a non-droppable classCURRICULUM-DERIVED from 32.1 §3: a class with no deadline and no droppability is deferred freely and never discarded, and a counter reset is a discard with no log entry.

And backlog_overflow reports rather than saturates for the same reason. A saturating backlog quietly forgets migrations the policy believes it issued, so the policy's model of placement diverges from reality — which is the stale-state hazard 30.2 owns, arriving at the interface between a controller and a policy rather than between a controller and a device.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  // ---- The limiter's contract.
  // The bound itself. An invariant: there is no window on which
  // exceeding the budget is acceptable.
  property p_budget_respected;
    @(posedge clk) disable iff (!rst_n)
      result_valid |-> (r_admitted <= BUDGET);
  endproperty
  assert property (p_budget_respected)
    else $error("migration admissions exceeded the window budget");

  // NOTHING IS DROPPED. Every offered beat is either admitted or
  // backlogged -- 32.1 §3's non-droppable class, and this is the
  // property that distinguishes deferral from discard.
  property p_nothing_dropped;
    @(posedge clk) disable iff (!rst_n)
      mig_offer |-> (mig_admit ^ mig_deferred);
  endproperty
  assert property (p_nothing_dropped)
    else $error("an offered migration beat was neither admitted nor deferred");

  // The backlog must DRAIN, or deferral becomes indefinite
  // postponement -- which for a class with no deadline is still
  // wrong, because the policy is waiting. A bounded liveness
  // property, and the bound is JUSTIFIED: a backlog of B beats
  // drains in ceil(B / BUDGET) windows once the offer stops.
  property p_backlog_drains;
    @(posedge clk) disable iff (!rst_n)
      (backlog != '0 && !mig_offer) |->
        ##[1:(WIN * 2)] (backlog < $past(backlog));
  endproperty
  assert property (p_backlog_drains)
    else $error("the migration backlog did not drain with the offer removed");

  // ---- Covers: the two regimes and the boundary.
  // A burst ABOVE the budget -- the case the limiter exists for.
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && (r_offered > BUDGET));
  // A window entirely WITHIN the budget, so the pass-through path is
  // known to work rather than assumed.
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && (r_offered > 0) && (r_offered <= BUDGET));
  // The boundary itself -- 30.3 §10: a rule respected but never
  // exercised at its boundary has not been tested.
  cover property (@(posedge clk) disable iff (!rst_n)
                  mig_offer && (spent == BUDGET - 1));

And the tie back to this module's first chapter is worth stating explicitly, because it is the reason the remedy is available at all.

Chapter 32.1 §3's droppability axis was introduced for prefetches on a CPU platform. It licenses this chapter's rate limiter on a server platform, because it is what proves that deferring migration traffic violates nothing — and it is what forbids the tidier implementation that would have cleared the backlog at a window boundary.

14. RTL Review — A Latency Advisory

The intended contract:

  1. Publish, per address region, the irreducible cost — row-state work, device latency and transfer occupancy — as a sum and a sample count.
  2. Publish the queueing cost separately and labelled. Never sum the two into one figure.
  3. A region with fewer than MIN_SAMPLES samples must be reported stale, not published with a noisy estimate.
  4. The published irreducible cost must not change when offered load changes and the access pattern does not.
  5. An accumulator that would overflow must flag, not wrap — an understated cost attracts pages to a region.
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// latency_advisory -- INTENTIONALLY DEFECTIVE, for review (§14).
//
// CLASSIFICATION: synthesisable telemetry, grade-D values, CONTAINS
// A BUG. Its output is consumed by a CONTROL LOOP (§2 row five).
//
// WHAT IT IS MEANT TO DO: the five-clause contract above -- publish
// 23.1's decomposition per region so a placement policy can act on
// the part that is a property of the page.
//
// WHY IT EXISTS HERE: §5 proves a blended export loses
// discrimination under load; §6 proves it closes a delayed
// negative-feedback loop; §7 proves hysteresis cannot damp it. This
// block is the blended export, and it is the natural design --
// "publish the latency" is what anybody would ask for.
//
// HOW TO RUN IT: hold the access pattern fixed and raise the offered
// load.
// EXPECTED RESULT under clause 4: the published cost is UNCHANGED.
// EXPECTED TRACE: irr_sum must not move with load. It does.
//
// SYNTHESIS: per-region accumulators and a sample counter.
//
// LIMITATIONS: this tier only, as §11's block. That is STATED and is
// not the bug.
// ---------------------------------------------------------------------
module latency_advisory #(
  parameter int NUM_REGIONS  = 16,
  parameter int WIN          = 1048576,
  parameter int MIN_SAMPLES  = 64,
  parameter int LAT_W        = 20,
  parameter int ACC_W        = LAT_W + $clog2(WIN + 1),
  parameter int SMP_W        = $clog2(WIN + 1),
  parameter int RID_W        = $clog2(NUM_REGIONS > 1 ? NUM_REGIONS : 2)
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                acc_done,
  input  logic [RID_W-1:0]    acc_region,
  // Both components ARE on the interface. 23.1's decomposer already
  // separated them, so clause 2 is available to this block.
  input  logic [LAT_W-1:0]    acc_irreducible,
  input  logic [LAT_W-1:0]    acc_queueing,

  input  logic                win_tick,

  output logic [ACC_W-1:0]    irr_sum   [NUM_REGIONS],
  output logic [SMP_W-1:0]    samples   [NUM_REGIONS],
  output logic [NUM_REGIONS-1:0] advisory_stale,
  output logic                result_valid,
  output logic                acc_saturated
);
  logic [ACC_W-1:0] a_tot [NUM_REGIONS];
  logic [SMP_W-1:0] a_n   [NUM_REGIONS];

  initial begin
    if (NUM_REGIONS < 1) $fatal(1, "latency_advisory: NUM_REGIONS >= 1");
    if (WIN < MIN_SAMPLES)
      $fatal(1, "latency_advisory: WIN < MIN_SAMPLES -- no region can be non-stale");
  end

  always_ff @(posedge clk) begin
    if (!rst_n) begin
      for (int r = 0; r < NUM_REGIONS; r++) begin
        a_tot[r] <= '0;  a_n[r] <= '0;
        irr_sum[r] <= '0;  samples[r] <= '0;
      end
      // Clause 3's initialisation is CORRECT: everything starts
      // stale, so an unmeasured region is never published as
      // zero-cost. A reviewer checking "can an unmeasured region
      // attract pages?" finds that it cannot.
      advisory_stale <= '1;
      result_valid   <= 1'b0;
      acc_saturated  <= 1'b0;
    end else if (win_tick) begin
      for (int r = 0; r < NUM_REGIONS; r++) begin
        irr_sum[r] <= a_tot[r];
        samples[r] <= a_n[r];
        // Clause 3 is HONOURED per region as well.
        advisory_stale[r] <= (a_n[r] < MIN_SAMPLES[SMP_W-1:0]);
        a_tot[r] <= '0;  a_n[r] <= '0;
      end
      result_valid <= 1'b1;
    end else begin
      result_valid <= 1'b0;
      if (acc_done) begin
        // Clause 5 is HONOURED: overflow flags rather than wraps.
        if (a_tot[acc_region] > ({ACC_W{1'b1}} - acc_irreducible - acc_queueing))
          acc_saturated <= 1'b1;
        else
          // <-- THE DEFECT
          a_tot[acc_region] <= a_tot[acc_region] + acc_irreducible + acc_queueing;

        if (a_n[acc_region] != {SMP_W{1'b1}}) a_n[acc_region] <= a_n[acc_region] + 1'b1;
        else acc_saturated <= 1'b1;
      end
    end
  end
endmodule

Before reading on: which clause, and why does every low-load test pass?

15. The Defect — Correct at Low Load, a Loop at High Load

The violated clauses are 2 and 4, and the defect is one expression: a_tot accumulates acc_irreducible + acc_queueing, and the output is named irr_sum. There is no separate queueing output at all — the two components arrive separated on the interface and are summed on the way in.

Clauses 1, 3 and 5 are all honoured, and that is what conceals it. The staleness discipline is correct — everything starts stale and a thin region is reported unusable — and the overflow flag is correct. A reviewer checking the things that usually go wrong with a telemetry block finds them all right.

Now the arithmetic, and the load dependence is the whole failure.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   GRADE D, using §5's figures.

   region P, row-hit dominant, irreducible cost 20 units.
   The access pattern is HELD FIXED throughout.

     offered load LOW      : queueing =   2  ->  published 22
     offered load MEDIUM   : queueing =  40  ->  published 60
     offered load HIGH     : queueing = 200  ->  published 220

   DERIVED: the published cost varies by 10x while the access
   pattern -- the only thing clause 4 says it may depend on --
   does not change at all.

   so the policy's input is  dL_adv/dload > 0, which is exactly
   §6's loop gain. With the CORRECT export the same quantity is
   ZERO, and §7's third remedy is the only one that makes it so.

Now the review question: why does every low-load test pass?

Because at low load the queueing term is small and the blended figure is the irreducible cost.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
   DERIVED from the figures above.

     at low load : published 22 against a true 20 -- a 10% error,
                   and it is CONSTANT, so a test comparing two
                   regions gets the ratio nearly right:
                       P: 22, Q: 50  ->  ratio 2.27
                       true             ->  ratio 2.40
                   a 5% discrimination error. Nobody notices.

     and the LOOP GAIN at low load is nearly zero, because the
     queueing term barely responds to load when the queue is
     empty. So §6's oscillation DOES NOT OCCUR in a low-load test.

   the defect's magnitude and its loop gain BOTH scale with load,
   and a functional test runs at whatever load it needs to
   exercise its cases -- which is usually not saturation.

And that is the third time in this module that a defect has been validated at an unrepresentative operating point. Chapter 32.1 §14's field position was validated on the first platform; 32.2 §13's throttle was validated at a bench thermal path; this advisory is validated at a bench load. The recurrence is worth stating as a module-level finding rather than three coincidences:

Three of this module's five defects are correct at the operating point where they were validated and wrong at the operating point where they ship. The common structure is that the defective term's magnitude scales with a quantity the test environment holds low — a platform constant, a thermal path, a load. So the validation condition must be chosen from where the defect is LARGEST, and the test engineer must be told which quantity that is.

The correction, and it is a split rather than a change:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
      // CORRECTED. Clauses 2 and 4: accumulate the two components
      // SEPARATELY and publish both. The irreducible sum then does
      // not move with load, so §6's loop gain is zero and §7's
      // delay stops mattering -- a loop with zero gain does not
      // oscillate at any delay.
      if (a_irr[acc_region] > ({ACC_W{1'b1}} - acc_irreducible))
        acc_saturated <= 1'b1;
      else
        a_irr[acc_region] <= a_irr[acc_region] + acc_irreducible;

      if (a_q[acc_region] > ({ACC_W{1'b1}} - acc_queueing))
        acc_saturated <= 1'b1;
      else
        a_q[acc_region] <= a_q[acc_region] + acc_queueing;

Three interface consequences, stated rather than assumed. The block gains a queue_sum output array, because clause 2 requires the load-dependent term to be published labelled rather than discarded — discarding it would be a second defect, since a policy legitimately wants to know DDR is busy as a capacity question. It gains a second accumulator array, roughly doubling the block's state. And irr_sum finally deserves its name, which it did not before — a signal whose name asserts a property the logic does not provide is a documentation defect on top of a functional one, and §17 asserts the property the name claims.

The general finding:

An advisory consumed by a control loop must publish only quantities that do not depend on the loop's own output. Any term that responds to the action the consumer will take is loop gain, and no amount of filtering, hysteresis or averaging on the consumer's side removes it — only excluding it from the export does.

16. SVA Review — Proving an Advisory Is Well-Formed

The properties written for §14's block, and all four are correct and conventional:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  // Offered as "proves the advisory is sound". All PASS on the
  // defective block.
  property p_unmeasured_region_is_stale;
    @(posedge clk) disable iff (!rst_n)
      (result_valid && samples[R] < MIN_SAMPLES) |-> advisory_stale[R];
  endproperty
  assert property (p_unmeasured_region_is_stale)
    else $error("a thin region was published as usable");

  property p_all_stale_after_reset;
    @(posedge clk) disable iff (!rst_n)
      $past(!rst_n) |-> (advisory_stale == '1);
  endproperty
  assert property (p_all_stale_after_reset)
    else $error("a region was usable before anything was measured");

  property p_overflow_flags_not_wraps;
    @(posedge clk) disable iff (!rst_n)
      (acc_done && (a_tot[acc_region] > LIMIT)) |=> acc_saturated;
  endproperty
  assert property (p_overflow_flags_not_wraps)
    else $error("an accumulator wrapped instead of flagging");

  property p_results_latched_cleanly;
    @(posedge clk) disable iff (!rst_n)
      (!win_tick) |=> $stable(irr_sum[R]);
  endproperty
  assert property (p_results_latched_cleanly)
    else $error("a published result changed outside a window tick");

Q. All four pass. What have they proved?

That the advisory is well-formed — stale when thin, stale at reset, flagging rather than wrapping, latched cleanly. Not one of them says what the published number is. Chapter 30.9 §6's variety 2 for the fifth chapter running, and the pattern is now the module's most consistent finding: the defect's key quantity is absent from the property set every single time.

And there is a sharper diagnosis available here. The contract's clause 4 — the published cost must not change when load changes and the pattern does not — is a property about a relationship between two runs, not about one run. CURRICULUM-DERIVED from 30.6 §11's variety 3: an obligation that compares against an alternative needs a model, not a property. So clause 4 is only checkable as a property if one of the two runs is turned into a signal — which is what acc_queueing already is.

What actually covers clauses 2 and 4:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  // Clause 4, and it names acc_queueing -- the signal the defective
  // block sums in and never publishes. Variety 2's repair, and it is
  // checkable within one run because the load-dependent term is
  // itself an input.
  property p_irreducible_excludes_queueing;
    @(posedge clk) disable iff (!rst_n)
      (acc_done && !win_tick) |=>
        (a_irr[$past(acc_region)] ==
           $past(a_irr[acc_region]) + $past(acc_irreducible));
  endproperty
  assert property (p_irreducible_excludes_queueing)
    else $error("the irreducible accumulator absorbed a queueing term");

  // Clause 2: the queueing term must be PUBLISHED, not discarded.
  // Stated separately because discarding it is a different defect
  // from blending it, and the correction to one can produce the
  // other.
  property p_queueing_is_published;
    @(posedge clk) disable iff (!rst_n)
      (result_valid && !advisory_stale[R]) |-> (queue_sum[R] != 'x);
  endproperty
  assert property (p_queueing_is_published)
    else $error("the queueing component was not published");

  // The SUM must reconstruct the total, which is 23.2 §9's
  // exhaustive-and-disjoint standard: the components sum exactly, so
  // a consumer wanting the total can compute it and a reviewer can
  // check the decomposition.
  property p_components_reconstruct_total;
    @(posedge clk) disable iff (!rst_n)
      result_valid |-> ((irr_sum[R] + queue_sum[R]) == total_observed[R]);
  endproperty
  assert property (p_components_reconstruct_total)
    else $error("the published components do not sum to the observed total");

  // And the property that encodes clause 4's LOAD-INDEPENDENCE
  // directly, as the invariant a control-loop consumer depends on.
  // It requires an environment signal for the offered load, which is
  // the point: the claim is about a relationship to load, so load
  // must appear.
  property p_irreducible_independent_of_load;
    @(posedge clk) disable iff (!rst_n)
      ($changed(offered_load) && $stable(pattern_id)) |=>
        $stable(irr_rate[R]);
  endproperty
  assert property (p_irreducible_independent_of_load)
    else $error("the irreducible rate moved with offered load");

p_irreducible_independent_of_load is the property this chapter exists to produce, and it has a cost worth naming: it requires the environment to supply an offered-load signal and a pattern identifier. CURRICULUM-DERIVED from 30.9 §2's obligation kinds — this is a safety obligation about a derived quantity, and a property about independence needs both variables present. A testbench that cannot vary load while holding the pattern fixed cannot check it, which is a stimulus requirement rather than a property one.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  // ---- Covers. The dimension this defect scales with is OFFERED
  //      LOAD, and 31.3 §15's rule says the cover must be on that
  //      dimension.
  // The HIGH-LOAD regime: the only one in which the defect is
  // material. §15 shows a low-load test sees a 5% discrimination
  // error and no oscillation at all.
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && (queue_sum[R] > irr_sum[R]));
  // The LOW-LOAD regime, so both are known to have run.
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && (queue_sum[R] * 4 < irr_sum[R]));
  // A load CHANGE with the pattern held -- the antecedent of
  // p_irreducible_independent_of_load, and a stimulus most
  // environments do not generate because they vary both together.
  cover property (@(posedge clk) disable iff (!rst_n)
                  $changed(offered_load) && $stable(pattern_id));
  // A stale region and a usable one, so clause 3 is exercised in
  // both directions rather than only at reset.
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && advisory_stale[R]);
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && !advisory_stale[R]);

The third cover is the one that matters and the one most environments cannot produce. A traffic generator that raises load by adding streams usually changes the access pattern at the same time — so $changed(offered_load) && $stable(pattern_id) requires a generator that can scale rate independently of pattern. CURRICULUM-DERIVED from 31.4 §14's rule: when the missing dimension belongs to the environment's model rather than its stimulus, running longer never reaches it — and here the missing capability is the generator's ability to hold one variable fixed.

17. What the Export Block's Assertions Prove

§16 reviewed the defective advisory. §11's block is the one whose output a control loop consumes, and its own contract deserves the stricter treatment.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  // The property §15 says `irr_sum` finally deserves: the name
  // asserts load-independence, so the logic must provide it. A
  // signal whose name claims a property the logic does not give is
  // a documentation defect on top of a functional one.
  property p_irr_sum_earns_its_name;
    @(posedge clk) disable iff (!rst_n)
      (acc_done && !win_tick) |=>
        (a_irr[$past(acc_region)] ==
           $past(a_irr[acc_region]) + $past(acc_irreducible));
  endproperty
  assert property (p_irr_sum_earns_its_name)
    else $error("irr_sum accumulated something other than the irreducible term");

  // The two accumulators must be INDEPENDENT -- the same isolation
  // class 31.3 §15 named for channels, 32.3 §16 for credit pools, and
  // here for decomposition components. Vacuously true in a blended
  // design, so a design grown from one has never had it checked.
  property p_components_are_independent;
    @(posedge clk) disable iff (!rst_n)
      (acc_done && acc_queueing != '0 && acc_irreducible == '0) |=>
        $stable(a_irr[$past(acc_region)]);
  endproperty
  assert property (p_components_are_independent)
    else $error("a pure-queueing sample moved the irreducible accumulator");

  // Clause 3, both directions. A region must be stale when thin AND
  // usable when thick -- a one-sided property is satisfied by a
  // block that reports everything stale forever, which would be
  // safe and useless (30.3 §9's variety 8).
  property p_staleness_is_two_sided;
    @(posedge clk) disable iff (!rst_n)
      result_valid |-> (advisory_stale[R] == (samples[R] < MIN_SAMPLES));
  endproperty
  assert property (p_staleness_is_two_sided)
    else $error("staleness does not match the sample count");

  // A stale region's numbers must not be actionable. The strongest
  // available statement is that the consumer is TOLD -- and §11's
  // reset-to-all-ones is what makes the unmeasured case safe rather
  // than attractive (30.7 §9: unknown is not a default).
  property p_stale_regions_flagged_before_publication;
    @(posedge clk) disable iff (!rst_n)
      (result_valid && samples[R] == '0) |-> advisory_stale[R];
  endproperty
  assert property (p_stale_regions_flagged_before_publication)
    else $error("a region with zero samples was published as usable");

  // The PLATFORM invariant, parameter in the ANTECEDENT and not in
  // the claim -- 31.1 §14's test. It may be shared across both arms.
  property p_single_tier_publishes_nothing_usable;
    @(posedge clk) disable iff (!rst_n)
      (!TIERED) -> (advisory_stale == '1);
  endproperty
  assert property (p_single_tier_publishes_nothing_usable)
    else $error("a single-tier platform published a usable advisory");

  // Saturation must be reported, because an understated cost
  // ATTRACTS pages -- the direction matters, and 30.8 §11's range
  // rule with its consequence named.
  property p_saturation_reported;
    @(posedge clk) disable iff (!rst_n)
      acc_saturated |-> ##[0:1] (advisory_stale != '0);
  endproperty
  assert property (p_saturation_reported)
    else $error("an accumulator saturated and the advisory stayed usable");

  // ---- Covers: the platform arms and both staleness directions.
  cover property (@(posedge clk) disable iff (!rst_n) TIERED && result_valid);
  cover property (@(posedge clk) disable iff (!rst_n) !TIERED && result_valid);
  cover property (@(posedge clk) disable iff (!rst_n)
                  result_valid && (samples[R] >= MIN_SAMPLES));
  cover property (@(posedge clk) disable iff (!rst_n) acc_saturated);

p_components_are_independent is the section's deliverable and it is the module's fourth instance of the same class. CURRICULUM-DERIVED from 31.3 §15 for replicated channels, 32.3 §16 for credit pools, and 32.3 §11 for queue pools: a design that separates things needs a property saying the separation holds, and that property is vacuously true in the combined predecessor it grew from. Here the separated things are not resources but components of a measurement, which is a domain the earlier instances did not cover.

And p_saturation_reported names the direction of the error, which matters more here than anywhere else in the module. An understated cost does not merely mislead — it attracts pages, so a saturating accumulator actively pulls traffic toward the region whose measurement broke. CURRICULUM-DERIVED from 30.8 §11's rule with its consequence made explicit: an instrument consumed by a control loop must fail toward unusable, never toward attractive.

18. What Would You Measure?

Q. You own the DDR controller on a tiered server platform. What do you measure, in what order?

MeasurementWhat it settlesCostOwner
Is the advisory blended or decomposed?§15's defect, answerable by reading one expressionhours§15
Does the placement policy consume the advisory automatically?§2 row five — whether the export is in a control loop at allhours§2
r_back_migrations against r_migrationsconverging policy versus thrashing one — §12days§12
r_bytes_moved, as a share of delivered bandwidth§6's cost, deductible from what applications could have haddays§12, 30.8 §2
irr_sum at two load levels, pattern held fixedclause 4 — and it needs a generator that can vary one variabledays§16
Correlate migration events with the advisory's swingwhether churn is §6's loop or a genuinely changing working setdays§12
The migration delay T_m and the window W§6's T_d, which sets the oscillation perioddays§6
advisory_stale counts per regionwhether the policy is acting on regions nobody measuredfree§11

Rows one and two cost hours and together decide whether any of this matters. A decomposed advisory has no loop gain, and an advisory nobody consumes automatically has no loop at all — so two questions answered by reading code and a configuration eliminate the whole failure class or confirm it.

Row five is the one that needs a capability rather than a measurement, and §16's third cover is why: most traffic generators raise load by adding streams, which changes the access pattern at the same time. Holding the pattern fixed while varying the rate is a generator feature, and if the environment lacks it, clause 4 was never checkable.

Row six is the disambiguation §12's block explicitly declines to perform. A legitimately changing working set also migrates backCURRICULUM-DERIVED from that block's stated limitations — so a high back-migration fraction is grade C about this run until it is correlated with the advisory's own swing. If the migrations track the advisory and the advisory tracks load, it is §6's loop; if the migrations track the application's phases, it is the policy working.

And row eight is free and easily overlooked. A policy acting on stale regions is making placement decisions on noise — and CURRICULUM-DERIVED from §11, the block reports exactly that, per region, at no cost.

19. Common Wrong Answers

“Export the memory latency so the policy can decide.” §5, §8. Which latency? A blended figure loses discrimination under load and closes a loop. Export the decomposition.

“Export the p99 instead of the mean.” §5. A tail percentile of a blended figure still contains the load-dependent term. The fix is decomposition, not a better statistic.

“Both tiers report a mean, so the comparison is fair.” §3, §5. Both tiers' owners independently rejected the mean — 23.1 §9 and /cxl/cxl-latency-anatomy — so a fair comparison of two rejected statistics is still wrong.

“Negative feedback is stabilising.” §6. Not when it is delayed, and here both the migration time and the measurement window contribute delay.

“Add hysteresis and the oscillation stops.” §7. Hysteresis damps a swing independent of the controller's action. Here the action sets the swing, so a band sized against the largest action cannot act at all.

“Shorten the window to reduce the delay.” §7. It helps, and it has a floor: a window too short reports a burst as a sustained state — 32.2 §5's failure in a different domain.

“The oscillation is harmless — it settles eventually.” §6. The cost is the migration traffic, which consumes rung-4 capacity for zero rung-5 value — 30.8 §2.

“A migration is the policy's business, not the controller's.” §2. It arrives as traffic the controller did not request and cannot decline, and it inflates the latency of everything queued behind it.

“Queueing is the biggest term, so it is the most important to export.” §4. It is the biggest and it is a property of the current load, not of the page. The policy deciding whether to send more traffic must not be told the current queue depth as a property of the destination.

“Publish a recommendation rather than raw numbers.” §8. The controller has one side of a two-sided comparison — the far tier is /cxl/cxl-latency-anatomy's — so a recommendation would be a decision taken with half the inputs.

“An unmeasured region can be published as zero cost.” §11. That makes it the most attractive region in the system. Unknown is not a default — 30.7 §9.

“Migration count tells you if the policy is working.” §12. A converging policy also migrates. Only the migrate-back fraction separates them, and it needs its denominator visible.

“The advisory assertions pass.” §16. They prove the advisory is well-formed and never say what the number is — variety 2, for the fifth chapter running.

“Clause 4 is unprovable because it compares two runs.” §16. It is provable because the load-dependent term is itself an input, so the comparison collapses into one run. The cost is that the environment must vary load with the pattern held.

“We tested the advisory thoroughly.” §15. At what load? The defect's magnitude and its loop gain both scale with load, and a functional test runs at whatever load exercises its cases.

“What is this server's tier composition?” §9. A category error with a validity period — it is a deployment choice changed during service, so there is no configuration to read once.

“This chapter should explain CXL.” §3. Fifteen live chapters in another track own it, including its latency anatomy and the instruction-to-port path. This chapter owns what the DDR controller must publish.

“The controller cannot bound migration traffic because it cannot decline it.” §13. It cannot drop it; it can defer it, and 32.1 §3's droppability axis is what proves deferral violates nothing.

“Clear the migration backlog at the window boundary.” §13. That is a silent drop of a non-droppable class, and the policy's model of placement then diverges from reality.

“A saturated accumulator is a minor telemetry bug.” §17. It understates a cost, and an understated cost attracts pages. An instrument in a control loop must fail toward unusable, never toward attractive.

20. Self-Check

  1. Give the five things that change on the DDR side when DDR is one tier, and say which one is new to this module.

  2. From §4's table, name the component that must be excluded from the export and state why — not that it is unimportant, but what it is a property of.

  3. Using §5's grade-D figures, compute the discrimination ratio under a blended export at queueing terms of 40 and 200. State the direction the error moves with load.

  4. Draw §6's loop as a signal path. State whether the feedback is positive or negative and contrast it with 32.2 §3's.

  5. Explain why a delayed negative-feedback loop oscillates, and name the two contributors to the delay on this platform.

  6. Explain why hysteresis is necessary and not sufficient here, and state the general rule about swings the controller's own action determines.

  7. Give §7's three remedies in increasing order of effectiveness, and say which one makes the delay irrelevant and why.

  8. State §8's three-part export contract, and the honest limit that stops it being a recommendation.

  9. Find the defect in §14 without reading §15. Then answer the harder question: why does every low-load test pass?

  10. State the module-level finding §15 draws from three of its five defects, and name the quantity each test environment held low.

  11. Explain why p_irreducible_independent_of_load needs two environment signals, and what a testbench must be able to do that most cannot.

  12. Place migration traffic on 32.1 §3's two axes and say what that licenses and what it forbids.

  13. Explain why an instrument consumed by a control loop must fail toward unusable, and give the specific harm of failing toward attractive.

21. Where This Goes — and the Module Closes

This platform class is the only one whose memory composition is chosen at deployment and changed in service. DDR stops owning the placement decision, and something above it consumes DDR's observed behaviour as an input — so the controller acquires an obligation to export a quantity a policy it does not own can safely act on. The mean is the wrong export for two independent reasons, and a blended figure closes a delayed negative-feedback loop that oscillates rather than amplifying; hysteresis is necessary and insufficient because the loop's own action sets the swing; and the only remedy that removes the oscillation rather than damping it is to export the component with zero load-dependence.


And the module closes on the law it opened with.

A CASE STUDY IS NOT A PRODUCT DESCRIPTION. IT IS A DESIGN DECISION, THE CONSTRAINT THAT FORCED IT, AND THE EVIDENCE GRADE OF EVERY CLAIM MADE ABOUT IT.

Five platform classes, five dominating constraints, and every one of them is a property of the requester rather than of the DRAM.

Platform classThe dominating constraintWhere it lives
CPUa heterogeneous requester mix with unequal obligationsthe requesters' kinds
Mobilea heat ceiling the memory closes a loop onthe enclosure
GPU-classa bimodal mix whose halves want opposite thingsthe requesters' patience
Acceleratorboth halves of the architecture committed before the workload existsthe schedule
Server — this chapterDDR is one tier, and it no longer owns the placement decisionthe deployment

Axis A1 — the cell — is identical in all five. CURRICULUM-DERIVED from 32.1 §1: the same thirteen obligations, none removed, none of the same kind added. So the module's single most useful result is that a platform's memory design is decided by who is asking and under what external limit, and never by the DRAM — which is 31.4 §19's finding for technology comparisons, arriving again for platform comparisons.

Three structural findings hold across all five.

Share and importance are anti-correlated, twice, for different reasons. Chapter 32.1 §7 found the class with the strongest obligation had the smallest share; 32.3 §2 found the population with the small share gated the large one's ability to start. Both rule out every policy that allocates attention in proportion to traffic, and both point at 17.4 §4's layered mask as the structure that survives.

The deciding quantity came from outside the memory team three times. theta in 32.2 §9, c/V in 32.4 §9, and the launch interval in 32.3 §8. So a memory-side analysis is incomplete by construction until somebody else supplies a number — and in two of the three cases the two possible assumptions recommend opposite actions. An analysis that does not name its external input has silently assumed one.

And three of the five defects were correct at the operating point where they were validated. A platform constant held fixed (32.1 §14), a thermal path held cool (32.2 §13), a load held low (§15). CURRICULUM-DERIVED from 31.4 §14's coverage rule: the coverage item must be on the dimension the defect scales with, and when that dimension belongs to the environment's model rather than its stimulus, running longer never reaches it. This module's contribution is to name the dimension in each case — and to observe that the test was correct every time.

The module also contributed one new entry to 30.9 §6's taxonomy. Variety 11 — evidence-grade escape, from 32.1 §15: a property can prove a configurable mechanism was consulted and cannot prove the configured value is correct, because the correctness of a grade-B value is a property of the platform rather than of the design. Varieties 1 to 10 are fixable by writing a better property; variety 11 is not, and its discharge is a read-back gate.

And the evidence discipline held throughout. CURRICULUM-DERIVED from 18.4 §1's four grades: every platform figure in these five chapters is grade D and labelled at use, every firm claim is CURRICULUM-DERIVED with its chapter and section named, and no product, vendor or part number appears anywhere. Chapter 32.4 §2 states the reason plainly, and it is the reason for all five: the facts a product tour would need are not published, so the alternatives were grade D dressed as grade A — 18.4 §1's category drift, manufactured deliberately — or case studies of decisions. This module chose the decisions, and the arithmetic in every chapter is reproducible without a single fact nobody published.

Continue learning

Standards & specifications

Governing standard
JEDEC JESD79 (DDR SDRAM)(opens JEDEC Solid State Technology Association in a new tab)

Defines the DDR SDRAM device itself — signals, command encoding, mode registers, timing parameters and the initialisation sequence — one document per generation. Memory-controller microarchitecture, address-mapping policy, PHY training algorithms and board-level design are not specified by it.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the DDR curriculum.