Persistent-Memory Devices

17.1 built the expander: a device whose media a host cannot inspect, so every property that matters has to be reported and then checked.

This chapter changes one thing about that device — the media does not forget — and follows what that single change costs.

1. The Engineering Problem — A Write That Returned Is Not A Write That Survived

Six things separate a persistent device from the DRAM expander of 17.1.

Acknowledgement and durability are two events, not one. A write is accepted in one cycle and reaches the media in another, and everything between them is data the device is holding and the host believes is safe. Section 5.

A flush is a wait, not a command. It completes when the buffer is empty, and a device that reports completion when the flush is issued has removed the only mechanism a host has. Section 6.

Power loss has an energy budget. The device holds writes it has not yet retired, and retiring them after the power goes costs stored energy. Accepting more than the energy covers is an overcommit that is invisible until the moment it is not. Section 7.

A dirty shutdown must be recorded by the device. The next boot cannot infer it. A device that cannot say "I lost power with data in flight" makes the following boot trust records it should be scrubbing. Section 9.

A persistent region must be re-found after reboot, and identity is the label. Binding by slot is right until somebody moves a device, and then it is silently wrong in the worst available way. Section 10.

And the media wears. A DRAM expander has no write-endurance limit; a persistent one does, which makes where a write lands a property of the device rather than of the request. Section 11.

This chapter against 17.1, stated precisely. That chapter owns what a memory device is and what it can be trusted to say about itself. This one owns what changes when the data has to still be there afterwards. Every section here would be vacuous on a volatile device.

2. The One-Sentence Model

Persistence is a claim about the future made at a moment in the past, so every mechanism in a persistent device exists to make that claim checkable — and every defect below is the device making the claim earlier than it can support.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

The models below are small on purpose. Four wear blocks, an eight-entry write buffer, a five-write energy budget and a twelve-write endurance limit are all far smaller than any real device, and they are sized so a reader can count the result by hand and so every boundary is reachable inside a short simulation.

What is not simplified is the structure: the separation of acknowledgement from durability, the gating of a flush on an empty buffer, the comparison of in-flight work against stored energy, and the five-bit failure mask are each shaped the way a real device shapes them. Scaling the numbers changes nothing about which check exists or where it sits.

Three things are deliberately absent. There is no media-error model — 17.1 section 10 owns that, and persistence does not change it. There is no interleaving across devices — 17.1 section 9 owns that. And there is no tiering policy — 17.3 owns that. A section here that could move into one of those chapters without loss is in the wrong chapter.

5. RTL 1 — The Durability Boundary

// A write that returned is not a write that persisted.
module pm_durability #(parameter int ACK_IS_PERSIST = 0) (
  input  logic clk, rst_n,
  input  logic       wr, persist_tick,
  output logic       wr_ack, persist_ack, media_take,
  output logic [7:0] n_acked, n_persisted, n_in_flight,
  output logic       durability_lie_err
);
  // A write is acknowledged the cycle it is accepted.
  assign wr_ack = wr;
  // The media takes one buffered write per persist tick. This is the only event
  // that actually makes data durable, in either build.
  assign media_take = persist_tick && (n_in_flight != 8'd0);
  // The broken build reports persistence at acknowledgement, so the two events
  // collapse into one and in-flight data stops existing as a concept.
  assign persist_ack = (ACK_IS_PERSIST != 0) ? wr : media_take;
  // Telling the host a write is durable in a cycle the media took nothing.
  assign durability_lie_err = persist_ack && !media_take;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_acked <= 8'd0; n_persisted <= 8'd0; n_in_flight <= 8'd0;
    end else begin
      if (wr_ack)     n_acked     <= n_acked + 8'd1;
      if (media_take) n_persisted <= n_persisted + 8'd1;
      // In-flight is the gap between acceptance and the media taking it.
      case ({wr_ack, media_take})
        2'b10: n_in_flight <= n_in_flight + 8'd1;
        2'b01: n_in_flight <= n_in_flight - 8'd1;
        default: n_in_flight <= n_in_flight;
      endcase
    end
  end
endmodule

Five writes and four media takes:

  durability: acked=5 persisted=4 in_flight=1 lies=0 | ack-is-persist build persisted=4 lies=4

Read the two numbers that are the same, not the two that differ. Both builds persisted four, because media_take is identical in both — the media does the same work either way. What differs is persist_ack, the signal the host sees. The broken build asserted it four times when the media had taken nothing, and its n_persisted counter still reads a perfectly honest 4.

The durability_lie_err check is written so it is provably absent in the correct build rather than merely unobserved: persist_ack and media_take are the same expression there, so the conjunction is structurally impossible. In the broken build it fired on every write that no media take accompanied — four of five, the fifth being the cycle where a write and a take coincided.

That coincident cycle is driven deliberately. {wr_ack, media_take} == 2'b11 leaves n_in_flight unchanged, and a model that never sees both in one cycle has an untested arm in its own case statement.

6. RTL 2 — A Flush Is A Wait, Not A Command

// A flush completes when the buffer is empty, not when it is issued.
module pm_flush #(parameter int FLUSH_IS_INSTANT = 0) (
  input  logic clk, rst_n,
  input  logic       flush_req, drain_tick, wr,
  output logic       flush_busy, flush_done,
  output logic [7:0] pending, n_flush_done, drain_cycles,
  output logic       early_done_err
);
  logic busy_q;          // Icarus binds in declaration order: declare before use.
  logic empty, take;
  assign empty = (pending == 8'd0);
  assign take  = drain_tick && !empty;
  assign flush_busy = busy_q;
  // The correct flush completes only when nothing is left to write.
  // The broken build completes the cycle it is asked.
  assign flush_done = (FLUSH_IS_INSTANT != 0) ? flush_req : (busy_q && empty);
  // Reporting a flush complete with data still buffered is the defect.
  assign early_done_err = flush_done && !empty;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending <= 8'd0; busy_q <= 1'b0; n_flush_done <= 8'd0; drain_cycles <= 8'd0;
    end else begin
      // Writes arrive whether or not a flush is running.
      case ({wr, take})
        2'b10: pending <= pending + 8'd1;
        2'b01: pending <= pending - 8'd1;
        default: pending <= pending;
      endcase
      if (flush_done)           busy_q <= 1'b0;
      else if (flush_req)       busy_q <= 1'b1;
      if (flush_done)           n_flush_done <= n_flush_done + 8'd1;
      if (busy_q && !flush_done) drain_cycles <= drain_cycles + 8'd1;
    end
  end
endmodule

Three flushes across the run:

  flush: done=3 drain_cycles=6 early=0 | instant build early=4

drain_cycles is the number a host would want and never gets to see: six cycles of real waiting, across three flushes, on a device whose instant-completing twin reported zero. The instant build is not slightly optimistic. It reports a flush that takes six cycles as taking none, four separate times, and the count of completed flushes is identical in both builds.

Three properties are driven that a casual bench would skip:

A drain tick with an empty buffer takes nothing. Three idle ticks leave pending at zero rather than underflowing it to 255. An eight-bit counter decremented past zero produces a buffer that will never drain again, and the failure appears cycles later as a flush that never completes.

A flush request held high across a multi-cycle drain completes exactly once. Holding the request is the natural thing for a host to do, and a busy_q whose priority is inverted re-arms every cycle and reports a completion every cycle. The mutation "busy priority inverted" is killed by this stimulus and by nothing else in the bench.

A flush of an already-empty buffer completes and adds no drain cycles. It is the case where the mechanism has nothing to do, and it must still terminate.

7. RTL 3 — The Energy Budget

// Power-loss protection: the energy budget must cover what is in flight.
module pm_power_loss #(parameter int IGNORE_BUDGET = 0) (
  input  logic clk, rst_n,
  input  logic        power_ok, wr, drain_tick,
  input  logic [15:0] energy_uj,        // hold-up energy available at power loss
  input  logic [15:0] uj_per_write,     // cost of pushing one buffered write to media
  output logic        accept_write,
  output logic [7:0]  in_flight, coverable, n_lost,
  output logic [15:0] energy_needed,
  output logic        data_loss_err, overcommit_err
);
  // Sixteen bits: 40 buffered writes at 800uJ each is 32000, which does not fit
  // in eight and would wrap to a plausible small number.
  assign energy_needed = uj_per_write * {8'd0, in_flight};
  // How many buffered writes the stored energy can actually retire. The quotient
  // needs its own name: a part-select of an expression is not legal here.
  logic [15:0] cov_q;
  assign cov_q     = (uj_per_write == 16'd0) ? 16'hFFFF : (energy_uj / uj_per_write);
  assign coverable = (cov_q > 16'd255) ? 8'hFF : cov_q[7:0];
  // Holding more than the energy can retire is an overcommit, whether or not
  // power is currently being lost.
  assign overcommit_err = (in_flight > coverable);
  // The correct build refuses the write that would cross the line. The broken
  // build accepts unconditionally and discovers the shortfall at power loss.
  assign accept_write = (IGNORE_BUDGET != 0) ? wr
                                             : (wr && ((in_flight + 8'd1) <= coverable));
  // Power lost while overcommitted loses exactly the uncovered writes.
  assign data_loss_err = !power_ok && overcommit_err;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      in_flight <= 8'd0; n_lost <= 8'd0;
    end else if (!power_ok) begin
      if (overcommit_err) n_lost <= n_lost + (in_flight - coverable);
      in_flight <= 8'd0;
    end else begin
      case ({accept_write, (drain_tick && (in_flight != 8'd0))})
        2'b10: in_flight <= in_flight + 8'd1;
        2'b01: in_flight <= in_flight - 8'd1;
        default: in_flight <= in_flight;
      endcase
    end
  end
endmodule

4000 µJ stored, 800 µJ to retire one buffered write, so the device can cover exactly five:

  power: coverable=5 in_flight=0 lost=0 | ignoring build in_flight=0 lost=3

Both builds accept the first five. The testbench then presents three more. The correct build refuses all three — the write that would make the buffer uncoverable is the write that does not happen — and the ignoring build accepts all three, reaching eight in flight against five covered.

Then the power is pulled, and the arithmetic is exact: three lost, which is in_flight - coverable, not "some data" and not "the whole buffer".

The width note in the code is not decoration. energy_needed is a product of two sixteen-bit quantities and forty buffered writes at 800 µJ is 32,000 — an eight-bit energy_needed reports that as 0, and a device needing thirty-two millijoules looks like a device needing none. This is the same class as the capacity_gain_pct defect in 17.1 section 18, and it is the second time in two chapters that a quantity which can exceed its neighbours' range was declared at their width by habit.

8. Waveform — A Write, A Flush, And The Power Going

Transcribed from the printed trace. One stimulus stream, both builds.

The flush_done row is flat at zero for the whole window on purpose: the flush was issued at cycle 4 and the buffer had not emptied by cycle 7. A host reading a completion here would be reading the instant build's answer.

9. RTL 4 — The Dirty-Shutdown Record

// The device must know it lost power with data in flight.
module pm_dirty_shutdown #(parameter int NO_DIRTY_FLAG = 0) (
  input  logic clk, rst_n,
  input  logic       power_ok, shutdown_req, flush_complete,
  input  logic [7:0] in_flight,
  output logic       clean_marker, dirty_shutdown,
  output logic [7:0] n_clean, n_dirty,
  output logic       silent_dirty_err
);
  // A clean shutdown is one that was asked for AND finished flushing.
  assign clean_marker = shutdown_req && flush_complete && (in_flight == 8'd0);
  // Power lost without a clean marker is a dirty shutdown. The broken build
  // has no dirty concept at all, so every restart looks clean.
  assign dirty_shutdown = (NO_DIRTY_FLAG != 0) ? 1'b0
                                               : (!power_ok && !clean_marker);
  // Losing power with data in flight and not recording it is the defect: the
  // next boot cannot know its data is suspect.
  assign silent_dirty_err = !power_ok && (in_flight != 8'd0) && !dirty_shutdown;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_clean <= 8'd0; n_dirty <= 8'd0;
    end else begin
      if (clean_marker)   n_clean <= n_clean + 8'd1;
      if (dirty_shutdown) n_dirty <= n_dirty + 8'd1;
    end
  end
endmodule

  dirty: clean=1 dirty=2 silent=0 | no-flag build dirty=0 silent=1

The clean marker is a conjunction of three terms and each one is falsified alone in the bench, because a marker that is satisfied by any subset is a marker that will eventually be satisfied by the wrong subset:

The fourth case matters more than the three: power lost while the clean marker is asserted is not a dirty shutdown. Without driving that, dirty_shutdown = !power_ok is indistinguishable from !power_ok && !clean_marker, and a device that calls every orderly shutdown dirty makes the next boot scrub records that were never at risk. That mutation survived the first run for exactly this reason.

silent_dirty_err fires once in the no-flag build and never in the correct one. It requires data actually in flight — a power loss with an empty buffer is a loss of nothing, and recording it as a silent loss would make the check fire on a device behaving correctly.

10. RTL 5 — Identity Across A Reboot

// A persistent region must be re-found after reboot, by label not by slot.
module pm_region_identity #(parameter int MATCH_BY_SLOT = 0) (
  input  logic clk, rst_n,
  input  logic        boot,
  input  logic [15:0] stored_label, presented_label,
  input  logic [3:0]  stored_slot, present_slot,
  output logic        region_found, label_match, slot_match,
  output logic [7:0]  n_boot, n_found, n_misbound,
  output logic        misbind_err
);
  assign label_match = (stored_label == presented_label);
  assign slot_match  = (stored_slot  == present_slot);
  // Identity is the label. The broken build binds by slot, which is right until
  // somebody moves a device and then silently binds the wrong data.
  assign region_found = (MATCH_BY_SLOT != 0) ? slot_match : label_match;
  // Binding a region whose label does not match is attaching the wrong data.
  assign misbind_err = boot && region_found && !label_match;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_boot <= 8'd0; n_found <= 8'd0; n_misbound <= 8'd0;
    end else if (boot) begin
      n_boot <= n_boot + 8'd1;
      if (region_found) n_found <= n_found + 8'd1;
      if (misbind_err)  n_misbound <= n_misbound + 8'd1;
    end
  end
endmodule

Three boots:

  identity: boots=3 found=2 misbound=0 | slot build found=2 misbound=1

Both builds found two of three regions, and they are not the same two. That is the entire point of the model, and it is why n_found alone cannot distinguish them:

The slot build's two failures are opposite in character. Losing a moved device is visible: something that should be there is missing, and somebody investigates. Binding a different device's data as though it were yours is invisible, and the region mounts.

misbind_err is gated on boot, which means it has to be sampled while boot is high — the testbench's first attempt checked it a cycle later and read a signal that had already deasserted. That is the same delta-cycle discipline the RTL uses internally, applied to the bench.

11. RTL 6 — Wear, Which DRAM Does Not Have

// Persistent media has a finite write endurance, so where a write lands matters.
// DRAM has no such property, which is why this model has no counterpart in 17.1.
module pm_wear #(parameter int NO_LEVELING = 0) (
  input  logic clk, rst_n,
  input  logic       wr,
  input  logic [1:0] req_block,
  output logic [1:0] target_block,
  output logic [15:0] w0, w1, w2, w3,
  output logic [15:0] max_wear, min_wear, spread,
  output logic [7:0]  n_writes,
  output logic        endurance_err
);
  // Sixteen bits: a wear count is cycles-of-writes deep and an eight-bit counter
  // wraps long before endurance is reached, reporting a worn block as fresh.
  localparam logic [15:0] ENDURANCE = 16'd12;
  logic [1:0] rr_q;
  // The correct build spreads writes; the broken build writes where asked, which
  // is right for the request and wrong for the media.
  assign target_block = (NO_LEVELING != 0) ? req_block : rr_q;
  assign max_wear = (w0 > w1 ? (w0 > w2 ? (w0 > w3 ? w0 : w3) : (w2 > w3 ? w2 : w3))
                             : (w1 > w2 ? (w1 > w3 ? w1 : w3) : (w2 > w3 ? w2 : w3)));
  assign min_wear = (w0 < w1 ? (w0 < w2 ? (w0 < w3 ? w0 : w3) : (w2 < w3 ? w2 : w3))
                             : (w1 < w2 ? (w1 < w3 ? w1 : w3) : (w2 < w3 ? w2 : w3)));
  assign spread   = max_wear - min_wear;
  // One block reaching endurance retires the device, however fresh the others are.
  assign endurance_err = (max_wear >= ENDURANCE);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      w0 <= 16'd0; w1 <= 16'd0; w2 <= 16'd0; w3 <= 16'd0;
      rr_q <= 2'd0; n_writes <= 8'd0;
    end else if (wr) begin
      n_writes <= n_writes + 8'd1;
      rr_q <= rr_q + 2'd1;
      case (target_block)
        2'd0: w0 <= w0 + 16'd1;
        2'd1: w1 <= w1 + 16'd1;
        2'd2: w2 <= w2 + 16'd1;
        2'd3: w3 <= w3 + 16'd1;
      endcase
    end
  end
endmodule

Every write in the stimulus asks for block 0 — the natural pattern for a log, a journal, or a metadata region:

  wear: levelled spread=1 max=12 | no-levelling spread=47 max=47 retired=1

Twelve writes, all requesting block 0. The levelling build put three on each block and no block is near the limit. The build that honours the request put all twelve on block 0, which is exactly the endurance limit, while three of four blocks are untouched.

The device is retired with three quarters of its media unused. That is what a spread of twelve on a twelve-write budget means, and the arithmetic scales: the levelled device reached the same limit only after 47 writes, very nearly four times as many, which is the ratio a four-block model can express.

endurance_err is judged on max_wear, and the mutation that judges it on min_wear is worth stating plainly: a device is retired by its worst block, not its average one. A mean wear figure across a device whose hot block has been written to death describes a device that no longer works.

12. RTL 7 — Read And Write Do Not Cost The Same

// Persistent media is asymmetric: a write costs far more than a read, which
// DRAM timing assumptions do not express.
module pm_asymmetry #(parameter int SYMMETRIC_MODEL = 0) (
  input  logic clk, rst_n,
  input  logic        req, is_write,
  input  logic [15:0] rd_ns, wr_ns,
  output logic [15:0] this_cost,
  output logic [31:0] total_ns,
  output logic [15:0] n_rd, n_wr, mean_ns,
  output logic        underestimate_err
);
  logic [31:0] denom, mean_q;
  // The symmetric model charges the read cost for everything, which is right
  // for DRAM and wrong for every persistent device.
  assign this_cost = (SYMMETRIC_MODEL != 0) ? rd_ns : (is_write ? wr_ns : rd_ns);
  // Thirty-two bits: 60 requests at 900ns is 54000, past sixteen bits.
  assign denom = {16'd0, (n_rd + n_wr)};
  // Charging a write less than the media costs is the defect, and it is silent.
  assign underestimate_err = req && is_write && (this_cost < wr_ns);
 
  // The quotient needs its own name: a part-select of an expression is not legal,
  // and a continuous assign avoids the always_* constant-select restriction too.
  assign mean_q  = (denom == 32'd0) ? 32'd0 : (total_ns / denom);
  assign mean_ns = mean_q[15:0];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      total_ns <= 32'd0; n_rd <= 16'd0; n_wr <= 16'd0;
    end else if (req) begin
      total_ns <= total_ns + {16'd0, this_cost};
      if (is_write) n_wr <= n_wr + 16'd1;
      else          n_rd <= n_rd + 16'd1;
    end
  end
endmodule

Four reads at 100 ns and four writes at 900 ns:

  asymmetry: total=4000ns mean=500ns | symmetric total=800ns mean=100ns under=4

A factor of five in the mean, from one ternary. The symmetric model is not approximately right — it reports the same 100 ns whatever the read/write mix is, so a workload that shifts from mostly-read to mostly-write shows no modelled change at all while its real latency rises fivefold.

total_ns is thirty-two bits because sixty requests at 900 ns is 54,000, past sixteen. A sixteen-bit total would wrap at request forty-nine and report a rising latency as a falling one — the most misleading available failure for a quantity a capacity planner reads directly.

13. RTL 8 — What Is Valid After A Dirty Boot

// After a dirty shutdown, what is valid? Not everything, and not nothing.
module pm_recovery #(parameter int TRUST_ALL = 0) (
  input  logic clk, rst_n,
  input  logic       boot, dirty, region_flushed,
  input  logic [7:0] committed, at_risk,
  output logic [7:0] valid_records, suspect_records,
  output logic [7:0] n_recovered, n_scrubbed,
  output logic       recovery_needed, false_trust_err
);
  // A clean boot trusts everything. A dirty boot trusts only what was committed
  // before the loss; the at-risk records must be scrubbed, not read.
  assign recovery_needed = dirty && !region_flushed;
  assign valid_records   = (TRUST_ALL != 0) ? (committed + at_risk)
                         : (recovery_needed ? committed : (committed + at_risk));
  assign suspect_records = (TRUST_ALL != 0) ? 8'd0
                         : (recovery_needed ? at_risk : 8'd0);
  // Returning a record that was in flight when power was lost, as though it
  // were committed, is the defect that makes persistence worse than useless.
  assign false_trust_err = boot && recovery_needed && (at_risk != 8'd0)
                           && (suspect_records == 8'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_recovered <= 8'd0; n_scrubbed <= 8'd0;
    end else if (boot) begin
      n_recovered <= n_recovered + 8'd1;
      if (recovery_needed) n_scrubbed <= n_scrubbed + suspect_records;
    end
  end
endmodule

Forty committed records and six at risk:

  recovery: valid=46 suspect=0 scrubbed=6 | trust-all valid=46 scrubbed=0

Three boots, and the third is the one that keeps the model honest:

The third row is why recovery_needed is a conjunction. A dirty flag on a region whose flush completed before the power went is a region with nothing at risk, and scrubbing it discards six perfectly good records for no reason. Without that row in the bench, recovery_needed = dirty is indistinguishable from the correct expression.

false_trust_err had to be observed on the clean path too. The mutation false_trust_err = boot && (suspect_records == 0) fires on every clean boot — where nothing is suspect and nothing should be — and the first run of the bench never looked at the signal except during the dirty boot. A checker asserted only where it is supposed to fire is a checker with half its specification untested.

14. RTL 9 — Volatile And Persistent Are Different Capacity

// A Type 3 device can present volatile and persistent capacity, and the two are
// not interchangeable. Mode is a property of the region, not the device.
module pm_mode_capacity #(parameter int ONE_MODE = 0) (
  input  logic clk, rst_n,
  input  logic       configure, want_persist,
  input  logic [7:0] req_gb,
  input  logic [7:0] vol_gb, pers_gb,
  output logic       granted,
  output logic [7:0] vol_free, pers_free, n_granted, n_refused,
  output logic       wrong_pool_err
);
  logic [7:0] pool_free;
  // The correct build allocates from the pool the request asked for. The
  // one-mode build has a single pool and hands out volatile capacity for a
  // persistent request, which succeeds and then loses the data at reboot.
  assign pool_free = (ONE_MODE != 0) ? vol_free
                                     : (want_persist ? pers_free : vol_free);
  assign granted   = configure && (req_gb <= pool_free);
  // Satisfying a persistent request out of the volatile pool.
  assign wrong_pool_err = granted && want_persist && (ONE_MODE != 0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      vol_free <= vol_gb; pers_free <= pers_gb; n_granted <= 8'd0; n_refused <= 8'd0;
    end else if (configure) begin
      if (granted) begin
        n_granted <= n_granted + 8'd1;
        if (ONE_MODE != 0)      vol_free  <= vol_free  - req_gb;
        else if (want_persist)  pers_free <= pers_free - req_gb;
        else                    vol_free  <= vol_free  - req_gb;
      end else n_refused <= n_refused + 8'd1;
    end
  end
endmodule

64 GB volatile, 32 GB persistent:

  mode: vol_free=0 pers_free=12 refused=2 | one-mode vol_free=3 pers_free=32

The one-mode build's final state is the whole argument: its persistent pool is untouched at 32 GB while it has served every persistent request out of volatile capacity. It refused nothing that the correct build refused, and it granted a second 20 GB persistent request that the correct build correctly turned down for want of persistent space.

Everything about it looks better until a reboot, at which point the data allocated as persistent is gone and the device still reports 32 GB of persistent capacity free.

The grant boundary is inclusive and driven exactly: a request for precisely the free capacity is granted, and one gigabyte more is refused. < instead of <= survives every test that does not sit on the boundary, and it silently strands the last allocatable region of every pool in the system.

15. RTL 10 — The Persistent Device Assembled

// Every gate that must pass before a host may treat a region as durable.
module pm_device #(parameter int SKIP_GATE = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       is_persistent,     // the device claims persistence
  input  logic       energy_ok,         // hold-up energy covers the buffer
  input  logic       dirty_flag_ok,     // the device can report a dirty shutdown
  input  logic       label_ok,          // the region is identified by label
  input  logic       flush_honoured,    // a flush waits for the media
  output logic       durable,
  output logic [4:0] fail_mask,
  output logic [7:0] n_eval, n_durable,
  output logic       undurable_claim_err
);
  // Each bit names one gate, so a refusal says which one. A single "not durable"
  // bit sends an engineer to check all five.
  assign fail_mask[0] = ~is_persistent;
  assign fail_mask[1] = ~energy_ok;
  assign fail_mask[2] = ~dirty_flag_ok;
  assign fail_mask[3] = ~label_ok;
  assign fail_mask[4] = ~flush_honoured;
  // The skipping build drops the energy gate, which is the one gate whose
  // failure is invisible until the power actually goes.
  assign durable = (SKIP_GATE != 0)
                 ? (is_persistent && dirty_flag_ok && label_ok && flush_honoured)
                 : (fail_mask == 5'd0);
  // Calling a region durable while a gate is failing.
  assign undurable_claim_err = evaluate && durable && (fail_mask != 5'd0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_durable <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (durable) n_durable <= n_durable + 8'd1;
    end
  end
endmodule

Six evaluations — one with all gates passing, then each gate falsified alone:

  device: evaluated=6 durable=1 | skip-gate durable=2

One durable region out of six, and the skipping build found two. The extra one is the energy failure, and the choice of which gate to drop is deliberate: it is the only one of the five whose failure produces no symptom whatsoever during normal operation. A device with no dirty flag, no label, or a lying flush misbehaves in ways somebody eventually notices. A device with insufficient hold-up energy is indistinguishable from a correct one until the power goes.

Each of the five gates is falsified alone, and each produces its own single-bit mask. That is what makes the mask worth having: a device refused with 5'b00010 sends an engineer to the hold-up energy, and a device refused with a single "not durable" bit sends them to all five.

16. Quantitative Reasoning

Every number here comes from a printed simulation line above. None is a specification.

The durability window. Five writes acknowledged, four taken by the media, one in flight. The window is one write deep in this model and thousands deep in a real device, and its size is exactly the amount of data a power loss can take.

Flush cost. Six drain cycles across three flushes, reported as zero by the instant-completing build. The ratio is not the point; the point is that a build reporting zero reports zero whatever the real number is.

Energy arithmetic. 4000 µJ ÷ 800 µJ = five coverable writes. Eight in flight in the ignoring build. Loss on power failure = 8 − 5 = three, exactly.

Wear. Twelve writes to one block reaches a twelve-write endurance with three of four blocks untouched. Levelled, the same limit arrives at 47 writes — a factor of 3.9 on a four-block device, approaching the factor of 4 that perfect levelling would give.

Asymmetry. 4 × 100 + 4 × 900 = 4000 ns, mean 500. The symmetric model: 8 × 100 = 800 ns, mean 100. A fivefold error at a 50/50 mix, and the error grows with the write fraction.

Recovery. 40 committed + 6 at risk = 46. A dirty unflushed boot yields 40 valid and 6 suspect. The trust-all build yields 46 valid and 0 suspect — the same total, six of them wrong.

Capacity. 32 GB persistent, one 20 GB grant, 12 GB left, a second 20 GB request refused. The one-mode build granted both out of a 64 GB volatile pool and finished with its 32 GB persistent pool entirely unused.

17. Assertions

Every property is an immediate check written as cond !== 1'b1, sampled after a settle. 143 assertion sites across three testbenches.

18. Mutation Testing

99 mutations, one at a time, each required to make the baseline print RESULT: FAIL.

99 of 99 were killed.

The first run killed 86 and left 13 survivors:

Four of the thirteen are worth stating individually.

"Every power loss is dirty." dirty_shutdown = !power_ok survived because the bench never lost power while a clean marker was asserted. The two halves of the conjunction had each been driven, but never in the combination where the second one does the work. A device with this bug scrubs records after every orderly shutdown, which looks like caution and is data loss on a slower schedule.

"False trust without a dirty boot." false_trust_err = boot && (suspect_records == 0) fires on every clean boot. The bench asserted the checker only during the dirty boot — the place it is supposed to fire — and never during the clean boots where it must stay quiet.

"Grant boundary off by one." req_gb < pool_free survived until the bench asked for exactly the free capacity. This is the fourth consecutive batch in which an inclusive boundary was tested only from the inside.

"Drain takes from an empty buffer." Removing && !empty survived because no drain tick ever arrived with the buffer empty. The mutant underflows an eight-bit pending to 255, and a flush on that device never completes again — a hang produced by a missing three-word guard that no test in the first run could reach.

A representative sample:

19. Verification Strategy

Two builds of every model, one stimulus stream. The parameter is the only difference, so any divergence in the printed line is attributable to the one line the parameter changes. This is the same discipline as 17.1 and the rest of the CXL track.

Assert the signal separately from its counter. RTL 1 exists to make this concrete: the two builds have identical n_persisted, and the defect lives entirely in persist_ack. A verification plan that checks counts finds nothing.

Falsify each term of a conjunction alone. The clean marker has three terms and the bench drives all three single-term failures plus the case where the marker is asserted and something else varies. Two mutations survive without this and both are silent in the field.

Drive every boundary from both sides. Exactly the free capacity, and one more. Exactly the endurance limit, and the write before it. Exactly the coverable buffer depth, and the write that would exceed it.

Reach every arm of every case statement. {wr_ack, media_take} == 2'b11 needs a write and a media take in one cycle, which does not happen by accident in a short bench.

Sample gated signals while the gate is open. misbind_err is gated on boot and had to be checked before the boot pulse cleared — the bench's own delta-cycle discipline, matching the RTL's.

20. Synthesis and Implementation Reality

The energy comparison is a divider, and it should not be. energy_uj / uj_per_write is written that way for readability; a real device stores the coverable depth as a configured constant computed once at initialisation, because a divider in the write-accept path is a timing problem in exchange for arithmetic nothing needs at line rate.

The write buffer is the physical design problem. It has to sit inside the power-loss domain — on the capacitor's side of the isolation — which constrains where it can be placed and how large it can be. Every entry added to the buffer is stored energy that must be added to cover it, so buffer depth is a board-level decision, not an RTL one.

Wear levelling has state that must itself be persistent. The round-robin pointer in RTL 6 is a register that resets to zero, which on a real device restarts levelling from the same block after every power cycle and concentrates wear exactly where levelling was supposed to prevent it. The pointer belongs in persistent metadata, and that metadata is itself written on every level operation, so it wears too.

The five gates are configuration reads, not logic. pm_device is drawn as combinational for clarity. On a real device four of the five are values read from device registers at enumeration and one — flush_honoured — cannot be read at all and has to be established by test.

Sixteen- and thirty-two-bit counters cost area and are not optional. Section 12's total_ns and section 11's wear counters are wide because the quantities they hold are wide. Narrowing them to save flops reintroduces exactly the truncation this chapter and 17.1 both found the hard way.

21. Silicon Observability

What a deployed persistent device must be able to tell you, and what it usually cannot.

The last row is the hard one. Rated hold-up energy is a datasheet number; measured hold-up energy declines over a device's life as the storage element ages, and a device that shipped with adequate margin can lose it silently years later. A device that reports only the rated figure is reporting a number that was true once.

The wear spread is the second-hardest. A device reporting mean wear reports a number that stays comfortable while a single block is written to death, and the mutation "endurance judged on the freshest block" in section 18 is that reporting choice written as RTL.

22. Debug Lab

Symptom: an application loses the last few seconds of writes after a power event, but the device reports zero errors.

Nothing is broken in the sense the device measures. Work through it in order.

Is the dirty-shutdown counter incrementing? If it is zero after a known power loss, the device cannot record dirty shutdowns and every boot has been taking the clean path. Stop here — this is section 9's defect, and all recovery decisions since deployment have been made on wrong information.

Does a flush actually wait? Issue a flush with a known-full buffer and measure the elapsed time. Zero is the answer from section 6's broken build. A flush that returns immediately makes every fsync in the stack a no-op.

Compare in-flight depth against coverable depth. If the device will accept more buffered writes than its hold-up energy retires, the loss is section 7's and its size is exactly the difference. This is measurable before a power event and is the only one of the five that can be.

Check whether the region was bound by label or by slot. If any device has been physically moved since deployment, and regions bind by slot, the data being read is a different device's. This presents as corruption rather than loss, and it is worth ruling out early because it invalidates every other measurement.

Compare the write count against the wear spread. A device retiring far earlier than its endurance rating suggests, with a large spread, has levelling that is not working — or a levelling pointer that resets on every power cycle, which is section 20's implementation note appearing as a field failure.

23. Design Review

Questions to ask about a persistent device design, each of which this chapter's models make answerable.

What separates your write acknowledgement from your durability acknowledgement? If the answer is "nothing", the device has no durability boundary and section 5 is describing it.

How many cycles does a flush take on a full buffer, and can the host see that number? A device that cannot report drain cost cannot be capacity-planned against.

What is the coverable buffer depth, and is it enforced by backpressure? If writes are accepted beyond it, the device has an overcommit whose size is known and whose consequence is not.

Where is the dirty-shutdown flag stored, and what writes it? It must be written by the power-loss path, not by the shutdown path — a flag written during an orderly shutdown records nothing about the shutdowns that were not orderly.

Is the wear-levelling pointer persistent? If it resets to zero, levelling restarts from the same block every boot.

Is the region identity a label or a position? And if it is a label, what happens when two devices present the same one?

24. How This Appears In Real Engineering

Persistence bugs are reported as data-loss incidents, not as device faults, and they arrive with the device reporting itself healthy.

The characteristic report is a database or filesystem losing committed transactions after a power event on hardware that shows no errors. The stack did everything correctly — it wrote, it flushed, it got a completion — and the completion was a lie from section 6. The investigation usually starts in the application because that is where the loss is visible.

The second characteristic report is capacity that behaves like memory. Somebody allocates a persistent region, uses it, reboots, and finds it empty, on a device that reports plenty of persistent capacity free. That is section 14, and the tell is that the persistent pool is untouched.

The third is a device that retires far too early. Endurance ratings are per-block and levelling is what turns them into a device rating; a device where levelling has failed retires at close to its single-block figure, which can be a small fraction of what was planned for.

25. Common Misconceptions

"The write returned, so the data is safe." The write returned when the device accepted it. Section 5 is a whole model about the gap, and its size is a device property nobody publishes.

"A flush completed, so everything before it is durable." Only if the flush waited. Section 6's instant build returns a completion for a flush that has done nothing, and both builds report the same number of completed flushes.

"Power-loss protection means data is not lost." It means data up to the covered depth is not lost. Section 7's device loses exactly in_flight - coverable writes, and a device that accepts writes without checking that difference has protection that covers an unknown fraction of what it holds.

"A clean boot means the last shutdown was clean." It means the device did not report otherwise. A device that cannot record a dirty shutdown reports clean boots forever, which is section 9.

"Persistent memory is DRAM that remembers." It is media with different read and write costs (section 12) and a finite write endurance (section 11), presented through the same interface. The interface being identical is the point of the standard; the timing being identical is an assumption that costs a factor of five at a 50/50 mix.

"Wear levelling is the device's problem." It is, until it stops working, at which point it becomes a retirement schedule nobody planned for. And levelling state that resets at power-on is levelling that does not work, in a way no functional test detects.

26. Interview Reasoning

Q1. What is the difference between a write acknowledgement and a persist acknowledgement? Time, and an amount of data. The acknowledgement says the device took it; the persist says the media has it. Everything between the two is data the host believes is safe and the device is still holding.

Q2. A device reports zero data loss after a power event and the application lost transactions. Where do you look first? The dirty-shutdown counter. If it is zero, the device cannot record dirty shutdowns and the "zero data loss" report carries no information at all.

Q3. How do you test whether a flush actually waits? Fill the buffer, issue a flush, measure the elapsed time. A flush that returns in the cycle it is issued is section 6's broken build, and the completed-flush count looks identical either way.

Q4. What does hold-up energy actually buy you? The right to have accepted a certain number of writes. It is not a recovery mechanism; it is the reason a device may say yes to a write at all, and the enforcement is backpressure before the fact.

Q5. A device accepts eight buffered writes and its energy covers five. How much data does a power loss cost? Three writes. Not "some" and not "the buffer" — the difference is exact, and it is computable while everything is working.

Q6. Why bind a persistent region by label rather than slot? Because devices move. Binding by slot loses a moved device, which is visible, and binds a replacement device's data as though it were yours, which is not.

Q7. Which of those two slot failures is worse? The second. A region that will not mount produces an error somebody investigates. A region that mounts with someone else's data produces a working system reading the wrong bytes.

Q8. Why does write endurance change the design and read endurance does not? Because it makes placement a device decision. A DRAM expander can put a write where the address says; a persistent device that does so retires at its hottest block while most of its media is untouched.

Q9. A device retires at a quarter of its rated endurance. What is your first hypothesis? Levelling is not working — and the most likely specific cause is that the levelling pointer is not persistent, so it restarts from the same block after every power cycle.

Q10. Why is a mean wear figure misleading? Because a device is retired by its worst block. A mean stays comfortable while one block is written to death, and the mean is the number most likely to be the one reported.

Q11. What does a symmetric latency model get wrong about persistent media? The write cost, and therefore the mix sensitivity. It reports the same mean for a read-heavy and a write-heavy workload, so it cannot be wrong about a particular one — it cannot be right about any.

Q12. After a dirty shutdown with 40 committed and 6 in-flight records, what is valid? Forty. The six are suspect and must be scrubbed rather than read. A device returning 46 has produced the same total and six wrong answers.

Q13. When is a dirty flag not a reason to scrub? When the region's flush completed before the power went. The flag says power was lost; the flush says nothing was at risk, and scrubbing on the flag alone discards good records.

Q14. A persistent allocation succeeds and the data is gone after reboot. What happened? It was satisfied from volatile capacity. The tell is that the device still reports its persistent pool free — section 14's one-mode build, where mode is a device property rather than a region property.

Q15. Which durability gate has no operational symptom? Hold-up energy. A device without a dirty flag, without label binding, or with a lying flush misbehaves in ways somebody eventually notices. A device with insufficient energy behaves perfectly until the power goes.

Q16. Why does the failure mask have five bits rather than one? Because "not durable" sends an engineer to check all five gates. 5'b00010 sends them to the hold-up energy. The bits cost nothing and the difference is the whole of the debugging time.

27. Exercises

1. Extend RTL 1 so the buffer holds identified writes and the durability acknowledgement names which write became durable. What does the host now know that a count could not tell it?

2. In RTL 2, make drain_tick retire two entries per cycle. Does drain_cycles still measure what a host needs? What would you report instead?

3. Add a second energy source to RTL 3 that is available only above a temperature threshold. Which of the existing assertions still hold, and which now need a condition?

4. In RTL 4, add a shutdown that is requested and begins flushing but loses power mid-drain. Which of the three clean-marker terms rejects it, and would any two of the three?

5. Make the wear-levelling pointer in RTL 6 persistent across reset. Re-run the 47-write experiment across three simulated power cycles and compare the spread against the resetting version.

6. RTL 7 charges a fixed cost per write. Make the write cost rise with the block's wear count, and determine at what wear level the asymmetry doubles.

7. In RTL 9, add a third pool for a region that may be either mode until first written. Which of wrong_pool_err's three conditions still applies?

8. Add a sixth gate to RTL 10 for media temperature. Where does it belong in the mask, and does its failure have an operational symptom?

28. Summary

A persistent device is the expander of 17.1 with one property added, and the property costs six mechanisms.

The durability boundary separates acknowledgement from persistence, and the data between them is the device's exposure. Both builds of RTL 1 persist the same amount; only one of them tells the truth about when.

A flush waits. Six drain cycles reported as zero, four times, by a build whose completed-flush count is identical to the correct one's.

Hold-up energy is spent by refusing writes, before the fact, while everything is fine. Eight in flight against five covered loses exactly three, and the difference is computable long before it matters.

A dirty shutdown must be recorded by the device, because the next boot cannot infer it, and the record must be a conjunction — a marker satisfied by any subset will eventually be satisfied by the wrong one.

Identity is a label, not a position. Binding by slot loses moved devices visibly and binds replacement devices invisibly, and only the first failure gets reported.

And the media wears, which makes placement the device's decision rather than the address's. Twelve writes to one block retires a device with three quarters of its media untouched; levelled, the same limit arrives at 47.

17.3 — Future Memory Systems takes these devices as built and asks what happens when a system has several kinds of memory at once, and has to decide which data lives where.