Flash

Three tiers in, and every one of them has shared a single limitation. A register file, an SRAM array and a DRAM device all lose their contents when power is removed. Chapter 1.4 made DRAM's storage as cheap as a random-access memory can be, but it did not make it persistent — DRAM forgets faster than any of them, which is why it needs refreshing even while it is powered.

So a system needs a tier that remembers without power, and this chapter is about it. But the interesting engineering question is not "what technology is non-volatile". It is:

Persistence is clearly valuable — so why can a persistent technology not simply take over the main-memory tier and make the whole problem disappear?

The answer is that persistence is not a free addition to a memory. It is a different storage mechanism, and it comes with three properties that a main-memory tier cannot tolerate: a write is not the inverse of a read, the unit you can clear is far larger than the unit you can address, and the storage wears out with use. Those three sentences are why flash is a storage tier rather than a memory tier, and why the DDR curriculum exists alongside it rather than being replaced by it.

1. Storing a Bit Without Power

Every mechanism so far has needed the supply. The static cell is a loop driven from it. The dynamic cell is charge on a capacitor that leaks away and is topped up by refresh — a process that cannot run with the power off.

Flash stores charge too, but it stores it somewhere fundamentally different: on a conductor that is electrically isolated from everything around it, inside the transistor itself. Because that storage node is surrounded by insulator rather than connected to a circuit, the charge on it has no ordinary path to leak through. It stays where it is put, without power, for a long time.

That charge shifts the threshold voltage of the transistor it sits in — the gate voltage at which the transistor begins to conduct. So reading the cell does not mean sensing a stored voltage directly; it means testing whether the transistor conducts under a particular applied condition. A charged cell and an uncharged cell respond differently to the same test, and that difference is the stored bit.

Two things follow from this immediately, and they set up everything else in the chapter.

Reading is cheap and harmless. Testing whether a transistor conducts does not move the stored charge. Unlike Chapter 1.4's destructive read, a flash read leaves the cell exactly as it was, and unlike Chapter 1.3's cell it needs no power to have retained it in the first place.

Changing the stored charge is a different kind of event entirely. Getting charge onto or off an isolated node means moving it through an insulator, and an insulator is precisely a material that resists that. It requires elevated voltages and a meaningful amount of time, and — this is the part that matters — it stresses the insulator every time it happens.

2. Consequence One — A Write Is Not the Inverse of a Read

In every tier so far, writing a location has been a complete operation: present an address and data, and afterwards the location holds the data. Flash does not work that way, and this is the single most consequential difference.

Charge can be added to a cell in a fine-grained, addressable way. Removing it, in the standard flash arrangement, cannot be done for one cell at a time — it is done for a large group of cells together. So the two halves of "write a new value" have completely different granularities:

• Program — move a cell from its erased state toward its programmed state. Available at a relatively fine granularity.
• Erase — return cells to the erased state, so they can be programmed again. Available only for a large block of cells at once.

Which means a flash write is not one operation. A location can only be programmed if it is already in the erased state, so writing new data over old data requires erasing first — and erasing affects far more than the location being written.

That single constraint is the origin of most of the complexity in every flash-based system:

Writing a small amount of data may require moving a large amount. If a block holds data that is still needed alongside the location being rewritten, that data must be copied elsewhere before the block can be erased. The system does more work than the request implied — and this is where terms like write amplification come from.

Somebody has to decide where data actually goes. Because a location cannot simply be overwritten in place, a write is usually directed to an already-erased location somewhere else, and a map is updated so the original address still finds it. That indirection layer — the flash translation layer in an SSD — exists entirely because of this consequence.

Erased space has to be manufactured in advance. A system that runs out of erased blocks must erase some before it can accept a write, which means a write can occasionally trigger a large amount of background work. That is what garbage collection is for, and it is why storage write latency has a long tail rather than a fixed cost.

None of that machinery is optional cleverness. It is all forced by the fact that the erase granularity is not the write granularity.

3. Consequence Two — Addressability Is Not Word-Sized

Chapter 1.1 drew the boundary at the bottom of the hierarchy around exactly this point, and now it can be stated mechanically rather than asserted.

Main memory must be reachable by ordinary loads and stores. A processor issues an instruction naming an address, and the value at that address comes back — a word at a time, any address, in any order, with no preparation.

Flash is organised around groups of cells that are read, programmed and erased together: cells share the lines that select and sense them, and the program and erase mechanisms act on whole groups. The consequence is that the natural unit of access is much larger than a word, and the units for reading, programming and erasing are not even the same size as each other.

So the interface a flash device presents is not a memory interface. It is a block-oriented interface: transfer this chunk, program that chunk, erase that larger chunk. Building a load/store memory on top of that is not a matter of adapting an interface — it would mean, for every single word a processor stores, reading a large unit, modifying a word of it, finding or preparing an erased destination, programming the whole unit, and updating a map. For a tier that must serve every cache miss in the machine, that is not a performance problem to be optimised. It is a category error.

And this is why flash sits outside the processor's memory path rather than at the end of it. Data is brought into main memory to be worked on, and written back when it must be preserved. That division is not a historical accident or a software convention — it is what the two technologies' access models force.

4. Consequence Three — The Storage Wears Out

This one has no counterpart anywhere above it in the hierarchy, and it changes what "correct" means for a storage system.

Every program and erase operation pushes charge through the insulator that isolates the storage node. That insulator degrades slightly each time. After enough program/erase cycles, a cell no longer holds or distinguishes its states reliably. Flash has a finite endurance, measured in program/erase cycles, and the specific figure is a property of the particular device and technology — this chapter quotes none, because the numbers vary by orders of magnitude across device types and generations, and a plausible-looking figure would be worse than no figure.

The directional facts are solid and they are what a systems engineer needs:

Endurance is consumed by writing, not by reading. Read-heavy workloads do not wear the storage out in the same way. This is why endurance appears in specifications as a write-volume rating rather than a lifetime in hours.

Wear must be spread deliberately. If a system repeatedly rewrote one location, that location's cells would exhaust their endurance while the rest of the device remained nearly new. So flash-based systems distribute writes across the physical storage — wear levelling — which requires the same indirection layer consequence one already forced. Two independent reasons for the same mechanism.

Retention and error rates are not constant over life. A worn cell holds its charge less well than a fresh one, so the error rate a controller must correct rises with use. This is why error correction in flash systems is substantial rather than token, and why it is designed for the device's end of life rather than its beginning.

And the failure mode is gradual, not sudden. Storage that degrades statistically, that must be corrected increasingly hard as it ages, and that must be monitored to predict end of life, is a fundamentally different engineering object from a DRAM device that either works or does not.

5. Where Flash Sits, and Why the Boundary Is There

Assemble the three consequences into the hierarchy picture.

The figure makes one claim, and it is the chapter's structural conclusion: the processor's load/store path terminates at DRAM. Flash attaches to main memory, not to the processor's memory path, and every byte of it that a program touches is first moved into main memory. The arrow between flash and main memory is a block transfer, usually performed by a device engine rather than by the processor issuing loads.

Three things this buys, which is why the tier is worth having despite everything in §2 to §4:

Persistence at capacity. The tier can be very large and it survives power loss, which is what a filesystem, an installed program and a saved dataset require.

Cost per bit below DRAM's. Density comes both from the cell and, in modern devices, from storing more than one bit's worth of distinguishable states per cell and from stacking structures vertically — mechanisms that trade endurance and error rate for capacity. The trade-off direction is the point; Chapter 1.6 analyses what cost per bit is actually made of.

A different failure model, deliberately. Losing power is a normal event for this tier, not a catastrophe. That is the whole reason it exists.

6. RTL — The Erase-Before-Program Sequence

Consequence one is a sequencing rule, which makes it the part of this chapter that RTL genuinely illuminates. A controller that must not program a location unless its block has been erased is a small state machine with an interesting invariant, and writing it makes the constraint concrete in a way prose does not.

What this is and is not. It is a behavioural controller-side model of the erase-before-program obligation: it tracks, per block, whether that block is currently in the erased state, and it refuses or sequences a program accordingly. It is not a flash device model, not a flash translation layer, not a garbage collector, not a wear-levelling algorithm, and it has no error correction. It contains no real device timings — the durations are parameters chosen to be visible in a waveform. A real flash controller is a large piece of engineering and this is the one rule from it that belongs in a memory-hierarchy chapter.

What it does. It accepts read, program and erase requests for a (block, page) address. Reads complete after READ_CYCLES. Erases clear a whole block after ERASE_CYCLES and mark it erased. Programs are accepted only for a block in the erased state, complete after PROGRAM_CYCLES, and mark the block no longer freshly erased. A program to a non-erased block is rejected with an error, rather than being silently sequenced — because making the caller confront the constraint is the teaching point.

How to simulate it. vlog flash_pgm_ctrl.sv tb_flash_pgm_ctrl.sv then vsim -c tb_flash_pgm_ctrl -do "run -all"; with VCS vcs -sverilog flash_pgm_ctrl.sv tb_flash_pgm_ctrl.sv && ./simv; with Xcelium xrun -sv flash_pgm_ctrl.sv tb_flash_pgm_ctrl.sv.

// BEHAVIOURAL CONTROLLER-SIDE MODEL of the erase-before-program rule.
// NOT a flash device, NOT an FTL, NOT wear levelling, NO error correction.
// The cycle counts are representative parameters chosen to be legible in a
// waveform; no real device timing is implied or should be inferred.
module flash_pgm_ctrl #(
  parameter int BLOCKS         = 4,
  parameter int PAGES_PER_BLK  = 4,
  // Representative durations. The ORDER of magnitude between them -- read
  // quickest, program slower, erase slowest -- is the real asymmetry (§1);
  // the specific values are not device figures.
  parameter int READ_CYCLES    = 1,
  parameter int PROGRAM_CYCLES = 3,
  parameter int ERASE_CYCLES   = 6,
  // DERIVED.
  parameter int BLK_W = (BLOCKS <= 1) ? 1 : $clog2(BLOCKS),
  parameter int PG_W  = (PAGES_PER_BLK <= 1) ? 1 : $clog2(PAGES_PER_BLK)
) (
  input  logic             clk,
  input  logic             rst_n,

  // Request interface. `req` is held until `done` or `err`.
  input  logic             req,
  input  logic [1:0]       req_op,      // see op_e below
  input  logic [BLK_W-1:0] req_blk,
  input  logic [PG_W-1:0]  req_pg,

  output logic             done,
  output logic             err,         // program attempted on a dirty block
  output logic             busy
);

  typedef enum logic [1:0] {
    OP_READ    = 2'd0,
    OP_PROGRAM = 2'd1,
    OP_ERASE   = 2'd2
  } op_e;

  typedef enum logic [2:0] {
    S_IDLE,
    S_READ,
    S_PROGRAM,
    S_ERASE,
    S_ERROR
  } state_e;

  state_e state, state_n;

  // THE INVARIANT THIS MODULE EXISTS FOR: one bit per block saying whether
  // that block is currently in the erased state and therefore programmable.
  // Erase sets it; a program clears it. Nothing else may touch it.
  logic [BLOCKS-1:0] blk_erased;

  // A program is legal only against an erased block (§2).
  logic pgm_legal;
  assign pgm_legal = blk_erased[req_blk];

  // Wait counter, sized from the longest operation so a parameter change
  // cannot silently truncate it.
  localparam int WAIT_MAX = (ERASE_CYCLES > PROGRAM_CYCLES)
                            ? ((ERASE_CYCLES > READ_CYCLES) ? ERASE_CYCLES
                                                            : READ_CYCLES)
                            : ((PROGRAM_CYCLES > READ_CYCLES) ? PROGRAM_CYCLES
                                                              : READ_CYCLES);
  localparam int WAIT_W = (WAIT_MAX <= 1) ? 1 : $clog2(WAIT_MAX + 1);
  logic [WAIT_W-1:0] wait_cnt;

  always_comb begin
    state_n = state;
    unique case (state)
      S_IDLE: begin
        if (req) begin
          unique case (op_e'(req_op))
            OP_READ:    state_n = S_READ;
            OP_ERASE:   state_n = S_ERASE;
            // The whole point: a program is gated on the erased state.
            OP_PROGRAM: state_n = pgm_legal ? S_PROGRAM : S_ERROR;
            default:    state_n = S_ERROR;
          endcase
        end
      end
      S_READ:    if (wait_cnt == 1) state_n = S_IDLE;
      S_PROGRAM: if (wait_cnt == 1) state_n = S_IDLE;
      S_ERASE:   if (wait_cnt == 1) state_n = S_IDLE;
      S_ERROR:   state_n = S_IDLE;   // one-cycle error report
      default:   state_n = S_IDLE;
    endcase
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state      <= S_IDLE;
      wait_cnt   <= '0;
      done       <= 1'b0;
      err        <= 1'b0;
      // Reset here is deliberate and NARROW: blk_erased is READ before it is
      // written (pgm_legal consults it on the very first request), so it must
      // start defined. Its reset value says "nothing is known to be erased",
      // which is the safe assumption -- a block must be erased before it can
      // be programmed, and after reset no erase has happened.
      blk_erased <= '0;
    end else begin
      state <= state_n;
      done  <= 1'b0;
      err   <= 1'b0;

      // Load on entry to a counted state, otherwise count down.
      if (state_n != state) begin
        unique case (state_n)
          S_READ:    wait_cnt <= WAIT_W'(READ_CYCLES);
          S_PROGRAM: wait_cnt <= WAIT_W'(PROGRAM_CYCLES);
          S_ERASE:   wait_cnt <= WAIT_W'(ERASE_CYCLES);
          default:   wait_cnt <= '0;
        endcase
      end else if (wait_cnt != '0) begin
        wait_cnt <= wait_cnt - 1'b1;
      end

      // Completion, and the block-state updates that enforce §2's rule.
      if (wait_cnt == 1) begin
        unique case (state)
          S_READ:    done <= 1'b1;
          S_PROGRAM: begin
            done                 <= 1'b1;
            // Programmed: this block is no longer freshly erased, so a
            // further program to it must be preceded by another erase.
            blk_erased[req_blk]  <= 1'b0;
          end
          S_ERASE: begin
            done                 <= 1'b1;
            // The erase granularity IS the block -- this single assignment
            // is consequence one in one line.
            blk_erased[req_blk]  <= 1'b1;
          end
          default: ;
        endcase
      end

      if (state == S_ERROR) begin
        err <= 1'b1;
      end
    end
  end

  assign busy = (state != S_IDLE);

endmodule

Interface and cycle behaviour. The caller holds req with an operation and an address until done or err. A read completes quickly. An erase takes the longest and leaves the block programmable. A program is accepted only if its block is currently erased, takes an intermediate time, and afterwards the block is no longer programmable without another erase. A program to a block that is not erased returns err in a single cycle — the caller is told, immediately, that it asked for something the technology does not allow.

Design decisions worth naming. The per-block blk_erased vector is the entire model: one bit that encodes "this block may be programmed". Note the granularity mismatch made explicit — the request carries a page address, but blk_erased is indexed by block, so programming one page clears the programmability of the whole block. That asymmetry between the address you supply and the state you affect is consequence one, and having it visible as a width mismatch in the code is more instructive than a paragraph about it. The rejection path is a state rather than a combinational flag so the error is a reported event with a defined cycle, and blk_erased is reset while the data path is not, for the reason the comment gives.

Expected behaviour. Erase block 1, then program block 1 page 0 — both complete with done. Then program block 1 page 1 without erasing again — err rises in one cycle and no program occurs. Erase block 1 again and the program succeeds. A testbench printing cycle counts will show erase as the longest operation and read as the shortest.

Expected waveform. §7 is exactly that sequence, including the rejected program.

Synthesis implication. The state machine, the counter and the blk_erased vector are synthesizable. blk_erased is BLOCKS bits wide, which is fine at four and would be a design decision at realistic block counts — a real controller holds block state in a memory, not in flops, which is itself an instance of Chapter 1.3's tier argument appearing inside a controller.

Limitations. No data storage at all — this models the sequencing rule, not the array. No partial-page programming, no program-order constraints within a block, no erase suspend, no bad-block management, no error correction, no wear counting, and no translation layer. A real controller has all of those, and each exists for a reason traceable to §2 to §4.

Debugging observations. If a program unexpectedly errors, check whether an earlier program to the same block cleared blk_erased — the second program to a block is the classic surprise. If err never asserts in a test that expects it, check that the stimulus is not erasing before every program, which makes the illegal case unreachable. If the model appears to hang, check that every duration parameter is at least 1, as the counted-wait control flow requires.

7. The Asymmetry, in Cycles

Cycles 0 to 4 — the erase. The longest operation in the figure, and the longest in real devices too. Nothing else can use the controller while it runs, which is why a storage system's write latency has a tail: a write that happens to require an erase costs dramatically more than one that does not.

Cycle 5 — the block becomes programmable. blk_erased for this block goes high, and the program the caller wanted is now legal. Note the ordering: the erase had to complete first. There is no overlapping these, because the second operation's legality depends on the first one's result.

Cycles 5 to 7 — the program. Faster than the erase, slower than a read. It completes at cycle 8, and blk_erased drops in the same step — the block is immediately no longer programmable.

Cycle 9 — the rejection. A second program to that block, with no intervening erase, produces err. This is the constraint refusing to be ignored, and it is reported in a single cycle rather than hidden: the caller learns immediately that what it asked for requires an erase it has not performed.

8. Verification — What a Sequencing Rule Makes Checkable

The interesting verification here is not data integrity but rule enforcement, which is a different discipline.

The core invariant. A program never proceeds against a block that is not in the erased state. This is the property the whole module exists to guarantee, and it should be asserted rather than tested by example, because the space of orderings is large.

Block state transitions only for the right reasons. blk_erased for a block may rise only on completion of an erase of that block, and fall only on completion of a program to it. A stray update — from the wrong index, or from a partially decoded address — is exactly the bug that produces a silent violation of the core invariant later.

The granularity mismatch is respected. A program to one page must affect the whole block's programmability, and an erase must affect exactly one block and no other. The "no other" half is where an address-decode fault hides.

Error reporting is exact. err asserts for the illegal case and only for it, and the operation genuinely does not happen. An error that is reported and performed is worse than either.

Mutual exclusion of completion signals. done and err must never both assert for one request, because a caller cannot act sensibly on both.

Liveness. Every held request terminates in done or err within a bound the parameters determine.

Coverage worth asking for. Each operation on each block; a program immediately after an erase of the same block (legal) and of a different block (illegal, and the case a sloppy implementation gets wrong); two programs to one block without an intervening erase; an erase of an already-erased block; reset asserted during each of the three counted operations; and a program to a block that was erased before a reset — which must be illegal, because reset clears the knowledge.

That last one deserves emphasis: after reset the model knows nothing about the device's actual erased state, so it assumes nothing is programmable. That is the safe direction, and a design that assumed the opposite would program over data.

9. Four Assertions Worth Writing

// VERIFICATION-ONLY, inside flash_pgm_ctrl.

// P1 -- THE rule of the chapter, and the only one that must never fail: a
// program is never entered unless its target block is in the erased state.
// Written on the state transition rather than on the completion, so it
// fails at the moment the illegal decision is made.
property p_program_requires_erased_block;
  @(posedge clk) disable iff (!rst_n)
    (state != S_PROGRAM && state_n == S_PROGRAM) |-> blk_erased[req_blk];
endproperty
assert property (p_program_requires_erased_block);

// P2 -- a block becomes programmable ONLY by completing an erase of that
// block. This is the property that catches a stray or mis-decoded update to
// the block-state vector, which would otherwise let P1 pass while the real
// device was not erased.
property p_erased_bit_rises_only_on_erase;
  logic [BLK_W-1:0] b;
  @(posedge clk) disable iff (!rst_n)
    ($rose(blk_erased[req_blk]), b = req_blk)
      |-> $past(state == S_ERASE && wait_cnt == 1 && req_blk == b);
endproperty
assert property (p_erased_bit_rises_only_on_erase);

// P3 -- done and err are mutually exclusive. A caller given both has no
// defined action, and the ambiguity would propagate upward silently.
property p_done_err_exclusive;
  @(posedge clk) disable iff (!rst_n)
    !(done && err);
endproperty
assert property (p_done_err_exclusive);

// P4 -- LIVENESS with a bound derived from the parameters: a held request
// terminates one way or the other within the longest operation plus the
// handshake overhead. A bound rather than an open eventuality, so a design
// that merely finishes slowly still fails.
localparam int WORST_CASE = WAIT_MAX + 2;
property p_request_terminates;
  @(posedge clk) disable iff (!rst_n)
    ($rose(req) && !busy) |-> ##[1:WORST_CASE] (done || err);
endproperty
assert property (p_request_terminates);

What each buys. P1 is the chapter's physical constraint as a temporal contract; it is written on the transition into the program state rather than on completion, so it fires at the decision rather than after the damage. P2 is the one that makes P1 trustworthy — without it, a bug that wrongly sets a block's erased bit would let P1 pass while the actual device had not been erased, which is the failure mode that corrupts data. The pairing is the lesson: an assertion that depends on a piece of state is only as strong as the assertions on how that state is set. P3 protects the caller's decision logic. P4 bounds termination using a localparam derived from the parameters, so the bound stays correct if the durations change.

What they do not claim. Nothing here verifies that the device was actually erased — only that the controller's model of it was consistent with its own rules. Closing that gap needs a device model that rejects illegal programs independently, which is precisely why real flash controllers are verified against vendor models rather than against their own bookkeeping. Saying so is part of the lesson: a controller that only checks itself proves nothing about the medium.

10. Common Mistakes

Treating "non-volatile" as "memory". Wrong mental model: persistent storage is memory that also survives power loss. What the engineer does: plans an architecture in which the processor addresses persistent storage directly for working data, or assumes a storage tier can absorb main-memory traffic. Resulting bug: not a functional bug but a design that cannot meet its performance requirement by a wide margin, discovered after the architecture is committed. Every word store becomes a read-modify-program-erase sequence through an indirection layer. How to detect it: check whether the plan requires word-granular, unlimited, symmetric writes at main-memory rates. If it does and the tier is flash, the plan does not work. How to prevent it: judge a tier against the three requirements in §3's callout, not against its marketing category.

Assuming a write costs the same every time. Wrong mental model: storage write latency is a number. What the engineer does: budgets for average write latency in a system with a real-time or bounded-latency requirement. Resulting bug: rare, severe latency excursions when a write triggers an erase or a garbage-collection burst. The system passes every benchmark and misses deadlines in the field, unreproducibly. How to detect it: measure the distribution, not the mean — specifically the far tail. A write-latency histogram with a long right tail is the signature, and an average hides it completely. How to prevent it: budget against the tail for bounded-latency requirements, and treat background activity in the storage device as a first-class part of the model.

Ignoring endurance because the test never reaches it. Wrong mental model: the storage either works or it does not. What the engineer does: validates against fresh devices and ships, with no accounting of write volume over the product's life. Resulting bug: rising error rates and eventual failure in the field, at a time that correlates with usage rather than with age — so it looks like a reliability mystery rather than a predictable consumption of a specified resource. How to prevent it: treat write volume as a budgeted resource with a lifetime figure from the device specification, and validate with aged or artificially worn devices where the requirement justifies it.

Confusing erase granularity with write granularity. Wrong mental model: if I can program a page, I can rewrite a page. What the engineer does: sizes buffers, plans update-in-place data structures, or estimates write amplification as though the two units were the same. Resulting bug: far more physical writing than planned, consuming both performance and endurance. A data structure that updates a small field in place can cause a large multiple of that field's size to be moved. How to detect it: compare logical bytes written by the application against physical bytes written by the device, if the device reports it. A large ratio is the direct measurement of this mistake. How to prevent it: design update patterns around the erase unit — append rather than overwrite, batch updates, and accept that the indirection layer exists precisely because this mismatch is unavoidable.

11. Debugging — Write Throughput Falls Off a Cliff

Symptom. A system writing to persistent storage sustains a high rate for a while, then drops sharply to a much lower rate and stays there. No errors are reported and reads remain fast.

Hypotheses, ordered by how cheaply they can be separated.

Hypothesis 1 — the initial rate was absorbing into erased space, and that space ran out. Evidence to look for: the drop correlating with cumulative bytes written rather than with elapsed time or with any change in the workload. Discriminator: this is the classic signature, and it goes first because the mechanism is exactly §2 — while previously erased blocks are available, writes are programs; once they are gone, each write must wait for erases to manufacture more. A drop at a volume threshold rather than a time threshold is close to conclusive.

Hypothesis 2 — background reclamation is now competing with the workload. Evidence to look for: device-reported background activity, or write latency that has become bimodal rather than uniformly slower. Discriminator: hypothesis 1 and 2 are two views of the same underlying cause, so separate them by looking at the shape of the latency distribution: a uniformly lower rate suggests a steady state where every write pays reclamation, while a bimodal distribution suggests bursts of it.

Hypothesis 3 — the access pattern is causing excessive data movement. Evidence to look for: the ratio of physical bytes written by the device to logical bytes written by the application, if available. Discriminator: a large ratio indicates the workload's update pattern is forcing whole units to be rewritten for small changes (§10's fourth mistake), which makes the steady-state rate much worse than the device is capable of. The fix here is in the workload, not the device.

Hypothesis 4 — the device is thermally or power limited. Evidence to look for: correlation with temperature or with sustained activity duration rather than with written volume. Discriminator: thermal effects track time-under-load and recover after an idle period; erased-space exhaustion does not recover just because the system rested.

Hypothesis 5 — the storage is worn. Evidence to look for: device health or wear indicators, and rising corrected-error counts. Discriminator: wear develops over the device's life rather than within one run, so a drop that appears in a single session and reproduces on a fresh device is not this.

Root-cause discrimination. One experiment separates most of it: repeat the workload after an idle period, and separately after restoring the device to a known-erased state. Recovery after idling points at reclamation or thermal limits; recovery only after re-erasing points at erased-space exhaustion; no recovery at all points at the workload's own write amplification or at wear.

And note the reasoning being taught. The naive conclusion — "the storage is slow" — is refuted by the first phase of the same run, which was fast. Whenever a tier's performance changes without the tier changing, the question is what state it has entered, and for flash that state is almost always about the availability of erased space.

12. Interview Reasoning

"Why can persistent storage not simply replace DRAM as main memory?" Three independent reasons, all from this chapter. A write is not the inverse of a read: programming requires a prior erase, and the erase unit is far larger than the address unit, so updating a word can require moving a large block. The natural access granularity is far larger than a word, so the interface is block-oriented rather than load/store. And the storage has finite endurance, so unlimited rewriting — which main memory requires — is not available. A strong answer adds the structural version: a main-memory tier needs fine-grained addressing, symmetric unlimited rewriting, and tolerable latency, and flash fails all three rather than merely being slower.

"Why is a flash write's latency variable when a DRAM write's is comparatively predictable?" Because a flash write may or may not require work beyond the write itself. If an erased destination is available, the operation is a program. If not, the system must erase — and possibly first relocate still-needed data out of the block it intends to erase — before the write can proceed. That conditional work is why storage write latency has a long tail, and why budgeting against the average is a mistake in any bounded-latency system.

"What does endurance mean, and what consumes it?" Endurance is the number of program/erase cycles a cell can take before it no longer distinguishes its states reliably, because each of those operations pushes charge through the insulator that isolates the storage node and degrades it slightly. Writing consumes it; reading essentially does not. Two engineering consequences follow: writes must be distributed across the device rather than concentrated, and error correction must be designed for the device's end of life rather than its start.

"Why do flash-based systems need an indirection layer at all?" Two independent forces demand the same mechanism. Because a location cannot be overwritten in place, a write must be directed to an already-erased location elsewhere and a map updated so the original address still resolves. And because endurance is finite, writes must be spread across the physical storage rather than concentrated. Both require a translation from logical address to physical location, which is why every flash-based storage device contains one.

"Reading flash is harmless and reading DRAM is destructive. Does that make flash the better memory?" No, and the question is a good test of whether the hierarchy argument has landed. Read behaviour is one property among several, and flash loses on the ones main memory actually requires — write symmetry, write granularity, unlimited rewriting and latency. DRAM's destructive read is a cost it pays inside an access that remains word-granular, unlimited and fast enough; flash's harmless read sits on top of a write model a processor cannot use directly. Comparing single properties across tiers is precisely the reasoning error Chapter 1.6 exists to correct.

13. Engineering Check

A design must keep a table that is updated frequently — many small updates per second — and that must survive power loss.

1. Can the table live in flash and be updated in place? No. Each small update would require the block containing it to be erased before that location could be programmed again, which means relocating any still-needed data in the block, erasing, and programming — for every update. The write amplification and the endurance consumption would both be severe, and the latency of each update would be dominated by work unrelated to its size.

2. What structure does the constraint push you toward? Appending rather than overwriting: write each update as a new record into already-erased space, and reconstruct the table's current state by replaying or indexing those records. This converts many small in-place updates into sequential programming, which is the pattern flash serves well. That it is also what real filesystems and databases on flash do is not a coincidence — the technology's constraint shapes the data structure.

3. Where does the frequently updated copy actually live? In DRAM, with flash holding the durable record. This is the staging relationship in Figure 1, arrived at from the requirements rather than assumed: the tier that tolerates unlimited fine-grained rewriting holds the working copy, and the tier that survives power loss holds the persistent one.

4. What has to be true for that to be safe across a power loss? The persistent record must be sufficient to reconstruct a consistent state, which means the ordering and completion of writes to flash matter, not just their content. A design that assumes a write is durable the moment it is issued will lose data on power loss — which is why storage interfaces have explicit durability operations at all.

5. What resource is being consumed that no tier above flash consumes, and how would you budget it? Endurance. Budget it as a write-volume figure over the product's intended life: estimate physical bytes written per unit time — including amplification, not just application bytes — multiply by the lifetime, and compare against the device's rating. If the margin is thin, the answer is usually to reduce physical writing through batching and append-style structures rather than to choose a bigger device.

6. Which measurement would tell you your structure is working? The ratio of physical bytes the device writes to logical bytes the application writes. A ratio near one means the update pattern matches the technology; a large ratio means small updates are still forcing large rewrites, and both performance and endurance are being spent on data movement rather than on data.

14. Summary

Flash is persistent because it stores charge on a node electrically isolated from its surroundings, and that charge shifts the transistor's threshold so a read is a measurement of conduction rather than a sensing of stored voltage. A read is therefore harmless and needs no power to have preserved the data.

Everything difficult about the tier comes from the other direction. Changing the stored charge means moving it through an insulator, so writing is a physical alteration rather than the inverse of a read — and three consequences follow.

A write is not the inverse of a read. Programming is relatively fine-grained; erasing is only available for a large block. A location can be programmed only from the erased state, so rewriting requires erasing, and erasing affects far more than the location being written. This single constraint is why flash systems have indirection layers, write amplification, garbage collection and long write-latency tails.

Addressability is not word-sized. The units of reading, programming and erasing are large and not equal to each other, so the device presents a block-oriented interface rather than a load/store one. That is why the processor's memory path ends at DRAM, and why flash data is staged through main memory to be used.

The storage wears out. Program and erase operations degrade the isolating insulator, so endurance is finite and consumed by writing. Wear must be deliberately spread, and error correction must be designed for end of life.

Which answers the chapter's question. Persistence is not a free upgrade to a memory; it is a different storage mechanism whose write model, granularity and lifetime are incompatible with what a main-memory tier must provide. Flash is below main memory because of how it writes, not because of how fast it reads — and that is why the tier above it still has to exist.

15. What Comes Next

Four technologies now have their mechanisms, and each one was described as making a trade. What has not been done is to look at the trade itself — to ask what exactly is being exchanged when a design moves from one tier to another, and why no single technology can win every axis at once.

Chapter 1.6 does that. It takes the axes apart: what cost per bit is actually composed of, why cell size dominates it, how the shared periphery of Chapter 1.3 changes the arithmetic, and why density and access speed pull in opposite directions. It is the chapter that turns four separate technology descriptions into one trade surface — and it is the groundwork for Chapter 1.7, which uses that surface to explain why DRAM, specifically, holds main memory.

Return to DRAM for the tier above, or The Memory Hierarchy for the map. The full path is on the DDR tutorials index.