CXL.mem Write Flows

Chapter 9.4 walked the read path: one question, one answer. The write path reuses that machinery and adds the thing 9.3 flagged as harder — a read is finished when it has a value, and a write is not finished until someone else can see it.

1. The Engineering Problem — A Write Has Two Endings

Ask when a read is done and there is one answer: when the value arrives. Ask when a write is done and there are two, and they are separated by real time:

Reporting the first as though it were the second is not an optimisation. It is the bug that breaks every producer/consumer handshake in software, and it is invisible to any test that writes and then reads back from the same agent.

Four more things follow from the gap between those two endings, and each is a section below: writes to one address must become visible in order, a sub-line write must read before it writes, an accepted write must survive until it is visible, and a read arriving in the gap must see the pending value.

2. The One-Sentence Model

A write is not done until someone else can see it. Acceptance is a flow-control event and visibility is a correctness event; between them the write lives in a buffer that must never lose it, must not let a later write to the same address overtake it, must not destroy bytes it never named, and must be searched by any read of that address.

Call it the second ending. Everything in this chapter lives in the gap between the two.

3. What This Chapter Owns

4. The Write, End to End

Architectural. The arrows are obligations, not named messages.

The distance between arrow two and arrow six is the whole chapter. Everything that can go wrong on a write path goes wrong in that gap, and a design that closes it by reporting completion at acceptance has not removed the gap — it has hidden it from the only party that could have accounted for it.

5. Teaching-model boundary

6. RTL 1 — Two Completions, and Only One Counts

  assign cmp_valid = REPORT_ON_ACCEPT ? wr_accept : wr_visible;
  ...
      // Reporting completion on a cycle that is not the visibility cycle.
      if (cmp_valid && !wr_visible) early_completion_err <= 1'b1;

=== EXP1: a write completes when it is VISIBLE, not when accepted ===
  at acceptance : correct cmp_valid=0 | report-on-accept cmp_valid=1
  acceptance does not complete the write             : ok
  the broken variant reported it immediately         : ok
  in flight : accepted=1 visible=0 inflight=1
  at visibility : cmp_valid=1 id=2
  visibility completes it, with the right id        : ok
  and the write left the in-flight set              : ok
  report-on-accept variant flagged early completion : ok

Acceptance is a flow-control event. It tells the writer the device has taken ownership, which is what a credit-returning scheme needs. It says nothing about observability, and using it as a completion converts a correctness guarantee into a throughput optimisation the writer did not ask for.

A checker that had to allow the legal case

The visible-without-acceptance check initially read:

if (!acc_mask_q[vis_id]) visible_without_accept_err <= 1'b1;

The baseline failed on correct stimulus: a write accepted and made visible in the same cycle is entirely legal — a device with a fast path can do exactly that — and the acceptance mask is still being set by that edge. The corrected condition admits it:

        // Becoming visible without ever being accepted is impossible. A write
        // accepted and made visible in the SAME cycle is legal, so the mask --
        // which this edge is still setting -- cannot be the whole condition.
        if (!acc_mask_q[vis_id] && !(wr_accept && (wr_id == vis_id)))
          visible_without_accept_err <= 1'b1;

Same state-versus-transition shape as 9.3's premature-visibility checker, in the opposite direction: there the check was too narrow to fire, here it was too broad and fired on correct behaviour.

7. Waveform — Accepted Early, Visible Late, Out of Order

Two things to read.

Between cycles 3 and 5, accepted is 2 and visible_cnt is 0. Two writes the device owns, neither observable, no completion sent. That interval is the accepted-but-invisible window 9.3 measured, drawn.

The completions come back in the opposite order to acceptance — id 3 then id 2. That is legal because they are different addresses; §8 is about the case where it is not.

8. RTL 2 — Same-Address Writes Become Visible in Order

For reads, ordering decides which value is returned. For writes it decides which value survives, and getting it wrong is permanent.

  // A write may become visible only if every earlier write to this address
  // already has. Sequence numbers make that checkable.
  assign in_order = (vis_seq == expect_seq_q);
  assign allowed  = vis_valid && (NO_WAW_ORDER || in_order);

=== EXP2: same-address writes become visible in order ===
  the first write in sequence proceeds              : ok
  seq 2 before seq 1 : correct allowed=0 | no-WAW allowed=1
  a later write cannot overtake an earlier one      : ok
  the no-WAW variant let it overtake                 : ok
  the correct next write proceeds                   : ok
  no-WAW variant flagged a write-after-write violation: ok

The asymmetry with reads is worth dwelling on. A read served out of order returns a stale value once — bad, and recoverable by reading again. A write made visible out of order means an older value lands after a newer one and the newer value is gone. No subsequent read recovers it.

That is why the ordering here is expressed as a sequence check rather than a simple busy flag: the device must know not just that an earlier write exists but that it has already become visible.

9. RTL 3 — A Partial Write Must Read Before It Writes

Media writes whole lines. A sub-line write is therefore a read-modify-write inside the device, and skipping the read is the classic corruption.

  // A full-line write needs no merge read; a partial one does.
  assign needs_merge_read = wr_valid && (byte_en != 8'hFF);
  assign can_write = wr_valid && (NO_MERGE_READ || !needs_merge_read || old_valid);
  assign merged_line = (wr_data & mask) | (old_line & ~mask);

=== EXP3: a partial write must not clobber unnamed bytes ===
  full-line write : needs_merge_read=0 can_write=1
  a full-line write needs no merge read             : ok
  partial, no old data : needs_merge_read=1 correct can_write=0 | no-merge can_write=1
  it waits for the old line                         : ok
  the no-merge variant wrote anyway                 : ok
  old data available : can_write=1 merged=0x11111111AAAAAAAA
  named bytes written, unnamed bytes preserved     : ok
  no-merge variant flagged a clobber                : ok

Read the merged value: 0x11111111AAAAAAAA. The low four bytes came from the write (AAAAAAAA), the high four from the old line (11111111). The bytes the request never named survived, which is the entire requirement.

The broken variant writes wr_data straight through, so the unnamed bytes become whatever the merge buffer held — usually zeros, sometimes another request's data. It corrupts memory the writer never touched.

This is also a performance fact worth stating plainly. A partial write costs a media read and a media write, so on a write path dominated by media it is roughly twice a full-line write. Software that writes whole lines is not being fastidious; it is halving its media traffic.

10. RTL 4 — Stall, Never Lose

=== EXP4: an accepted write is never lost ===
  12 pushes into depth 8 : correct level=8 pushed=8 stalled=4
  four were stalled, none lost                      : ok
  peak buffer level was exactly 8                   : ok
  the drop-on-full variant lost accepted writes     : ok
  and the buffer drained completely                 : ok
  everything pushed was drained exactly once        : ok

Same contract as 9.1's inbound path, with a harsher consequence. A dropped read hangs a requester — bad, and detectable, because someone is waiting. A dropped write produces no waiter at all: the data is simply gone, and the next read of that address returns the old value with no error anywhere.

n_drain_q == n_push_q is the conservation law that makes it checkable, and it is the one property worth asserting continuously rather than at the end of a test.

11. RTL 5 — A Read in the Gap Must See the Newest Write

    if (NEWEST_WINS) begin
      // Scan so the LAST match wins -- the newest write to this address.
      for (k = 0; k < DEPTH; k = k + 1)
        if (vld_q[k] && (addr_m[k] == rd_addr)) begin sel = k[3:0]; found = 1'b1; end
    end

=== EXP5: a read must see the NEWEST buffered write ===
  two writes to one address : correct data=0x2222 | oldest-wins data=0x1111
  the NEWEST buffered write was forwarded           : ok
  the oldest-wins variant returned the stale value  : ok
  an unbuffered address falls through to media      : ok

9.3 required forwarding; this is where it lives. The write buffer is precisely the reason a write can be accepted and not yet in the media, so the buffer is what a read must search.

The subtle part is which match wins. With several writes to one address buffered, the newest is the correct answer — and the difference between a scan that keeps the last match and one that stops at the first is one loop direction. Both are one line, both look right, and one silently returns stale data.

12. RTL 6 — A Failed Write Must Reach the Writer

  assign cmp_is_error = drain && media_fail && !SILENT_FAIL;
  ...
      // The media could not store it and the writer was told it succeeded.
      if (media_fail && !cmp_is_error) silent_loss_err <= 1'b1;

=== EXP6: a failed write must be reported ===
  media failed : correct is_error=1 | silent variant is_error=0
  the failed write was reported as an error         : ok
  the silent variant reported success               : ok
  silent-fail variant flagged a silent data loss    : ok

A failed write is harder to notice than a failed read, and the asymmetry is worth stating. A failed read has nothing to return, so a device that does not report it must fabricate data — at least there is a value to be suspicious of. A failed write has nothing to return either way: success and failure look identical on the wire, so a device that omits the error is indistinguishable from one that worked.

The data is gone and every party believes it was stored. This is the single most dangerous failure in the write path.

13. RTL 7 — Write Accounting

=== EXP7: write accounting, against an oracle ===
  accepted=36 buffered=36 visible=32 failed=2 max_uncommitted=4
  partial=9 merge-reads=9 forwards=6
  accept/visible matched an independent oracle      : ok
  nothing became visible that was not accepted      : ok
  peak uncommitted was exactly 4                    : ok
  every partial write took a merge read             : ok

partial == merge-reads is the invariant that catches §9's defect in telemetry rather than in data: every partial write must have taken a merge read, so a mismatch means some partial write went straight to media.

max_uncommitted is the write path's most important number. It is the size of the accepted-but-invisible window — how many writes the device owes visibility for at the worst moment — and it bounds both how wrong a report-on-accept design would be and how long a fence must wait.

Read it as a list of losses rather than a datapath. Every stage between acceptance and completion can destroy data, and each does so without a waiter: the order gate loses the newer value, the merge read loses unnamed bytes, the buffer loses the write outright, the forward path returns a value already superseded, and the media loses it while reporting success. That is why §18 lists four alarms — the write path earns them.

14. Quantitative Reasoning — What the Write Path Costs

Illustrative, from the measured runs.

Partial writes cost double

A full-line write is one media operation. A partial write is a media read plus a media write:

At 9.1's 100 ns media latency and 4-deep media concurrency, moving from all-partial to all-full-line halves the media occupancy of the write stream. That is a software alignment decision with a hardware-scale payoff.

The accepted-but-invisible window

  accepted=36 visible=32 max_uncommitted=4

Four writes owed visibility at the peak. Two consequences:

A fence issued at that moment waits for all four — at 100 ns media latency and 4-deep concurrency, roughly one media round trip.

A report-on-accept design would be wrong by up to four writes, which is exactly how many producer/consumer handshakes could observe a flag before its data.

Buffer sizing

  12 pushes into depth 8 : pushed=8 stalled=4

The buffer must cover the acceptance rate times the drain latency — the same bandwidth-delay reasoning as every other resource in this module. Undersizing it converts write acceptance into a stall the writer sees, which on a store-heavy workload is a core stall.

15. Assertions

Icarus Verilog 13.0 does not support concurrent SVA here, so every property is synthesisable checker logic verified in simulation.

Safety

Liveness

The first is the one that matters most: an accepted write that never becomes visible is silent data loss with no waiter — nothing anywhere is blocked on it, so nothing reports it. That is why buffer drain conservation is a continuously-checked safety property and not merely an end-of-test tally.

Performance goals

16. Mutation Testing

Twenty-one mutations. Twenty-one killed.

The first run scored 18 of 21, and the escapes were the shapes this batch has established: visibility-without-acceptance and the conservation law both needed instances reserved for illegal stimulus, and the in-flight counter needed accept and visible in the same cycle — the ninth appearance of that defect family in this track.

Two repairs came from the baseline rather than from mutation testing, and both were mine.

The visible-without-acceptance checker fired on correct behaviour. A write accepted and made visible in the same cycle is legal, and the acceptance mask is still being set by that edge. The checker was too broad, which is the mirror image of 9.3's premature-visibility checker being too narrow.

My testbench accepted the same id four times and made it visible four times. An id may be accepted once and become visible once; the stimulus was illegal and the design was right to complain. Distinct ids fixed it. Across this batch the baseline found four defects the mutation suite could not, because mutation testing verifies the checkers and the baseline verifies the design.

17. Verification Plan

The coverage cross is write size × buffer state × same-address relationship: full / partial, crossed with room / full / draining, crossed with unique / repeated address. The repeated-address-with-buffer-full point is where ordering, forwarding and back-pressure interact, and it must be directed.

18. Silicon Observability

Four alarms — more than any other chapter in this module, and that is the honest signature of the write path: it has more ways to lose data silently than the read path has to return it wrongly. A lost write, a silently failed write, an out-of-order visibility and a clobbered byte all destroy data with no waiter and no error.

The most useful tuning ratio is partial writes over total writes, because it converts directly into media load, and it is usually fixable in software.

19. Debug Lab

assign cmp_valid = wr_accept;      // completion at acceptance

  at acceptance : correct cmp_valid=0 | report-on-accept cmp_valid=1
  report-on-accept variant flagged early completion : ok

assign cmp_valid = wr_visible;
if (cmp_valid && !wr_visible) early_completion_err <= 1'b1;

assign allowed = vis_valid;      // any buffered write may drain

  seq 2 before seq 1 : correct allowed=0 | no-WAW allowed=1
  no-WAW variant flagged a write-after-write violation: ok

assign in_order = (vis_seq == expect_seq_q);
assign allowed  = vis_valid && in_order;
if (allowed && !in_order) waw_violation_err <= 1'b1;

media_line <= wr_data;      // partial write straight to media

  old data available : can_write=1 merged=0x11111111AAAAAAAA
  named bytes written, unnamed bytes preserved     : ok

assign needs_merge_read = wr_valid && (byte_en != 8'hFF);
assign can_write   = wr_valid && (!needs_merge_read || old_valid);
assign merged_line = (wr_data & mask) | (old_line & ~mask);

assign ready = 1'b1;                    // always accept
if (push && !full) store_write();       // ...but only store when there is room

  12 pushes into depth 8 : pushed=8 stalled=4
  the drop-on-full variant lost accepted writes     : ok

assign ready   = !full;
assign do_push = push && ready;
// and assert continuously
if (n_drain_q != n_push_q) /* investigate */ ;

for (k = DEPTH-1; k >= 0; k = k - 1)
  if (vld[k] && addr[k] == rd_addr) begin sel = k; found = 1; end   // first match

  two writes to one address : correct data=0x2222 | oldest-wins data=0x1111
  the oldest-wins variant returned the stale value  : ok

for (k = 0; k < DEPTH; k = k + 1)
  if (vld[k] && addr[k] == rd_addr) begin sel = k; found = 1; end   // last match wins

assign cmp_is_error = 1'b0;      // completion never carries a failure

  media failed : correct is_error=1 | silent variant is_error=0
  silent-fail variant flagged a silent data loss    : ok

assign cmp_is_error = drain && media_fail;
if (media_fail && !cmp_is_error) silent_loss_err <= 1'b1;

if (wr_accept)  inflight_q <= inflight_q + 1;
if (wr_visible) inflight_q <= inflight_q - 1;

  3 cycles of accept+visible together : inflight=4
  the case-based counter held steady                : ok

case ({wr_accept, wr_visible})
  2'b10: inflight_q <= inflight_q + 4'd1;
  2'b01: inflight_q <= inflight_q - 4'd1;
  default: ;                                  // both or neither: hold
endcase

if (!acc_mask_q[vis_id]) visible_without_accept_err <= 1'b1;

  3 cycles of accept+visible together : inflight=4
  the case-based counter held steady                : ok

if (!acc_mask_q[vis_id] && !(wr_accept && (wr_id == vis_id)))
  visible_without_accept_err <= 1'b1;

20. Design Review

1. Which event does a write completion report — acceptance or visibility?
2. Can two writes to one address become visible out of order?
3. Does a partial write read the line first, and what asserts it?
4. Can an accepted write be dropped, and what conservation law would catch it?
5. Does a read search the write buffer, and does the newest match win?
6. Does a media write failure reach the writer?
7. Is the in-flight count correct when an acceptance and a visibility coincide?
8. What is max_uncommitted under real traffic, and what waits on it?
9. What fraction of writes are partial, and what does that cost in media?
10. What happens on reset with accepted-but-invisible writes?

21. How This Appears in Real Engineering

Architecture. The partial-write fraction is a software-visible lever with a hardware-scale payoff — up to 2× media load on the write stream.

RTL. The write buffer is a small structure carrying four of this chapter's alarms: lost writes, out-of-order visibility, clobbered bytes and stale forwards.

DV. Two writes to one address, a partial write over a non-zero background, and a same-cycle accept-and-visible are all directed. None arises naturally.

Post-silicon. Four correctness alarms is more than any other chapter here, and each catches a failure whose only symptom is wrong data much later.

Software. Whole-line writes halve media traffic and avoid the merge-read path entirely, which makes alignment a performance decision rather than a stylistic one.

22. Common Misconceptions

23. Interview Reasoning

24. Exercises

1. Calculation. A workload issues 10,000 writes, 60% of them sub-line. Compute the media operations with and without a merge read, and the media time at 100 ns per operation. Then state the saving from aligning the workload to full lines.

2. Analysis. A device reports accepted=10^6, visible=10^6, max_uncommitted=0, and a producer/consumer handshake that fails intermittently. Explain why those three counters are suspicious together and name the defect.

3. RTL task. Extend write_order to track sequences per address rather than globally. State the storage cost and the new failure mode a shared sequence counter would have had on a multi-address workload.

4. Assertion task. Write the property that catches a partial write proceeding without its merge read, and explain why it must be expressed over the byte enables rather than over the write's size field.

5. Debug task. Data adjacent to a written field is being zeroed. Give your investigation order and the single directed test that distinguishes a device merge bug from a software bounds bug.

6. Design review. A colleague proposes reporting write completion at acceptance to improve throughput, noting the device never loses an accepted write. Give the strongest version of that argument, then name exactly what it breaks and what would have to be true system-wide for it to be safe.

25. Summary

A write is not done until someone else can see it.

• A write has two endings: accepted (flow control) and visible (correctness). Reporting the first as the second breaks every producer/consumer handshake.
• Same-address writes must become visible in acceptance order. An out-of-order write destroys the newer value permanently, unlike an out-of-order read.
• A partial write must read before it writes, or it corrupts bytes the writer never named — and it costs two media operations, up to 2× media load.
• Stall, never lose. A dropped write has no waiter, so nothing reports it; push-equals-drain is the law that catches it.
• A read in the gap must see the newest buffered write. The scan direction that returns the oldest is one character away and silently stale.
• A failed write must reach the writer, because success and failure are otherwise indistinguishable on the wire.
• Measured: peak accepted-but-invisible was exactly 4 — the fence cost and the bound on how wrong a report-on-accept design would be.
• Four correctness alarms — lost write, silent failure, write-after-write violation, clobber — more than any other chapter in this module.
• The two-assignment counter defect appeared for the ninth time, and again turned a counter bug into a correctness violation.
• Verification lesson: the baseline found two defects mutation testing could not — a checker that fired on legal same-cycle completion, and a testbench driving illegal stimulus the design was right to reject. Mutation testing verifies the checkers; the baseline verifies the design.

That completes the read and write flows. Chapter 9.6 — CXL.mem Performance Implications turns these obligations into their latency and throughput cost, and it is not part of this batch.