Memory Controllers in CHI

Module 5 · Chapter 5.8 · CHI System Components

Project thread — 5.5 gave SN-F abstractly and 5.7 gave the cache side; this chapter grounds SN-F in a real DRAM controller. 5.9 closes the module with the interconnect fabric.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Map a DRAM controller onto the SN-F role — front end, request queue, scheduler, DRAM.
• Translate downstream transactions (ReadNoSnp, WriteBack) into DRAM commands.
• Explain why the controller may reorder freely — coherency was resolved upstream at the HN.
• State the limit on that freedom: it must preserve same-address ordering (memory-consistency).
• Distinguish coherency ordering (the HN's job) from same-address memory-consistency (the controller's).
• Implement a representative same-address hazard check in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

This is where the CHI abstraction meets the hardware you actually build. SN-F is a clean role; a DRAM controller is banks, rows, refresh, and a scheduler chasing bandwidth. Seeing how one becomes the other — and, crucially, why the controller is allowed to reorder aggressively — is what lets you build a fast, correct memory subordinate.

It also draws a line engineers routinely blur: coherency ordering versus memory ordering. The Home Node handles the first; the memory controller must still handle the second. Conflating them either kills performance (over-ordering) or corrupts data (under-ordering). This chapter makes the division precise.

3. Key Terms

4. Previous Chapter Connection

Chapter 5.5 defined SN-F abstractly: a coherency-agnostic completer that handles the downstream transactions HN-F forwards, via the DBID write handshake. It deliberately left the actual storage — "a memory array" — as a black box.

This chapter opens that box: the storage is a DRAM controller, with a request queue, a scheduler, and a DDR PHY. The SN-F front end (completion, DBID) sits in front of it. And the controller does something SN-F's abstraction hinted at but did not detail — it reorders requests for performance. This chapter shows why that is safe and where its limit lies.

5. Core Concept — a reordering DRAM controller behind a coherency-free front end

A memory controller in the SN-F role is a pipeline with one crucial freedom and one crucial limit.

• Front end. The SN-F interface accepts the downstream transactions HN-F forwards — ReadNoSnp (a coherent miss reading memory), WriteNoSnp / WriteBack (writes, including dirty evictions) — and runs the DBID write handshake (Chapter 5.5): allocate a buffer, return the DBID, accept the data.
• Queue and scheduler. Requests enter a queue; a scheduler picks the order they hit DRAM, chasing bank parallelism and row-buffer hits. Each request becomes DRAM commands — ACT a row, RD or WR, PRE to close it.
• The freedom: reorder across addresses. The scheduler may reorder requests to different addresses however it likes — because coherency was already resolved upstream at the Home Node, which serialized per address. Nothing the controller reorders can violate coherency.
• The limit: preserve same-address order. For the same address, order still matters: a read after a write to that address must see the write (RAW), and same-address writes must land in order (WAW). This is memory-consistency, and it is the controller's job — the HN did not do it at the DRAM level.

The synthesis:

A DRAM controller fills the SN-F role: it completes downstream transactions and schedules them onto DRAM for bandwidth. It may reorder freely across addresses, because the Home Node already resolved coherency and per-address order upstream. But it must still preserve same-address ordering (RAW/WAR/WAW), the memory-consistency the CPU depends on. Coherency ordering is the HN's; same-address memory ordering is the controller's — two different jobs at two different points.

6. Engineering Mental Model — a kitchen expediting orders

Picture the memory controller as a kitchen taking orders (requests) the front desk (HN-F) has already validated.

• The kitchen reorders dishes for efficiency: batch everything that uses the open oven (row-buffer hits), run stations in parallel (bank parallelism). The order tickets arrive validated, so the kitchen is free to cook them in whatever sequence is fastest.
• That freedom is safe because the front desk already sequenced anything that had to be sequenced (coherency, per-address order) — the kitchen never has to worry about which table conflicts with which.
• But for a single dish revised twice — "make it, then add sauce" (a write then a read of the same address) — the kitchen must keep that order. Serve the un-sauced version and the customer gets the wrong dish.

Reorder across tables for speed; preserve order within a single dish. That is the memory controller's rule.

7. Engineering Diagram — the DRAM controller as SN-F

The pipeline is a normal DRAM controller; what makes it an SN-F is only the front end (completion, DBID) and the fact that coherency has already been handled before requests arrive.

8. Downstream Transactions Become DRAM Commands

The HN-F's downstream set maps to DRAM operations; coherency is nowhere in this mapping.

Two facts to carry: every downstream transaction is a plain memory access — the scheduler sees reads and writes, not coherent states — and a row hit (the needed row already open) skips ACT/PRE and is much faster, which is exactly what the scheduler reorders to exploit. Coherency did its work upstream; here it is pure DRAM efficiency.

9. Why Reordering Is Safe — and Where It Isn't

The controller's aggression is licensed by the division of labor.

• Coherency ordering is done (HN). The Home Node serialized every access to each address and resolved which cache had the current data. By the time a request reaches the controller, it is a settled memory operation. So reordering requests to different addresses cannot break coherency — there is no coherency left to break here.
• Same-address ordering is not done (controller). The DRAM level still has memory-consistency hazards: a read after a write to one address must return the write (RAW); a write after a write must land in order (WAW); a write after a read must not overtake it (WAR). The controller must detect and honor these.
• So: reorder across addresses, order within an address. The scheduler is free across distinct addresses and constrained within one. Only read-after-read to the same address is unconstrained (no order needed).

The point to carry:

Two orderings, two owners. Coherency ordering — which cache's copy wins, in what global order — is the Home Node's and is finished upstream. Same-address memory ordering — RAW/WAR/WAW at one location — is the controller's and must be preserved even while everything else is reordered for speed. Forget the first and you over-order (slow); forget the second and you corrupt memory (the DebugLab).

10. Transaction Walkthrough — a writeback then a read to the same line

CPU0 writes back a dirty line A; moments later, CPU1 reads line A — both reach the controller.

1. Writeback arrives. HN-F forwards a WriteBack of line A; the SN-F front end allocates a DBID, accepts the dirty data, and queues the write.
2. Read arrives. HN-F forwards a ReadNoSnp of line A (CPU1 missed and, per the directory, memory now holds the current value). It queues behind the write.
3. Scheduler wants to reorder. For bandwidth, the scheduler would happily service the read first (maybe its row is open). But A's read and A's write are the same address — a RAW hazard.
4. Hazard held. The controller detects the same-address conflict and holds the read until the write to A has committed. Only then does it schedule the read.
5. Correct data. CPU1's read returns the just-written value. Had the read overtaken the write, CPU1 would have read the stale pre-writeback memory — the DebugLab.

For any other address, the scheduler could have reordered freely. It was the same address that forced the order — memory-consistency, not coherency.

11. RTL / Hardware View — a same-address hazard check

The controller's correctness guard is a same-address hazard detector: a candidate request must wait if an older pending request to the same address would be reordered past. Representative and combinational.

// Representative memory-controller same-address hazard check (educational).
// The scheduler may reorder across DIFFERENT addresses (coherency resolved
// upstream). For the SAME address it must preserve order: a candidate hazards if
// an older pending request to that address is a write, or the candidate is a
// write. Only read-after-read to one address is free.
module memctrl_hazard #(
  parameter int NSLOT = 4
)(
  input  logic [11:0] cand_addr,               // candidate request address
  input  logic        cand_is_read,            // candidate is a read
  input  logic [11:0] pend_addr    [0:NSLOT-1], // older pending requests
  input  logic        pend_valid   [0:NSLOT-1],
  input  logic        pend_is_write [0:NSLOT-1],
  output logic        hazard                    // candidate must wait
);
  logic h;
  always_comb begin
    h = 1'b0;
    for (int i = 0; i < NSLOT; i++)
      if (pend_valid[i] && (pend_addr[i] == cand_addr) &&
          (pend_is_write[i] || !cand_is_read))   // RAW / WAW / WAR (not R-after-R)
        h = 1'b1;
    hazard = h;
  end
endmodule

The same behavior in Verilog-2001:

// Representative memory-controller same-address hazard check (Verilog-2001).
module memctrl_hazard #(
  parameter NSLOT = 4
)(
  input  [11:0] cand_addr,
  input         cand_is_read,
  input  [11:0] pend_addr     [0:NSLOT-1],
  input         pend_valid    [0:NSLOT-1],
  input         pend_is_write [0:NSLOT-1],
  output        hazard
);
  reg h;
  integer i;
  always @* begin
    h = 1'b0;
    for (i = 0; i < NSLOT; i = i + 1)
      if (pend_valid[i] && (pend_addr[i] == cand_addr) &&
          (pend_is_write[i] || !cand_is_read))
        h = 1'b1;
  end
  assign hazard = h;
endmodule

And in VHDL:

-- Representative memory-controller same-address hazard check (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity memctrl_hazard is
  generic ( NSLOT : integer := 4 );
  port (
    cand_addr     : in  std_logic_vector(11 downto 0);
    cand_is_read  : in  std_logic;
    pend_addr     : in  std_logic_vector(12*NSLOT-1 downto 0);  -- packed
    pend_valid    : in  std_logic_vector(NSLOT-1 downto 0);
    pend_is_write : in  std_logic_vector(NSLOT-1 downto 0);
    hazard        : out std_logic
  );
end entity;
 
architecture rtl of memctrl_hazard is
begin
  process(cand_addr, cand_is_read, pend_addr, pend_valid, pend_is_write)
    variable h : std_logic;
  begin
    h := '0';
    for i in 0 to NSLOT-1 loop
      if pend_valid(i) = '1'
         and pend_addr(12*i+11 downto 12*i) = cand_addr
         and (pend_is_write(i) = '1' or cand_is_read = '0') then
        h := '1';
      end if;
    end loop;
    hazard <= h;
  end process;
end architecture;

All three flag a same-address conflict for every combination except read-after-read. The scheduler reorders freely subject to this one guard — the exact boundary between the controller's freedom (across addresses) and its duty (within one).

12. Verification View — same-address order held, cross-address free

Two properties: two reads to one address never hazard, and a write always hazards against any older same-address request.

// Bind to memctrl_hazard (illustrative single-pending checks).
// 1. Read-after-read to the SAME address is free (no hazard) — order irrelevant.
property p_rar_free;
  @(*) (pend_valid[0] && pend_addr[0] == cand_addr && !pend_is_write[0] && cand_is_read)
       |-> !hazard;   // (only this pending entry; extend across all slots in practice)
endproperty
 
// 2. A candidate write always hazards against any older same-address request.
property p_write_hazards;
  @(*) (pend_valid[0] && pend_addr[0] == cand_addr && !cand_is_read) |-> hazard;
endproperty

The system point, beyond the two checks:

The hazard guard encodes the exact split this chapter is about. The scheduler is maximally free: it may reorder anything to different addresses, because coherency and inter-address order were settled upstream. But it is not free on the same address, where RAW/WAR/WAW define what a program's memory operations mean. Get the guard right and the controller is both fast (reorders freely) and correct (never violates same-address order). Drop it and memory — the final store of coherent data — returns values from the wrong point in time.

• What it proves: read-after-read is free; writes are ordered against same-address predecessors.
• What it does not prove: DRAM timing correctness (a separate scheduling/PHY concern).
• Bug signature: a read returning pre-write data because it overtook a same-address write (the DebugLab).

13. Testbench — reads reorder freely, same-address writes do not

Checks that a different-address read is free, a same-address read-after-read is free, and a same-address read-after-write hazards.

module tb_memctrl_hazard;
  localparam int NSLOT = 4;
  logic [11:0] cand_addr;
  logic cand_is_read;
  logic [11:0] pend_addr [0:NSLOT-1];
  logic pend_valid [0:NSLOT-1];
  logic pend_is_write [0:NSLOT-1];
  logic hazard;
  int errors = 0;
 
  memctrl_hazard #(.NSLOT(NSLOT)) dut (.*);
 
  task automatic setpend(input int i, input logic v, input logic [11:0] a, input logic w);
    pend_valid[i] = v; pend_addr[i] = a; pend_is_write[i] = w;
  endtask
 
  task automatic check(input logic [11:0] a, input logic rd, input logic exp, input string tag);
    cand_addr = a; cand_is_read = rd; #1;
    if (hazard !== exp) begin errors++; $display("FAIL [%s] hazard=%b exp=%b", tag, hazard, exp); end
    else $display("PASS [%s] hazard=%b", tag, hazard);
  endtask
 
  initial begin
    for (int i=0;i<NSLOT;i++) setpend(i, 1'b0, 12'h0, 1'b0);
    setpend(0, 1'b1, 12'hA00, 1'b1);   // an older WRITE to A00 is pending
 
    check(12'hB00, 1'b1, 1'b0, "read to B00 (diff addr) -> free");
    check(12'hA00, 1'b1, 1'b1, "read to A00 after write -> hazard (RAW)");
    setpend(0, 1'b1, 12'hA00, 1'b0);   // older READ to A00
    check(12'hA00, 1'b1, 1'b0, "read to A00 after read -> free (RAR)");
    check(12'hA00, 1'b0, 1'b1, "write to A00 after read -> hazard (WAR)");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS [read to B00 (diff addr) -> free] hazard=0
PASS [read to A00 after write -> hazard (RAW)] hazard=1
PASS [read to A00 after read -> free (RAR)] hazard=0
PASS [write to A00 after read -> hazard (WAR)] hazard=1

14. DebugLab — a read reordered ahead of an older same-address write

queue order   op      addr    scheduled   note
    1        WRITE    0xA00      2nd       older write, delayed for a row hit
    2        READ     0xA00      1st       reordered ahead -> reads stale memory

15. Common Mistakes

• Reordering same-address accesses. Assumption: coherency resolved means reorder anything. Bug: RAW violation, stale read (the DebugLab). Prevention: preserve same-address RAW/WAR/WAW.
• Over-ordering everything. Assumption: keep all requests in order to be safe. Bug: no bandwidth. Prevention: reorder freely across different addresses.
• Putting coherency in the controller. Assumption: memory must snoop. Bug: duplicating the HN. Prevention: SN-F does no coherency; the HN did it.
• Ignoring row-buffer locality. Assumption: order does not affect speed. Bug: constant ACT/PRE, low bandwidth. Prevention: schedule for row hits and bank parallelism.
• Mishandling the DBID buffer. Assumption: write data lands anywhere. Bug: crossed writes (Chapter 5.5). Prevention: commit data only for an allocated DBID.
• Forgetting refresh/timing. Assumption: DRAM is always ready. Bug: timing violations. Prevention: honor tRCD/tRAS/refresh in the scheduler.

16. Engineering Checklist

• Map the SN-F front end onto a queue + scheduler + PHY; keep the DBID handshake.
• Translate ReadNoSnp / WriteBack into ACT / RD / WR / PRE.
• Reorder freely across addresses — coherency is resolved upstream.
• Preserve same-address order — RAW / WAR / WAW; only read-after-read is free.
• Schedule for row-buffer hits and bank parallelism; honor DRAM timing / refresh.
• Keep no coherency in the controller — that is the Home Node's.

17. Key Takeaways

• A DRAM controller fills the SN-F role: SN-F front end (completion, DBID) plus a queue, scheduler, and DRAM.
• Downstream transactions (ReadNoSnp, WriteBack) become plain DRAM commands — no coherency in the mapping.
• The scheduler may reorder freely across addresses, because the Home Node resolved coherency and per-address order upstream.
• It must still preserve same-address ordering (RAW/WAR/WAW) — memory-consistency the controller owns.
• Two orderings, two owners: coherency ordering = Home Node; same-address memory ordering = controller.
• Over-order and lose bandwidth; under-order the same address and corrupt memory. The model here is representative.

18. Quick Revision

Memory controllers in CHI. A real DRAM controller implements the SN-F role: the SN-F front end (completion + DBID write buffers) feeds a request queue and a scheduler that issues DRAM commands (ACT / RD / WR / PRE) to banks. Downstream transactions (ReadNoSnp, WriteBack) become plain memory accesses — no coherency in the controller. The scheduler may reorder freely across different addresses, because the Home Node resolved coherency and per-address order upstream. But it must preserve same-address ordering — RAW / WAR / WAW (only read-after-read is free) — the memory-consistency the controller owns. Two orderings, two owners: coherency = HN; same-address memory order = controller. Reorder across addresses for bandwidth; hold order within one for correctness. Representative model; 5.9 covers the interconnect fabric.

Coming Next

Chapter 5.9 — The Interconnect Fabric. The nodes are complete; the last chapter of the module returns to what connects them. Chapter 5.9 revisits the interconnect fabric — mesh, ring, and crossbar topologies — from the node's-eye view built up in Module 5: how RN, HN, and SN nodes attach to the fabric, how the topology choice shapes latency and bandwidth between them, and how the distributed Home Nodes of Chapter 4.5 sit on the mesh. It closes Module 5 by placing every node kind onto the physical structure that carries their packets.