The Data Channel (DAT)

Module 6 · Chapter 6.4 · CHI Channels

Project thread — 6.1–6.3 covered the control channels; this chapter takes DAT, the payload channel, and completes the four. 6.5 details SNP.

1. Learning Outcomes

By the end of this chapter you should be able to:

• List the key fields of a DAT packet — opcode, Data, DBID, TxnID, BE (byte mask), DataID.
• Name the data opcodes — CompData, DataSepResp, CopyBackWrData, WriteData, SnpRespData.
• Explain why a cache line transfers as multiple beats and how DataID orders them.
• Describe the byte mask (BE) and how it enables partial writes without a read-modify-write.
• State why write data carries a DBID and read data a TxnID.
• Implement a representative byte-mask write merge in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

DAT is where the actual data moves, so it dominates bandwidth and buffering. Understanding its beats — how a 64-byte line becomes several bus-width transfers — is essential for sizing data paths and reasoning about throughput.

Its subtlety is the byte mask. A partial write (a single-byte store to non-cached space, for instance) must touch only the written bytes and leave the rest untouched. The BE field makes that possible without a read-modify-write — and ignoring it silently corrupts neighbouring bytes. Getting the mask right is a correctness essential this chapter builds.

3. Key Terms

4. Previous Chapter Connection

Chapter 6.3 detailed RSP — the data-less responses, including CompData's partner form. It repeatedly deferred one thing: the payload. Reads need data, writes carry data, snoops may supply data — and none of that rides RSP.

It rides DAT, this chapter's subject. DAT is the only channel with bulk payload, and it completes the four. It carries the data those RSP completions referred to (CompData is a DAT packet), the write data a DBIDResp granted a buffer for, and the dirty data a snoop returned. With DAT, the channel set is complete.

5. Core Concept — payload, in beats, with a mask

DAT is the payload channel, defined by its fields, its opcodes, and two mechanics.

• The packet. A DAT flit carries the data opcode, the payload, a DBID (for writes — ties data to its buffer), SrcID/TgtID (routing), TxnID (for reads — correlates to the request), a BE byte mask, a DataID (which beat), and Resp/RespErr (for CompData). It is the only channel with bulk data.
• The opcodes. CompData (read data + completion) and DataSepResp (read data, completion separate) for reads; CopyBackWrData (writeback of a dirty eviction) and WriteData / NonCopyBackWrData (write data) for writes; SnpRespData (a snooped cache supplying data). The opcode names the data's role.
• Beats. A cache line (64 bytes) is wider than the data bus, so it transfers as multiple beats — 4 beats on a 128-bit bus, 2 on a 256-bit bus. Each beat carries a chunk tagged with a DataID, and the receiver reassembles the line by DataID.
• The byte mask (BE). Each beat carries byte enables — one bit per byte, marking which are valid. A full-line transfer has all BE set; a partial write sets only the written bytes, so the receiver writes only those and leaves the rest unchanged — no read-modify-write needed.

The synthesis:

DAT moves the payload: reads' data, writes' data, snoops' data — the only channel with bulk content. A line spans several beats (ordered by DataID), and a byte mask (BE) marks which bytes each beat actually writes, enabling partial writes without a read-modify-write. Write data is routed by DBID, read data correlated by TxnID. Honor the beats and the mask and the payload lands correctly.

6. Engineering Mental Model — a palletized shipment with a packing list

If REQ was the order and RSP the receipts, DAT is the actual shipment.

• A large order (a cache line) does not fit on one pallet, so it ships as several pallets (beats), each numbered (DataID) so the receiver stacks them in the right place.
• Each pallet comes with a packing list (BE) marking which boxes on it are real cargo versus empty slots. On a full shipment every box is cargo; on a partial shipment only some boxes are — and the receiver must unload only the marked boxes, leaving the existing stock in the empty slots untouched.
• The shipment carries the shelf number it was assigned (DBID) so the warehouse files it in the reserved bay.

Numbered pallets with per-pallet packing lists — that is DAT: beats with byte masks.

7. Engineering Diagram — the DAT packet

Two rows of fields, one of them the bulk payload. BE and DataID are the two that make DAT more than a wide data bus — mask and beat.

8. Data Opcodes

The opcode names the data's role in the transaction. A representative set:

Two facts to carry: read data flows home-to-requester and correlates by TxnID; write data flows requester-to-home/memory and routes by DBID (the buffer DBIDResp granted, Chapter 5.5). The opcode plus the ID tells the receiver what the data is and where it belongs.

9. Beats and the Byte Mask

Two mechanics define DAT beyond "a wide bus".

• Beats — a line in pieces. A 64-byte line exceeds the data bus, so it transfers as N beats (N = line size / bus width): 4 beats on 128-bit, 2 on 256-bit. Each beat carries a DataID (its position), and the receiver reassembles the line by DataID. The final beat is marked last.
• The byte mask (BE) — which bytes count. Each beat carries byte enables, one bit per byte. A full transfer (a read, a full-line write) has all BE set. A partial write sets only the written bytes, so the receiver writes only those and preserves the others.
• Why the mask matters. Without BE, a partial write would need a read-modify-write (read the line, merge the new bytes, write it back). BE lets the write carry its own mask and touch just the intended bytes directly — faster, and essential for correctness on device/non-cacheable stores.

The point to carry:

A DAT transfer is beats (reassembled by DataID) plus a byte mask (BE) on each beat. The mask is not decoration: on a partial write, the receiver must write exactly the enabled bytes and leave the rest untouched. Ignore BE and a partial write clobbers bytes it never meant to change — the DebugLab.

10. DAT Beats in Time

A full-line read shows the beats: several data transfers reassembled into one line.

Four beats, DataID 0 to 3, all bytes enabled (a full read), the last flagged. A partial write would look the same but with BE marking only some bytes on the written beat.

11. RTL / Hardware View — a byte-mask write merge

The byte mask's whole job is at the receiver: write enabled bytes, keep the rest. Here is that merge for one beat. Representative and combinational.

// Representative DAT byte-mask write merge (educational).
// A write beat carries data + a byte-enable mask (BE). Each output byte takes
// the new data when its BE bit is set, otherwise keeps the old stored value.
// This lets a partial write touch only some bytes — no read-modify-write.
module dat_be_merge #(
  parameter int NB = 4                            // bytes per beat
)(
  input  logic [NB-1:0]     be,                    // byte-enable mask (1 = write)
  input  logic [NB*8-1:0]   wdata,                 // incoming write data
  input  logic [NB*8-1:0]   old,                   // current stored value
  output logic [NB*8-1:0]   merged                 // value written back
);
  always_comb begin
    for (int i = 0; i < NB; i++)
      merged[i*8 +: 8] = be[i] ? wdata[i*8 +: 8] : old[i*8 +: 8];
  end
endmodule

The same behavior in Verilog-2001:

// Representative DAT byte-mask write merge (Verilog-2001).
module dat_be_merge #(
  parameter NB = 4
)(
  input      [NB-1:0]    be,
  input      [NB*8-1:0]  wdata,
  input      [NB*8-1:0]  old,
  output reg [NB*8-1:0]  merged
);
  integer i;
  always @* begin
    for (i = 0; i < NB; i = i + 1)
      merged[i*8 +: 8] = be[i] ? wdata[i*8 +: 8] : old[i*8 +: 8];
  end
endmodule

And in VHDL:

-- Representative DAT byte-mask write merge (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity dat_be_merge is
  generic ( NB : integer := 4 );
  port (
    be     : in  std_logic_vector(NB-1 downto 0);
    wdata  : in  std_logic_vector(NB*8-1 downto 0);
    old    : in  std_logic_vector(NB*8-1 downto 0);
    merged : out std_logic_vector(NB*8-1 downto 0)
  );
end entity;
 
architecture rtl of dat_be_merge is
begin
  process(be, wdata, old)
  begin
    for i in 0 to NB-1 loop
      if be(i) = '1' then
        merged(i*8+7 downto i*8) <= wdata(i*8+7 downto i*8);   -- write this byte
      else
        merged(i*8+7 downto i*8) <= old(i*8+7 downto i*8);     -- keep old byte
      end if;
    end loop;
  end process;
end architecture;

All three write a byte from wdata only when its BE bit is set, and keep old otherwise. That per-byte gate is the entire correctness of a partial write; dropping it is the DebugLab.

12. Verification View — enabled bytes written, disabled bytes preserved

Two properties: an enabled byte equals the new data, and a disabled byte equals the old value.

// Bind to dat_be_merge (NB = 4).
// 1. Enabled bytes take the incoming write data.
genvar b;
generate for (b = 0; b < 4; b++) begin : g_en
  always_comb if (be[b]) assert (merged[b*8 +: 8] == wdata[b*8 +: 8]);
end endgenerate
 
// 2. Disabled bytes preserve the old stored value.
generate for (b = 0; b < 4; b++) begin : g_dis
  always_comb if (!be[b]) assert (merged[b*8 +: 8] == old[b*8 +: 8]);
end endgenerate

The system point, beyond the two checks:

The byte mask is a correctness contract, not a hint. A partial write's disabled-byte lanes may carry anything on the bus — the sender only guarantees the enabled bytes. So the receiver must merge per-BE: enabled bytes from the write, disabled bytes from memory. Writing the whole beat treats the don't-care lanes as data and corrupts bytes that were never part of the write. Every DAT receiver that stores partial writes must honor BE byte-by-byte, exactly as this merge does.

• What it proves: enabled bytes are written, disabled bytes preserved.
• What it does not prove: beat reassembly or DBID routing (separate DAT mechanics).
• Bug signature: neighbouring bytes of a partial write corrupted — an unmasked whole-beat write (the DebugLab).

13. Testbench — a partial write merges correctly

Merges a partial write (two bytes enabled) into an old value and checks each byte.

module tb_dat_be_merge;
  localparam int NB = 4;
  logic [NB-1:0] be;
  logic [NB*8-1:0] wdata, old, merged;
  int errors = 0;
 
  dat_be_merge #(.NB(NB)) dut (.*);
 
  initial begin
    old   = 32'hAA_BB_CC_DD;        // existing bytes: [3]=AA [2]=BB [1]=CC [0]=DD
    wdata = 32'h11_22_33_44;        // new bytes:      [3]=11 [2]=22 [1]=33 [0]=44
    be    = 4'b0101;                // write bytes 0 and 2 only
    #1;
    // expect: byte0=44 (BE), byte1=CC (keep), byte2=22 (BE), byte3=AA (keep)
    if (merged !== 32'hAA_22_CC_44) begin
      errors++; $display("FAIL merged=%h exp=AA22CC44", merged);
    end else $display("PASS partial merge = %h (enabled 0,2 written; 1,3 kept)", merged);
 
    // Full mask -> all new.
    be = 4'b1111; #1;
    if (merged !== wdata) begin errors++; $display("FAIL full mask"); end
    else $display("PASS full mask = wdata");
 
    // Empty mask -> all old.
    be = 4'b0000; #1;
    if (merged !== old) begin errors++; $display("FAIL empty mask"); end
    else $display("PASS empty mask = old");
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS partial merge = aa22cc44 (enabled 0,2 written; 1,3 kept)
PASS full mask = wdata
PASS empty mask = old
ALL TESTS PASSED

14. DebugLab — writing the whole beat, ignoring BE

be = 0101 (write bytes 0,2)   wdata = 11 22 33 44   old = AA BB CC DD
correct merge (per-BE):       AA 22 CC 44
buggy whole-beat write:       11 22 33 44   <- bytes 1,3 clobbered (BE ignored)

15. Common Mistakes

• Ignoring BE on a partial write. Assumption: write the whole beat. Bug: clobbered neighbouring bytes (the DebugLab). Prevention: merge per byte from BE.
• Assuming one beat per line. Assumption: a line fits in one transfer. Bug: dropped chunks. Prevention: a line spans N beats; reassemble by DataID.
• Reassembling by arrival order. Assumption: beats arrive in order. Bug: mis-ordered line. Prevention: reassemble by DataID, not arrival.
• Confusing DBID and TxnID. Assumption: one ID for all data. Bug: mis-routed data. Prevention: write data routes by DBID; read data correlates by TxnID.
• Trusting disabled-byte lanes. Assumption: all lanes carry data. Bug: acting on don't-care bytes. Prevention: only BE-enabled bytes are valid.
• Looking for control on DAT. Assumption: completions live here. Bug: mishandled flow. Prevention: DAT is payload; control is on RSP (except CompData's bundled completion).

16. Engineering Checklist

• Read the DAT opcode (CompData / DataSepResp / CopyBackWrData / WriteData / SnpRespData).
• Transfer a line as N beats; reassemble by DataID, honor last.
• Merge partial writes per BE — write enabled bytes, preserve the rest.
• Route write data by DBID; correlate read data by TxnID.
• Treat disabled-byte lanes as don't-care — never store them.
• Size data buffers for the beat count and outstanding transfers.

17. Key Takeaways

• DAT is the only payload channel: reads' data, writes' data, snoops' data.
• Its opcodes: CompData / DataSepResp (reads), CopyBackWrData / WriteData (writes), SnpRespData (snoops).
• A cache line transfers as multiple beats, each tagged with a DataID for reassembly; the last beat is flagged.
• The byte mask (BE) marks valid bytes, enabling partial writes without a read-modify-write.
• A receiver must honor BE byte-by-byte — writing the whole beat corrupts disabled (don't-care) lanes.
• Write data routes by DBID, read data correlates by TxnID; the model here is representative.

18. Quick Revision

The DAT channel. The only payload channel. Fields: data opcode, the Data, DBID (write data → its buffer), TxnID (read data → its request), BE (byte mask), DataID (which beat), Resp/RespErr. Opcodes: CompData / DataSepResp (read data, combined / separate completion), CopyBackWrData / WriteData (writes), SnpRespData (snoop data). Two mechanics: beats — a 64B line exceeds the bus, so it transfers as N beats (4 on 128-bit, 2 on 256-bit), each tagged with DataID and reassembled, last beat flagged; and the byte mask (BE) — one bit per byte, marking valid bytes, so a partial write touches only enabled bytes (no read-modify-write) and the receiver preserves disabled (don't-care) lanes. Honor BE byte-by-byte or a partial write clobbers neighbours. Representative model; 6.5 covers SNP.

Coming Next

Chapter 6.5 — The Snoop Channel (SNP). Three channels down; the fourth is the snoop. The next chapter details SNP: its packet structure, the snoop request types (SnpShared, SnpUnique, SnpClean, SnpOnce…), and the home-to-RN routing that sends a snoop from the Home Node to exactly the caches its directory names. Where DAT carried the data a snoop returns, SNP carries the snoop itself — completing the four-channel tour of Module 6.