Data Packets

Module 13 · Chapter 13.1 · CHI Data Transfers

Project thread — Module 12 settled ordering. This module is data transport. 13.1 is the data packet's structure; 13.2 covers who supplies the data.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Name the fields of a CHI data packet — data, DataID, correlation, byte-enables, error status.
• Explain that a cache line is delivered over multiple beats when the bus is narrower than the line.
• State that DataID names the beat's position in the line.
• Describe reassembly — each beat placed at the position its DataID specifies.
• Diagnose the scrambled line from reassembling by arrival order instead of DataID.
• Implement a representative beat reassembler in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Every transaction you have studied ultimately moves data, and the data packet is how it moves. Understanding its structure is understanding the concrete unit the DAT channel carries — the payload plus the fields that let the receiver place it, correlate it, and check it. Without that framing, a chunk of bytes on a bus is meaningless; with it, the receiver knows exactly which transaction and which part of which line it belongs to.

The practical crux is multi-beat delivery. A cache line is typically wider than the data bus, so it arrives in pieces — beats — and each beat carries a DataID naming its position in the line. The receiver must place each beat by its DataID, because beats can arrive out of order in a mesh. Reassembling by the order beats happen to arrive instead of by their DataID silently scrambles the line — a data-corruption bug that is invisible until the wrong bytes are read. This chapter is the foundation of data transport: what a data packet is, and how a line is put back together correctly.

3. Key Terms

4. Previous Chapter Connection

Modules 6–12 treated the DAT channel abstractly — CompData, SnpRespData, WriteData were "the data." This module opens up that data, starting with its packet structure.

You have seen the correlation ideas already: reads echo the TxnID (Chapter 6.6), writes are tagged with the DBID (Chapter 7.2), partial writes carry byte-enables (Chapter 8.5), and dirty data carries error/status alongside. This chapter collects those into the data packet's fields and adds the transport reality Modules 6–12 abstracted away: a line is delivered over beats, each with a DataID. So the data you have been routing and forwarding has an internal structure, and 13.1 is that structure.

5. Core Concept — a framed packet, a line over beats

A CHI data packet is a framed unit, and a cache line is delivered as several beats, each placed by its DataID.

• The fields. A data packet carries the data payload (a chunk of the line), a DataID (the chunk's position in the line), a correlation tag (DBID for writes, echoed TxnID for reads, plus SrcID/TgtID), byte-enables (valid bytes for partial writes), and a RespErr (error/poison status).
• Lines are wider than the bus. A cache line (say 64 bytes) is usually wider than the data bus (say 32 bytes), so a full line takes multiple beats — here two.
• DataID names the position. Each beat's DataID says which part of the line it carries — beat 0 is the low half, beat 1 the high half. The DataID is the beat's address within the line.
• Reassemble by DataID. The receiver places each beat at the position its DataID specifies, not the order beats arrive — because the interconnect may deliver them out of order (Chapter 6.7). The line is complete when all positions are filled.

The synthesis:

A CHI data packet frames a chunk of line data with its DataID (position), a correlation tag (DBID/TxnID), byte-enables, and error status. A cache line, wider than the bus, arrives as several beats, each carrying one chunk and its DataID. The receiver reassembles by DataID — placing each beat at its named position — not by arrival order, because beats may arrive reordered.

6. Engineering Mental Model — a numbered jigsaw

Think of a cache line as a picture cut into a few numbered jigsaw pieces.

• Each piece (a beat) has a number printed on the back — its position in the picture. That number is the DataID.
• The pieces are mailed separately and may arrive in any order — piece 1 might come before piece 0.
• To rebuild the picture, you place each piece by its printed number, not by the order the envelopes arrived. Piece 0 goes in slot 0 no matter when it turned up.
• If you instead laid pieces down in arrival order — first arrival in slot 0, second in slot 1 — a reordered delivery would put the pieces in the wrong slots, and the picture would be scrambled.

The printed number is the truth about where a piece belongs; the mail order is noise. A data packet's DataID is that printed number, and reassembly follows it.

7. Engineering Diagram — the data packet's fields

Five fields frame every data packet. The DataID is the one this chapter turns on: it is the beat's position, and the receiver uses it to place the chunk. The correlation tag ties the packet to its transaction; byte-enables and RespErr qualify the payload.

8. The Data Packet's Fields

Each field and its job.

The rule to carry: a data packet is payload plus placement plus provenance plus qualifiers. The payload is a chunk; the DataID places it in the line; the correlation ties it to a transaction; the byte-enables and RespErr qualify which bytes are valid and whether the data is good. The receiver needs all four to use a chunk correctly — and the DataID is what turns a stream of chunks back into a line.

9. Multi-Beat Delivery and Reassembly

The transport reality deserves its own statement.

• A line takes several beats. When the line is wider than the data bus, the line is split into beats — one chunk per bus transfer. A 64-byte line on a 32-byte bus is two beats.
• Each beat carries its DataID. The DataID names the beat's position in the line — beat 0 the low chunk, beat 1 the next, and so on.
• Beats may arrive reordered. Across a mesh, the beats of one line may be delivered out of order — beat 1 before beat 0 (Chapter 6.7).
• Reassemble by DataID. The receiver places each beat at the position its DataID specifies, filling the line's slots regardless of arrival order. The line is complete when every position has been received.

The point to carry:

The DataID exists precisely because arrival order is unreliable. In a point-to-point bus, beats might arrive in order and you could get away with reassembling by arrival — but a mesh reorders, so the packet must carry its own position, and the receiver must trust that position over the order of arrival. This is the same lesson as routing (a packet carries its destination) applied to reassembly (a beat carries its position): the transport does not preserve order, so order must be encoded in the data. Reassembling by arrival is assuming an ordering the fabric does not provide — and the result is a line with its chunks in the wrong places.

10. Reading a Data Packet — reassembling a two-beat line

A 64-byte line arrives as two beats on a 32-byte bus; they arrive out of order.

1. Beat with DataID 1 arrives first. It carries the high 32 bytes of the line. The receiver places it at position 1 — the high half — because its DataID says so.
2. Beat with DataID 0 arrives second. It carries the low 32 bytes. The receiver places it at position 0 — the low half.
3. Both positions filled. The line's low and high halves are now correct, each placed by its DataID, regardless of the reversed arrival.
4. Line complete. With every DataID position received, the line is reassembled correctly and delivered to the cache.
5. Correlated and checked. The correlation tag matched the beats to the outstanding transaction; RespErr confirmed the data is good.

The line came out correct because each beat was placed by its DataID, not its arrival. Had the receiver placed the first arrival (DataID 1) at position 0, the halves would be swapped — the DebugLab.

11. RTL / Hardware View — a beat reassembler

The receiver places each beat's chunk at the position its DataID names and completes when all positions are received. Representative — a two-beat line for concreteness.

// Representative data-beat reassembler (educational).
// Each beat carries a DataID (its position in the line) and a data chunk. Place the
// chunk at the position DataID names -- NOT in arrival order -- and mark that
// position received. The line is complete when all positions have arrived.
module chi_data_reasm #(parameter BEATS = 2, parameter CW = 256) (  // BEATS chunks, CW-bit each
  input  logic                    clk, rst_n,
  input  logic                    beat_valid,   // a beat is arriving
  input  logic [$clog2(BEATS)-1:0] beat_dataid, // its position in the line
  input  logic [CW-1:0]           beat_data,    // the chunk
  output logic [BEATS*CW-1:0]     line,         // the reassembled line
  output logic [BEATS-1:0]        received,      // which positions have arrived
  output logic                    line_complete  // all positions received
);
  logic [BEATS-1:0] rcv_q;
  logic [BEATS*CW-1:0] line_q;
 
  assign received      = rcv_q;
  assign line          = line_q;
  assign line_complete = &rcv_q;   // all positions received
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rcv_q  <= '0;
      line_q <= '0;
    end else if (beat_valid) begin
      // Place the chunk at the position DataID names -- reassemble BY DataID.
      line_q[beat_dataid*CW +: CW] <= beat_data;
      rcv_q[beat_dataid]           <= 1'b1;
    end
  end
endmodule

The same behavior in Verilog-2001:

// Representative data-beat reassembler (Verilog-2001).
module chi_data_reasm #(parameter BEATS = 2, parameter CW = 256, parameter IDW = 1) (
  input                    clk, rst_n, beat_valid,
  input  [IDW-1:0]         beat_dataid,
  input  [CW-1:0]          beat_data,
  output [BEATS*CW-1:0]    line,
  output [BEATS-1:0]       received,
  output                   line_complete
);
  reg [BEATS-1:0]     rcv_q;
  reg [BEATS*CW-1:0]  line_q;
  assign received      = rcv_q;
  assign line          = line_q;
  assign line_complete = &rcv_q;
 
  always @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      rcv_q  <= {BEATS{1'b0}};
      line_q <= {(BEATS*CW){1'b0}};
    end else if (beat_valid) begin
      line_q[beat_dataid*CW +: CW] <= beat_data;
      rcv_q[beat_dataid]           <= 1'b1;
    end
  end
endmodule

And in VHDL:

-- Representative data-beat reassembler (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_data_reasm is
  generic ( BEATS : integer := 2; CW : integer := 256 );
  port (
    clk, rst_n    : in  std_logic;
    beat_valid    : in  std_logic;
    beat_dataid   : in  unsigned(0 downto 0);     -- position (1 bit for 2 beats)
    beat_data     : in  std_logic_vector(CW-1 downto 0);
    line          : out std_logic_vector(BEATS*CW-1 downto 0);
    received      : out std_logic_vector(BEATS-1 downto 0);
    line_complete : out std_logic
  );
end entity;
 
architecture rtl of chi_data_reasm is
  signal rcv_q  : std_logic_vector(BEATS-1 downto 0) := (others => '0');
  signal line_q : std_logic_vector(BEATS*CW-1 downto 0) := (others => '0');
begin
  received      <= rcv_q;
  line          <= line_q;
  line_complete <= '1' when (rcv_q = (rcv_q'range => '1')) else '0';
 
  process (clk, rst_n)
    variable p : integer;
  begin
    if rst_n = '0' then
      rcv_q  <= (others => '0');
      line_q <= (others => '0');
    elsif rising_edge(clk) then
      if beat_valid = '1' then
        p := to_integer(beat_dataid);
        line_q((p+1)*CW-1 downto p*CW) <= beat_data;  -- place BY DataID
        rcv_q(p) <= '1';
      end if;
    end if;
  end process;
end architecture;

All three place each beat's chunk at the position its DataID names — line[beat_dataid*CW +: CW] — never at an arrival-order index, and complete when every position bit is set. The DebugLab is placing by arrival instead.

12. Verification View — reassembly follows DataID

The properties that keep reassembly correct: each beat lands at its DataID position, and completion needs all positions.

// Bind to chi_data_reasm.
// 1. A received beat sets exactly its DataID's position bit.
property p_beat_marks_its_position;
  @(posedge clk) disable iff (!rst_n)
    beat_valid |=> received[$past(beat_dataid)];
endproperty
 
// 2. The chunk lands at the DataID's slice (checked by placing a known value).
//    line[dataid*CW +: CW] == the beat_data written for that DataID.
 
// 3. The line is complete only when every position has been received.
property p_complete_needs_all;
  @(posedge clk) disable iff (!rst_n)
    line_complete == (&received);
endproperty

The system point, beyond the checks:

A data packet is a small lesson in self-describing transport: because the fabric guarantees neither order nor timing, each unit must carry the metadata needed to be placed and interpreted on its own — its position (DataID), its transaction (correlation), its valid bytes (byte-enables), its health (RespErr). The receiver is then a pure function of the packets it holds, not of the sequence they arrived in. This is what makes the DAT channel robust across a reordering mesh: reassembly, correlation, and validation all key off fields in the packet, never off arrival order or timing. The DataID in particular is the antidote to reordering — it lets a scattered set of beats be put back into one correct line, which is exactly the property a naive arrival-order reassembly throws away.

• What it proves: each beat marks and fills its DataID position; completion needs all positions.
• What it does not prove: the data's correctness or source — those are RespErr (13.7) and the sources (13.2).
• Bug signature: a beat placed at an arrival-order index rather than its DataID — a scrambled line.

13. Testbench — reassembling out-of-order beats

Delivers a two-beat line out of order and checks it reassembles correctly by DataID.

module tb_chi_data_reasm;
  localparam BEATS = 2, CW = 8;   // small chunks for the test
  logic clk = 0, rst_n = 0, beat_valid = 0;
  logic [0:0] beat_dataid;
  logic [CW-1:0] beat_data;
  logic [BEATS*CW-1:0] line;
  logic [BEATS-1:0] received;
  logic line_complete;
  int errors = 0;
 
  chi_data_reasm #(.BEATS(BEATS), .CW(CW)) dut (.*);
  always #5 clk = ~clk;
 
  initial begin
    @(posedge clk) rst_n = 1;
 
    // Beat DataID 1 (high half) arrives FIRST.
    @(posedge clk) begin beat_valid = 1; beat_dataid = 1'd1; beat_data = 8'hAA; end
    @(posedge clk) begin beat_valid = 0; end
    if (line_complete) begin errors++; $display("FAIL complete after 1 beat"); end
    else $display("PASS not complete after 1 beat");
 
    // Beat DataID 0 (low half) arrives SECOND.
    @(posedge clk) begin beat_valid = 1; beat_dataid = 1'd0; beat_data = 8'h55; end
    @(posedge clk) begin beat_valid = 0; end
 
    // Line should be high=AA (pos 1), low=55 (pos 0) -> 0xAA55, despite reversed arrival.
    if (!line_complete) begin errors++; $display("FAIL not complete after both beats"); end
    else if (line !== 16'hAA55) begin errors++; $display("FAIL line=%04h exp AA55 (scrambled)", line); end
    else $display("PASS reassembled by DataID: line=%04h", line);
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS not complete after 1 beat
PASS reassembled by DataID: line=aa55
ALL TESTS PASSED

14. DebugLab — reassembling beats by arrival order

line = 2 beats; DataID 0 = low half, DataID 1 = high half
interconnect delivers out of order: DataID 1 arrives first, DataID 0 second
receiver places by ARRIVAL: first arrival -> position 0, second -> position 1
  -> DataID 1 (high) written to position 0, DataID 0 (low) to position 1
  -> line halves SWAPPED -> scrambled
correct: place by DataID -> DataID 1 to position 1, DataID 0 to position 0

15. Common Mistakes

• Reassembling by arrival order. Assumption: beats arrive in order. Bug: scrambled line (the DebugLab). Prevention: place by DataID.
• Ignoring the DataID. Assumption: single-beat only. Bug: mis-assembled wide lines. Prevention: use DataID for placement.
• Losing the correlation tag. Assumption: data matches by timing. Bug: mismatched transaction. Prevention: correlate by DBID/TxnID.
• Dropping byte-enables. Assumption: full-line data. Bug: partial-write corruption (Chapter 8.5). Prevention: carry byte-enables.
• Ignoring RespErr. Assumption: data is always good. Bug: consuming poisoned data (Chapter 13.7). Prevention: check RespErr.
• Completing before all beats. Assumption: one beat suffices. Bug: an incomplete line used. Prevention: complete only when all positions received.

16. Engineering Checklist

• Frame each data packet with payload, DataID, correlation, byte-enables, RespErr.
• Deliver a wide line over multiple beats, one chunk per beat.
• Tag each beat with its DataID — its position in the line.
• Reassemble by DataID, never by arrival order.
• Mark the line complete only when every position is received.
• Correlate the data to its transaction and check RespErr.

17. Key Takeaways

• A CHI data packet carries the payload, DataID, correlation, byte-enables, and RespErr.
• A cache line wider than the bus is delivered over multiple beats.
• Each beat's DataID names its position in the line.
• The receiver reassembles by DataID, not by arrival order — beats may arrive reordered.
• Reassembling by arrival order scrambles the line under out-of-order delivery.
• Place by DataID, complete on all positions; the model here is representative.

18. Quick Revision

Data packets. A CHI data packet on the DAT channel is a framed unit: a data payload (a chunk of the cache line), a DataID (the chunk's position in the line), a correlation tag (a DBID for writes or the echoed TxnID for reads, with SrcID/TgtID), byte-enables (valid bytes for partial writes), and a RespErr (error/poison status). Because a cache line is usually wider than the data bus, a full line arrives as several beats, each carrying one chunk and its DataID. The receiver must reassemble by DataID — placing each beat at the position its DataID names — not by arrival order, because a mesh may deliver the beats out of order. Reassembling by arrival assumes an ordering the fabric does not provide, so a reordered delivery writes chunks to the wrong positions and scrambles the line — silent corruption. The DataID is the beat's position; trust it over arrival, and complete the line only when every position has been received. Representative model; 13.2 covers who supplies the data.

Coming Next

Chapter 13.2 — Data Sources. A data packet has to come from somewhere; this next chapter enumerates where. Chapter 13.2 covers the data sources — memory (the Subordinate Node), the home node's own cache, and a peer Request Node's cache — how the home chooses which source holds the current data, and why sourcing from the wrong one is either slow or, for dirty data, stale.