The Request Channel (REQ)

Module 6 · Chapter 6.2 · CHI Channels

Project thread — 6.1 surveyed the four channels; this chapter details the first, REQ, where every transaction starts. 6.3 takes the response channel, RSP.

1. Learning Outcomes

By the end of this chapter you should be able to:

• List the key fields of a REQ packet — opcode, addr, SrcID, TgtID, TxnID, Size, QoS, Order.
• Group the transaction kinds REQ carries: reads, writes, cache-maintenance operations, atomics.
• Trace source-to-home routing: address → system address map → owning Home Node → TgtID.
• Explain the request retry: RetryAck, then a protocol-credit grant, then re-send.
• State why a requester must wait for PCrdGrant before retrying — forward progress.
• Implement a representative REQ retry machine in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

REQ is where every transaction is born, so its fields and rules shape everything downstream. Read the opcode and you know the transaction; read the TgtID and you know where it goes; read Order and QoS and you know how it is handled. Fluency in the REQ packet is fluency in CHI transactions.

The retry mechanism is the part engineers most often get wrong. A Home Node can be temporarily out of resources, and CHI handles that with a credit-based retry — not by re-sending blindly. Understanding RetryAck and PCrdGrant is what separates a design that makes forward progress from one that livelocks under load.

3. Key Terms

4. Previous Chapter Connection

Chapter 6.1 mapped the four channels and their orthogonality; it named REQ as requests from RN to HN without opening the packet.

This chapter opens it. It details the REQ packet's fields, the full menu of transactions it carries, how it is routed to the right Home Node, and the retry handshake that keeps requests flowing when a Home Node is momentarily full. Where 6.1 was the survey, 6.2 is the first channel in depth — and, because every transaction starts here, the most consequential.

5. Core Concept — the request packet, routed and retried

The REQ channel is a packet, a routing rule, and a retry protocol.

• The packet. A REQ flit carries the opcode (what transaction), addr (where in memory), SrcID (the requester), TgtID (the Home Node), TxnID (to correlate responses), Size (transfer size), and attributes — Order (ordering requirement), QoS (priority), MemAttr (cacheable / device), Excl (exclusive access), and more. Everything the Home Node needs to act is in the fields.
• The transaction kinds. REQ carries the whole request menu: reads (ReadShared, ReadUnique, ReadNoSnp…), writes (WriteBackFull, WriteUnique, WriteNoSnp…), cache-maintenance (CleanShared, CleanInvalid, MakeUnique, Evict), and atomics (AtomicStore, AtomicLoad, AtomicSwap, AtomicCompare). The opcode selects one.
• Source-to-home routing. The requester computes the TgtID from the address via the system address map — address → owning Home Node → Node ID — and the fabric routes the REQ there. Every request goes to the one Home Node that owns its address (Chapter 4.5).
• The retry. A Home Node with no free resources returns RetryAck on RSP. The requester does not re-send blindly; it waits for a PCrdGrant (protocol credit) and only then re-issues the request. This credit-based retry guarantees the Home Node has room, so the retry succeeds and the system makes progress.

The synthesis:

The REQ channel starts every transaction: a self-describing request packet, routed to the owning Home Node by the system address map, and — when the home is busy — retried under a credit grant so it always eventually lands. Read the packet's fields and you know the transaction; follow the TgtID and you know its home; honor the retry and you keep the fabric moving.

6. Engineering Mental Model — an addressed, tracked work order

Picture a REQ as a work order dropped into a company's mail system.

• The order form has all the fields filled in: what to do (opcode), which file (address), who sent it (SrcID), which department it goes to (TgtID), and a tracking number (TxnID), plus handling notes — priority (QoS) and ordering (Order).
• The mailroom routes by department: the sender looks up which department owns that file (the system address map) and addresses the order there. It always reaches the one department responsible.
• If that department is swamped, it does not lose the order — it sends back "we are full, hold on" (RetryAck) and, when a desk frees up, "go ahead now" (PCrdGrant). The sender waits for the go-ahead before resubmitting, so the resubmission is guaranteed a desk.

An addressed, tracked, retry-safe work order — that is a CHI request.

7. Engineering Diagram — the REQ packet

Two rows of fields, one packet. The opcode and address say what and where; the IDs say who and to whom; Size, Order, and QoS say how.

8. Transaction Kinds on REQ

The opcode selects one of four families. A representative sample:

Two facts to carry: the opcode fully determines the transaction the Home Node runs, and the families map to intent — read to obtain, write to store, CMO to manage state, atomic to update in place. Every request on the channel is one of these, selected by the opcode field.

9. Source-to-Home Routing

A request must reach the one Home Node that owns its address. The path:

1. Address in hand. The requester has the transaction's address.
2. Consult the SAM. The system address map turns the address into the owning Home Node's Node ID — the same global, interleaved map from Chapters 3.8 and 4.5.
3. Set TgtID. That Node ID becomes the REQ packet's TgtID.
4. Route by TgtID. The fabric routes the REQ to that Home Node, hop by hop, reading only TgtID (Chapter 4.7).

The point to carry:

Every request is source-routed to its home: the requester computes the owning Home Node from the address via the SAM and stamps it as TgtID; the fabric does the rest. Because the SAM is global and identical everywhere (Chapter 3.8), a given address always reaches the same Home Node — the single point of coherence for that line. Get the SAM wrong on one requester and its requests reach the wrong home — split coherence.

10. The REQ Handshake and Retry

A request is not always accepted on the first try. The retry flow on the waveform:

The requester did not re-send between the RetryAck and the PCrdGrant. Waiting for the credit is what makes the retry succeed — the Home Node granted it because it now has room.

11. RTL / Hardware View — the REQ retry machine

The requester side of the retry is a small state machine: send, and on a RetryAck wait for a credit before re-sending. Representative and simplified.

// Representative REQ retry machine, requester side (educational).
// Send a request; if the Home Node returns RetryAck, wait for a PCrdGrant, then
// re-send. Never re-send without the credit -> credit-based retry, forward
// progress. States: IDLE, SENT (awaiting accept/retry), WAIT_CRD (awaiting credit).
module req_retry_fsm (
  input  logic clk,
  input  logic rst_n,
  input  logic req_pending,     // requester has a request to send
  input  logic accepted,        // request accepted by the Home Node
  input  logic retry_ack,       // RetryAck: no protocol credit
  input  logic pcrd_grant,      // PCrdGrant: a protocol credit
  output logic send_req,        // drive a REQ flit this cycle
  output logic done             // accepted -> complete
);
  localparam logic [1:0] IDLE = 2'd0, SENT = 2'd1, WAIT_CRD = 2'd2;
  logic [1:0] st;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) st <= IDLE;
    else case (st)
      IDLE:     if (req_pending)    st <= SENT;             // send first try
      SENT:     if (accepted)       st <= IDLE;             // done
                else if (retry_ack) st <= WAIT_CRD;         // must retry
      WAIT_CRD: if (pcrd_grant)     st <= SENT;             // credit -> re-send
      default:  st <= IDLE;
    endcase
  end
  // Send when starting a request, or on a credit grant (the retry).
  assign send_req = (st == IDLE && req_pending) || (st == WAIT_CRD && pcrd_grant);
  assign done     = (st == SENT && accepted);
endmodule

The same behavior in Verilog-2001:

// Representative REQ retry machine (Verilog-2001).
module req_retry_fsm (
  input  clk, rst_n, req_pending, accepted, retry_ack, pcrd_grant,
  output send_req, done
);
  localparam IDLE = 2'd0, SENT = 2'd1, WAIT_CRD = 2'd2;
  reg [1:0] st;
  always @(posedge clk or negedge rst_n)
    if (!rst_n) st <= IDLE;
    else case (st)
      IDLE:     if (req_pending)    st <= SENT;
      SENT:     if (accepted)       st <= IDLE;
                else if (retry_ack) st <= WAIT_CRD;
      WAIT_CRD: if (pcrd_grant)     st <= SENT;
      default:  st <= IDLE;
    endcase
  assign send_req = (st == IDLE && req_pending) || (st == WAIT_CRD && pcrd_grant);
  assign done     = (st == SENT && accepted);
endmodule

And in VHDL:

-- Representative REQ retry machine (VHDL).
library ieee;
use ieee.std_logic_1164.all;
 
entity req_retry_fsm is
  port (
    clk, rst_n   : in  std_logic;
    req_pending  : in  std_logic;
    accepted     : in  std_logic;
    retry_ack    : in  std_logic;
    pcrd_grant   : in  std_logic;
    send_req     : out std_logic;
    done         : out std_logic
  );
end entity;
 
architecture rtl of req_retry_fsm is
  type state_t is (IDLE, SENT, WAIT_CRD);
  signal st : state_t := IDLE;
begin
  process(clk, rst_n)
  begin
    if rst_n = '0' then
      st <= IDLE;
    elsif rising_edge(clk) then
      case st is
        when IDLE     => if req_pending = '1' then st <= SENT; end if;
        when SENT     => if accepted = '1' then st <= IDLE;
                         elsif retry_ack = '1' then st <= WAIT_CRD; end if;
        when WAIT_CRD => if pcrd_grant = '1' then st <= SENT; end if;
      end case;
    end if;
  end process;
 
  send_req <= '1' when (st = IDLE and req_pending = '1') or (st = WAIT_CRD and pcrd_grant = '1') else '0';
  done     <= '1' when (st = SENT and accepted = '1') else '0';
end architecture;

All three re-send only on a PCrdGrant after a RetryAck — never blindly. That one rule is what turns a busy Home Node into a guaranteed-eventually-accepted request instead of a livelock.

12. Verification View — no retry without a credit grant

Two properties: after a RetryAck the machine does not send until a grant, and it completes only on acceptance.

// Bind to req_retry_fsm.
localparam logic [1:0] SENT = 2'd1, WAIT_CRD = 2'd2;
 
// 1. While waiting for a credit, never send a REQ except on the grant.
property p_no_send_without_credit;
  @(posedge clk) disable iff (!rst_n)
    (st == WAIT_CRD && !pcrd_grant) |-> !send_req;
endproperty
assert property (p_no_send_without_credit);
 
// 2. Completion happens only when a sent request is accepted.
property p_done_on_accept;
  @(posedge clk) disable iff (!rst_n) done |-> (st == SENT && accepted);
endproperty
assert property (p_done_on_accept);

The system point, beyond the two checks:

The retry protocol exists to guarantee forward progress under contention. A Home Node returns RetryAck when it is out of resources; a PCrdGrant is its promise that a resource is now reserved for this requester. Re-sending only on the grant means the re-send is guaranteed to be accepted — no wasted retries, no livelock. Re-sending without the grant floods a full Home Node with requests it must keep rejecting, and under load the system spins without progressing. The credit is not a formality; it is the forward-progress guarantee.

• What it proves: no re-send without a credit; completion only on acceptance.
• What it does not prove: end-to-end fairness across requesters (an arbitration property).
• Bug signature: repeated RetryAcks with no completion — a retry storm (the DebugLab).

13. Testbench — send, retry, grant, accept

Runs a request that is retried once, then granted a credit and accepted.

module tb_req_retry_fsm;
  logic clk = 0, rst_n, req_pending, accepted, retry_ack, pcrd_grant, send_req, done;
  int sends = 0, errors = 0;
 
  req_retry_fsm dut (.*);
  always #5 clk = ~clk;
  always @(posedge clk) if (send_req) sends++;
 
  initial begin
    rst_n = 0; req_pending = 0; accepted = 0; retry_ack = 0; pcrd_grant = 0;
    @(posedge clk); rst_n = 1;
 
    req_pending = 1; @(posedge clk); #1; req_pending = 0;   // first try (send in IDLE)
    // Home Node is full -> RetryAck.
    retry_ack = 1; @(posedge clk); #1; retry_ack = 0;
    // Wait a couple cycles with no credit -> must NOT re-send.
    @(posedge clk); @(posedge clk); #1;
    if (sends != 1) begin errors++; $display("FAIL re-sent without credit (sends=%0d)", sends); end
    else $display("PASS no re-send while awaiting credit");
    // Credit granted -> re-send.
    pcrd_grant = 1; @(posedge clk); #1; pcrd_grant = 0;
    // Accepted on the retry.
    accepted = 1; @(posedge clk); #1;
    if (!done) begin errors++; $display("FAIL retry not accepted"); end
    else $display("PASS retry accepted after credit (sends=%0d)", sends);
    accepted = 0;
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

PASS no re-send while awaiting credit
PASS retry accepted after credit (sends=2)

14. DebugLab — retrying without waiting for the credit

cyc  REQ         RSP        note
 0   Read try1   -
 1   -           RetryAck   Home Node full
 2   Read try2   -          re-sent immediately (no credit!)
 3   -           RetryAck   still full -> rejected again
 4   Read try3   -          re-sent again... storm

15. Common Mistakes

• Retrying without a credit. Assumption: re-send on RetryAck. Bug: retry storm, livelock (the DebugLab). Prevention: wait for PCrdGrant, then re-send.
• Misreading the opcode. Assumption: one read fits all. Bug: wrong coherency behavior. Prevention: the opcode selects the exact transaction — read it precisely.
• Wrong TgtID. Assumption: any Home Node works. Bug: request to the wrong home. Prevention: compute TgtID from the address via the SAM.
• Inconsistent SAM. Assumption: each requester maps addresses itself. Bug: split coherence (Chapter 3.8). Prevention: one global, identical map.
• Ignoring Order / QoS. Assumption: all requests are equal. Bug: ordering violations or priority inversion. Prevention: honor the Order and QoS fields.
• Forgetting Size / attributes. Assumption: every request is a full line. Bug: wrong transfer size or cacheability. Prevention: read Size and MemAttr.

16. Engineering Checklist

• Build the REQ packet with opcode, addr, SrcID, TgtID, TxnID, Size and attributes.
• Select the transaction by opcode (read / write / CMO / atomic).
• Compute TgtID from the address via the system address map.
• On RetryAck, wait for PCrdGrant before re-sending — never re-send blindly.
• Honor Order and QoS on each request.
• Correlate responses by TxnID (and SrcID at the receiver, Chapter 4.7).

17. Key Takeaways

• The REQ channel starts every transaction with a self-describing request packet.
• Its fields: opcode (what), addr (where), SrcID/TgtID (who/home), TxnID, Size, Order, QoS.
• It carries reads, writes, cache-maintenance, and atomics, selected by the opcode.
• Requests are source-routed to their home — TgtID from the system address map, routed by the fabric.
• The credit-based retry (RetryAck → PCrdGrant → re-send) guarantees forward progress; never retry without the grant.
• REQ is the most consequential channel — every transaction begins here; the model here is representative.

18. Quick Revision

The REQ channel. Where every transaction begins. The request packet carries opcode (transaction type), addr, SrcID (requester), TgtID (Home Node), TxnID (correlate), Size, and attributes (Order, QoS, MemAttr, Excl…). Opcode families: reads (ReadShared/Unique/Once/NoSnp), writes (WriteBackFull, WriteUnique, WriteNoSnp), cache-maintenance (CleanShared, CleanInvalid, MakeUnique, Evict), atomics (AtomicStore/Load/Swap/Compare). Source-to-home routing: the requester computes TgtID from the address via the system address map and the fabric routes there. Retry: a full Home Node returns RetryAck; the requester waits for a PCrdGrant and only then re-sends — a credit-based retry guaranteeing forward progress. Never re-send without the grant, or a retry storm livelocks. Representative model; 6.3 covers RSP.

Coming Next

Chapter 6.3 — The Response Channel (RSP). REQ starts a transaction; RSP carries the acknowledgements that steer and finish it. The next chapter details the response channel: its packet structure, the response types it carries — Comp, CompAck, SnpResp, DBIDResp, RetryAck, PCrdGrant — and the completion semantics that tell a requester when a transaction is truly done. Having met RetryAck and PCrdGrant here as REQ's retry partners, you will see them next as members of the full RSP family.