Heterogeneous Compute

Chapter 18.1 treated one accelerator chiplet; 18.2 gave several of them a fabric to share. Both assumed the chiplets were alike. This chapter removes that assumption.

1. The One-Sentence Model

Heterogeneous compute is the controlled transfer of ownership between agents with different execution models. The agents differ in latency tolerance, access shape, synchronisation style, work granularity and completion semantics — and the integration architecture exists to preserve one semantic view across all of it.

2. What This Chapter Owns

What is new here, and none of it exists in 18.1 or 18.2:

Both earlier chapters had one execution model replicated. Here there are three, and §4 shows they differ in six dimensions that all matter architecturally. Every failure below needs unlike agents to exist.

Command completion is not data visibility (§9–§12). The single most common heterogeneous integration bug, and it produces a stale read with a perfectly clean transport.

Ownership moves between agents, and one job has one owner (§13–§16). Two owners writing one state register lose a transition to a last-write-wins race.

Locality beats speed (§35–§38). Choosing the fastest idle agent when the data lives beside a slower one makes the system slower, and the arithmetic says by how much.

And a failure in one agent must not fail the package (§30–§34) — which requires a dependency bitmap, generation identities, and a query path rather than a re-dispatch.

3. Sourcing

4. Three Execution Models, Six Differences

They are not three speeds. They are three shapes.

Three architectural consequences, and each is a section below.

Row 1 makes a shared queue actively harmful. A CPU command behind a GPU burst waits for the burst — and the CPU is the agent that cannot tolerate waiting. §19 through §22.

Row 3 means their synchronisation points do not naturally align, so the package needs an explicit cross-agent barrier rather than each agent's own (§25).

And row 6 is the deepest one. A CPU's store is visible by the memory model's rules; a GPU kernel's completion is an explicit event that says nothing by itself about visibility. Merging those two notions is §10, and it is the chapter's flagship failure.

5. The Package

Read the two separate return paths. A completion says the agent finished. A visibility confirmation says the consumer would now read the right value. They arrive at different times, they are produced by different mechanisms, and §10 is the design that has only one of them.

6. Four Paths, Four Meanings

The third and fourth are different questions with different answers at different times. Every other section in this chapter depends on that sentence being taken seriously.

7. The Heterogeneous Work Descriptor

// ILLUSTRATIVE. NOT a UCIe, CXL or CHI format, and no field corresponds to any
// specified field of any of them (Section 3).
typedef enum logic [1:0] {
  AGENT_CPU = 2'd0,
  AGENT_GPU = 2'd1,
  AGENT_AI  = 2'd2
} agent_type_e;
 
typedef struct packed {
  logic [JOB_ID_W-1:0]   job_id;
  logic [GEN_W-1:0]      generation;       // Section 33
  agent_type_e           producer;         // who last owned the data
  agent_type_e           consumer;         // who should own it next
  logic [ADDR_W-1:0]     data_base;
  logic [LEN_W-1:0]      length;
  logic [EPOCH_W-1:0]    ownership_epoch;  // which handoff produced this binding
  logic [DOM_W-1:0]      access_domain;    // Section 39
  logic [PHASE_W-1:0]    phase;            // which cross-agent barrier (Section 25)
  logic                  idempotent;       // from the OPERANDS, not the opcode (Section 34)
} heterogeneous_work_t;

Architecture. A descriptor that names both ends of a handoff, not just a destination. producer and consumer together are what let the visibility gate (§11) know whose writes must be visible to whom.

State. One register per pipeline stage plus a job-table entry (§13).

Cycle behaviour. Formed at the scheduler, held stable while offered, and nothing on the agent side recomputes any field.

Contract. The visibility gate reads producer; the admission check reads consumer and access_domain; the barrier reads phase; the recovery path reads generation and idempotent. Four consumers, one object.

Failure. Omitting idempotent and inferring it from the operation type — which gets the accumulate-into-a-shared-output case wrong in the silent direction (§34). Or omitting ownership_epoch, which leaves a late completion unable to prove which handoff it belongs to.

DV. Assert stability under stall; assert every field an agent uses was carried rather than derived.

8. Four Questions at Every Boundary

Take a pipeline: the CPU prepares data, the GPU processes it, the AI consumes the GPU's result, the CPU observes the final answer. At each arrow, four questions must have answers.

Two consequences.

Question 4 is the one with no obvious owner, which is why it gets skipped. Questions 1 to 3 are naturally answered by the scheduler and the descriptor. Question 4 is answered by the coherence protocol or by an explicit software ownership handoff (§17), and by nothing else.

And the answers differ per agent pair. A CPU-to-GPU handoff and a GPU-to-AI handoff may use entirely different mechanisms in the same package. A design with one universal handoff rule has assumed the agents are alike, which is the thing this chapter exists to deny.

9. Completion and Visibility Are Different Events

GPU kernel completes         ->  the agent has stopped working
    ...
writes drain from its buffers ->  the values have left the agent
    ...
the coherence protocol
  makes them observable       ->  a CONSUMER would now read the right value

The gap between the second and the fourth is where §10 lives. It is not a small gap, it is not constant, and it is not observable from the completion message.

10. Wrong Architecture — Completion Used as Visibility

// WRONG — the consumer is launched on the producer's completion.
always_ff @(posedge clk)
  if (gpu_job_complete[id])
    launch_consumer(job_q[id].consumer, job_q[id].data_base);   // ← too early

1. The GPU finishes its kernel and reports completion.
2. Some of its writes are still buffered, or are visible to it and not
   yet to other agents by the memory model's rules.
3. The AI agent is launched and reads the range.
4. -> it reads a MIXTURE: some new values, some stale.
5. It computes a confident, well-formed, entirely wrong result.

Four properties, and this is the chapter's flagship failure.

Transport is perfect. The descriptor crossed cleanly, the completion crossed cleanly, no CRC failed, nothing retried. Every link-level metric is healthy.

The corruption is partial and timing-dependent. Only the writes still in flight are missed, so the result is nearly right — which for a numerical workload can pass a tolerance check and be blamed on precision rather than on data integrity.

It is load-dependent in the worst direction. Under light load the writes drain before the consumer starts and everything works. It fails when the fabric is busy, which is the operating point the package exists for.

And the fix is not a delay. Inserting a fixed wait works until it does not — the gap depends on congestion, on the coherence protocol's own timing, and on how much the producer wrote. The fix is an explicit visibility condition (§11).

11. The Visibility Gate

// ILLUSTRATIVE. Three named terms, and the middle one is the whole section.
assign next_stage_admit =
      producer_complete            // the producer agent has stopped working
   && data_visibility_confirmed    // a consumer's read would now return the writes
   && destination_ready;           // the consumer has the resources (Section 23)

Architecture. A conjunction between a producer's completion and a consumer's launch. data_visibility_confirmed is deliberately abstract, because what satisfies it is a property of the memory and coherence architecture in use, not of this chapter.

State. None of its own — a conjunction over three facts owned by three mechanisms.

Cycle behaviour. Combinational into the consumer's launch, evaluated once and bound to the launched job rather than polled underneath work already started.

Contract. The consumer relies on reading the producer's values. That reliance is invisible at the consumer's interface — it has no signal saying "is my input ready" — which is why §12 asserts it.

Failure. §10. Also making data_visibility_confirmed a fixed delay, which is a guess that fails under congestion.

DV. §12, plus a memory model that predicts what the consumer should read (§41).

12. SVA — the Consumer Cannot Start Before Visibility

// MANDATORY. The property that makes Section 10 impossible.
property p_consumer_starts_after_visibility;
  @(posedge clk) disable iff (!rst_n)
    consumer_launch_fire |-> (producer_complete[launch_job]
                           && data_visibility_confirmed[launch_job]);
endproperty
a_consumer_starts_after_visibility:
  assert property (p_consumer_starts_after_visibility);
 
// Visibility is never asserted before the producer completes.
property p_visibility_after_completion;
  @(posedge clk) disable iff (!rst_n)
    data_visibility_confirmed[ID] |-> producer_complete[ID];
endproperty
a_visibility_after_completion: assert property (p_visibility_after_completion);
 
// The consumer reads what the producer wrote — the end-to-end check, using a
// verification memory model rather than the design's own belief.
property p_consumer_reads_producer_values(bit [ADDR_W-1:0] a);
  @(posedge clk) disable iff (!rst_n)
    (consumer_read_fire && (read_addr == a))
      |-> (read_data == tb_expected_value(a));
endproperty
a_consumer_reads_producer_values:
  assert property (p_consumer_reads_producer_values(ADDR_UT));

Architecture. Three properties: the gate holds, the ordering between the two events holds, and the values actually observed are the right ones.

Why the third is not redundant. The first two check the protocol; the third checks the effect. A design can satisfy both gates and still have a visibility mechanism that is itself wrong — and only an independent memory model catches that.

DV. The third needs a model of what each address should contain after each producer stage, which is §41's Layer 3. It is the only detector for §10 if the gate's implementation is broken rather than absent.

13. The Cross-Agent Job State

// ILLUSTRATIVE cross-agent job state. NOT a protocol FSM.
typedef enum logic [3:0] {
  JOB_FREE            = 4'd0,
  JOB_CREATED         = 4'd1,
  JOB_CPU_OWNED       = 4'd2,
  JOB_GPU_INFLIGHT    = 4'd3,
  JOB_GPU_DONE_WAIT_VIS = 4'd4,   // the state Section 10's design does not have
  JOB_AI_INFLIGHT     = 4'd5,
  JOB_RESULT_PENDING  = 4'd6,
  JOB_COMPLETE        = 4'd7,
  JOB_FAILED          = 4'd8
} job_state_e;
 
typedef struct packed {
  logic                     valid;
  job_state_e               state;
  agent_type_e              owner;            // exactly ONE at a time (Section 16)
  logic [GEN_W-1:0]         generation;
  logic [PHASE_W-1:0]       phase;
  logic [NUM_AGENTS-1:0]    dependency_pending;   // Section 29
  logic [ADDR_W-1:0]        data_base;
  logic [LEN_W-1:0]         length;
  logic                     idempotent;
  logic                     result_reserved;
} het_job_t;
 
het_job_t job_q [MAX_JOBS];

Architecture. Nine states, and JOB_GPU_DONE_WAIT_VIS is the one that matters: it exists precisely because completion and visibility are different events (§9). A design without that state has nowhere to be between them, which is why it launches early.

State. MAX_JOBS entries. owner is a field rather than an implication of state, because ownership must be checkable independently of the phase (§16).

Cycle behaviour. One next-state owner (§14). generation is written at creation and never elsewhere.

Contract. The scheduler reads owner and dependency_pending; the barrier reads phase; the recovery path reads generation and idempotent. Four consumers, one table.

Failure. §15 — two blocks writing state.

DV. §16's properties; cover every state and every legal transition.

14. One Next-State Owner

// ILLUSTRATIVE. ONE always_comb computes the next state; ONE always_ff commits
// it. Every event that can occur simultaneously is resolved explicitly, in
// priority order, in one place.
always_comb begin
  nxt = job_q[i].state;
  unique case (job_q[i].state)
    JOB_FREE:        if (create_fire[i])                     nxt = JOB_CREATED;
 
    JOB_CREATED:     if (cancel_fire[i])                     nxt = JOB_FAILED;
                     else if (cpu_accept[i])                 nxt = JOB_CPU_OWNED;
 
    JOB_CPU_OWNED:   if (fatal_error[i])                     nxt = JOB_FAILED;
                     else if (cancel_fire[i])                nxt = JOB_FAILED;
                     else if (gpu_admit[i])                  nxt = JOB_GPU_INFLIGHT;
 
    JOB_GPU_INFLIGHT:if (fatal_error[i])                     nxt = JOB_FAILED;
                     else if (gpu_complete[i])               nxt = JOB_GPU_DONE_WAIT_VIS;
 
    // The state that exists because completion is not visibility (Section 9).
    JOB_GPU_DONE_WAIT_VIS:
                     if (fatal_error[i])                     nxt = JOB_FAILED;
                     else if (visibility_confirmed[i] && ai_admit[i])
                                                             nxt = JOB_AI_INFLIGHT;
 
    JOB_AI_INFLIGHT: if (fatal_error[i])                     nxt = JOB_FAILED;
                     else if (ai_complete[i])                nxt = JOB_RESULT_PENDING;
 
    JOB_RESULT_PENDING:
                     if (result_consumed[i])                 nxt = JOB_COMPLETE;
 
    JOB_COMPLETE:    if (host_ack[i])                        nxt = JOB_FREE;
    JOB_FAILED:      if (host_ack[i])                        nxt = JOB_FREE;
    default:                                                 nxt = JOB_FAILED;
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) job_q[i].state <= JOB_FREE;
  else        job_q[i].state <= nxt;

Architecture. One unique case, one committer. The simultaneous events are resolved by the arm order, which is stated rather than emergent: a fatal error beats a completion in the same cycle; a cancellation beats an admission.

State. Four bits per job.

Cycle behaviour. Note the JOB_GPU_DONE_WAIT_VIS arm requires both visibility and the consumer's admission. A design that advances on visibility alone launches into a consumer with no resources (§23's family).

Contract. Every reader of state sees a value produced by one process. No block outside this one may write it, and §15 is what happens when that rule is not enforced by construction.

Failure. §15. Also checking gpu_complete before fatal_error, which lets a completion in the same cycle as a fatal error mask the error — and the error is then only discovered later, at a worse point.

DV. Assert the legal-transition function (12.4 §10); cover simultaneous completion-and-error and admission-and-cancel.

15. Wrong RTL — Two Owners for One Job State

// WRONG — the scheduler and the completion path each write job_state_q.
// In the scheduler block:
always_ff @(posedge clk)
  if (gpu_admit[i]) job_q[i].state <= JOB_GPU_INFLIGHT;
 
// In the completion block, in a different module:
always_ff @(posedge clk)
  if (cpu_complete[i]) job_q[i].state <= JOB_GPU_DONE_WAIT_VIS;

On a cycle where both fire, the last non-blocking assignment in the simulator's ordering wins — and the other transition is silently lost.

Four properties.

Simulation and synthesis may disagree, and in a design with the two writes in different modules, synthesis may not even produce a driver conflict warning if the writes are gated differently. The behaviour is then tool-dependent, which is the worst class of bug to carry into silicon.

The lost transition is not an error anywhere. The job simply sits in a state it should have left, and the symptom is a hang whose cause is a cycle thousands of cycles earlier.

And it is rare and load-dependent. The two events coincide only occasionally, so it appears as an intermittent hang and is often attributed to the fabric.

The rule is structural. One state, one always_comb next-state function, one always_ff committer — and every simultaneous event resolved by an explicit priority in that one place (§14). It is not a style preference; it is the only way the priority is reviewable.

16. SVA — Exactly One Owner Per Job

// MANDATORY. A live job has exactly one owning agent.
property p_one_owner_per_job;
  @(posedge clk) disable iff (!rst_n)
    job_q[ID].valid |-> $onehot(owner_onehot(job_q[ID].owner));
endproperty
a_one_owner_per_job: assert property (p_one_owner_per_job);
 
// Ownership changes only at a defined handoff, never spontaneously.
property p_ownership_changes_at_handoff_only;
  @(posedge clk) disable iff (!rst_n)
    $changed(job_q[ID].owner) |-> $past(handoff_fire[ID]);
endproperty
a_ownership_changes_at_handoff_only:
  assert property (p_ownership_changes_at_handoff_only);
 
// Only the owning agent may write the job's data range.
property p_only_owner_writes;
  @(posedge clk) disable iff (!rst_n)
    (agent_write_fire && in_range(write_addr, ID))
      |-> (write_agent == job_q[ID].owner);
endproperty
a_only_owner_writes: assert property (p_only_owner_writes);
 
// The state register has one driver — checked structurally, and asserted too.
property p_state_changes_only_from_next_state_fn;
  @(posedge clk) disable iff (!rst_n)
    $changed(job_q[ID].state) |-> (job_q[ID].state == $past(nxt_for(ID)));
endproperty
a_state_changes_only_from_next_state_fn:
  assert property (p_state_changes_only_from_next_state_fn);

Architecture. Four properties: one owner, ownership changes only at handoffs, only the owner writes, and the state comes only from the next-state function.

Why the fourth catches §15. With two writers, the committed state on a conflicting cycle will not equal what the single next-state function computed. The assertion fires on the exact cycle of the race, rather than on the hang thousands of cycles later.

Why the third is worth its cost. It is the effect of ownership, not the bookkeeping. A design can track ownership correctly and still let a non-owner write — through a stale descriptor, a debug path, or a second engine on the same die.

DV. All four always-on. Cover simultaneous handoff-and-completion, which is §15's precondition.

17. Shared Memory or Explicit Ownership — Both Work

Two consequences, and neither approach is universally better.

Explicit ownership is often cheaper for exactly the traffic this chapter describes — large buffers moving between phases, handed once. Paying coherence traffic per line for a buffer touched once by each agent is expensive.

But it moves a correctness obligation into software, where a missed step produces §18 with no hardware fault at all. A hardware ownership-context check catches the ordering mistake and none of the concurrent ones (17.3 §30's argument).

18. Wrong Design — Explicit Handoff With a Dirty Cache Retained

1. The CPU writes the buffer. Its cache holds the lines DIRTY.
2. Software "hands ownership" of the buffer to the AI agent — updating an
   ownership table, and nothing else.
3. The AI agent reads the buffer from memory.
4. Memory holds the OLD values; the newest are in the CPU's cache.
5. -> the AI computes from stale input, confidently and with no error.
6. Later the CPU evicts its dirty lines, overwriting whatever the AI wrote.

Four properties.

The ownership table is correct. It says the AI owns the buffer, and the AI does. The bug is that ownership was transferred without transferring the data.

Both directions fail. The AI reads stale input at step 5, and the CPU's later eviction destroys the AI's output at step 6. Two independent corruptions from one missing step.

And it is the same failure as 17.5 §16's migration copy. A copy — or a handoff — captures what memory holds; the newest value may be somewhere else.

The handoff protocol must include the data movement, not just the bookkeeping:

1. The producer ensures its caches hold no dirty copies of the range.
2. Ownership is transferred.
3. The consumer may read.
4. The producer must not access the range until ownership returns.

Steps 1 and 4 are the ones omitted, and a hardware ownership-context check catches violations of 4 but not of 1 — so step 1 must be verified by a memory model (§41).

19. Three Traffic Shapes, One Fabric

§4's row 2 becomes a queueing problem the moment the agents share anything.

ILLUSTRATIVE. One shared queue, GPU burst of 64 entries ahead of one CPU command.
 
  CPU command service time  = 64 × (GPU entry service) + its own
  If a GPU entry takes 8 cycles: 64 × 8 = 512 cycles of pure queueing
  The CPU's own service:      ~12 cycles
 
  -> the CPU command takes ~524 cycles instead of ~12, a 43× inflation,
     entirely from being behind traffic it has nothing to do with.

The agent least able to tolerate latency is the one a shared queue punishes most, because it offers the least traffic and therefore waits behind the most.

20. Per-Class Queues

// ILLUSTRATIVE. One queue per agent class, so a CPU command is never behind a
// GPU burst. The alternative — a shared pool with per-class reservations —
// trades utilisation for state and must reserve a non-zero minimum per class.
typedef struct packed {
  logic                  valid;
  logic [JOB_ID_W-1:0]   job_id;
  agent_type_e           consumer;
  logic [ADDR_W-1:0]     data_base;
} het_cmd_t;
 
het_cmd_t cmd_mem [NUM_CLASSES][CLASS_DEPTH];       // inferred RAM, not reset
logic [PTR_W-1:0] cmd_wr_q  [NUM_CLASSES];
logic [PTR_W-1:0] cmd_rd_q  [NUM_CLASSES];
logic [OCC_W-1:0] cmd_occ_q [NUM_CLASSES];
 
logic [NUM_CLASSES-1:0] class_request;
always_comb
  for (int c = 0; c < NUM_CLASSES; c++)
    class_request[c] = (cmd_occ_q[c] != '0);

Architecture. One queue per class, so each class presents its own head to the arbiter. This is 18.2 §14's virtual output queue argument applied to traffic classes rather than destinations, and it removes the same head-of-line blocking for the same reason.

State. NUM_CLASSES × CLASS_DEPTH entries plus three small registers per class. The payload is inferred RAM and deliberately not reset — occupancy starts at zero and gates every pop (17.4 §14).

Cycle behaviour. Push on admission into the class's own queue; pop on a grant that fired.

Contract. The arbiter reads class_request; admission reads cmd_occ_q for the command's own class, never an aggregate (17.4 §17).

Failure. Sizing all classes equally when their traffic shapes differ by orders of magnitude. The CPU class needs a shallow queue and low latency; the GPU class needs a deep one — equal sizing wastes storage on one and starves the other.

DV. Cover each class empty, mid and full; cover simultaneous push and pop per class.

21. Wrong Priority — GPU Traffic First

// WRONG at system scope. A GPU stream can monopolise the shared fabric.
assign grant = gpu_req ? GRANT_GPU : (ai_req ? GRANT_AI : GRANT_CPU);

1. A sustained GPU stream keeps gpu_req asserted.
2. CPU completions returning through the shared path are never granted.
3. The CPU cannot retire its jobs, so it holds the job-table entries and
   buffers that the GPU's next phase needs.
4. The GPU's next phase cannot be dispatched — no entries free.
5. The GPU stream eventually needs a result the CPU was to produce.
6. -> deadlock, or a tail latency that is functionally a hang.

Four properties.

It is safety-correct. Nothing is corrupted, nothing is duplicated, no assertion about ownership or visibility fires. A functional regression passes.

And it starves exactly the agent that cannot tolerate it (§19). The CPU's tail latency explodes first, long before the deadlock — so the symptom is "the system feels unresponsive under GPU load", which is easy to dismiss as expected.

The class that is starved is the one that releases resources — completions (18.2 §31) — which is why it becomes a deadlock and not merely slow.

And unlike a single-link strict-priority arbiter, it does not self-resolve. The starved traffic is what would free the resources the favoured traffic needs.

22. A Weighted Arbiter With a Progress Reserve

// ILLUSTRATIVE. Weights for performance, plus a bounded override for progress.
// Fairness state advances ONLY on an actual transfer.
logic [NUM_CLASSES-1:0]  req;
logic [NUM_CLASSES-1:0]  grant;
logic [W_W-1:0]          weight     [NUM_CLASSES];   // configured
logic [W_W-1:0]          deficit_q  [NUM_CLASSES];   // weighted-deficit credit
logic [AGE_W-1:0]        wait_age_q [NUM_CLASSES];   // saturating
logic [NUM_CLASSES-1:0]  is_progress_class;          // completions, sync
 
wire [NUM_CLASSES-1:0] aged_out;
generate for (genvar c = 0; c < NUM_CLASSES; c++)
  assign aged_out[c] = req[c] && (wait_age_q[c] >= class_bound(c));
endgenerate
 
always_comb begin
  grant = '0;
  if (|(aged_out & is_progress_class))  grant = lowest_set(aged_out & is_progress_class);
  else if (|aged_out)                   grant = lowest_set(aged_out);
  else if (|(req & has_deficit))        grant = lowest_set(req & has_deficit);
  else if (|req)                        grant = lowest_set(req);   // all deficits spent
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    for (int c = 0; c < NUM_CLASSES; c++) begin
      deficit_q[c]  <= weight[c];
      wait_age_q[c] <= '0;
    end
  end else begin
    // Replenish when every requesting class has spent its deficit.
    if (all_deficits_spent)
      for (int c = 0; c < NUM_CLASSES; c++) deficit_q[c] <= weight[c];
    else if (|grant && xfer_fire)
      deficit_q[onehot_to_index(grant)] <= deficit_q[onehot_to_index(grant)] - 1'b1;
 
    for (int c = 0; c < NUM_CLASSES; c++) begin
      if (req[c] && !(grant[c] && xfer_fire))
        wait_age_q[c] <= (wait_age_q[c] == AGE_MAX) ? AGE_MAX : wait_age_q[c] + 1'b1;
      else if (grant[c] && xfer_fire)
        wait_age_q[c] <= '0;
    end
  end

Architecture. Weighted deficits for performance proportioning, plus a per-class age bound for liveness, with progress classes checked first among the aged-out. Two mechanisms because §23 establishes they are two properties.

State. One deficit and one saturating age per class. class_bound(c) differs per class — the CPU's bound is tight because §19 says it must be; the GPU's is loose because it does not care.

Cycle behaviour. grant is combinational. Both the deficit decrement and the age reset are qualified by xfer_fire — a grant the downstream refused has served nobody, which is 18.2 §18's rule and the sixth appearance of it in this curriculum.

Contract. The CPU relies on a tight bound; the GPU relies on a large weighted share; the completion class relies on the progress override. Three different guarantees from one block, none visible at its interface — which is why §24 asserts all three.

Failure. §21. Also setting every class_bound to the same value, which gives the CPU the GPU's tolerance and defeats the point of per-class bounds.

DV. Saturate all classes and measure worst-case per-class wait against each class_bound.

23. QoS Priority Is Not a Progress Guarantee

A class can have the highest priority and still starve, if a higher-priority class is continuously requesting. Priority is a comparison; a bound is a promise — and §21's deadlock is a priority scheme with no bound anywhere in it.

24. SVA — Bounded Service, Per Class

// LIVENESS, bounded, per class, with assumptions stated (15.2 §36).
//
//   A1: the downstream eventually accepts
//   A2: a requesting class keeps requesting until served
//   A3: recovery terminates (14.2 §30)
assume property (@(posedge clk) disable iff (!rst_n)
  (|grant && !recovery_active) |-> ##[1:XFER_BOUND] xfer_fire);
assume property (@(posedge clk) disable iff (!rst_n)
  (req[CLS] && !xfer_fire) |=> req[CLS]);
 
// Each class is served within ITS OWN bound — not one shared bound.
property p_class_served_within_its_bound(int c);
  @(posedge clk) disable iff (!rst_n)
    req[c] |-> ##[1:CLASS_BOUND(c)] (grant[c] && xfer_fire);
endproperty
a_cpu_served:  assert property (p_class_served_within_its_bound(CLASS_CPU));
a_gpu_served:  assert property (p_class_served_within_its_bound(CLASS_GPU));
a_ai_served:   assert property (p_class_served_within_its_bound(CLASS_AI));
 
// The progress class has the tightest bound of all.
property p_progress_class_served;
  @(posedge clk) disable iff (!rst_n)
    (|(req & is_progress_class))
      |-> ##[1:PROGRESS_BOUND] (|(grant & is_progress_class) && xfer_fire);
endproperty
a_progress_class_served: assert property (p_progress_class_served);

Architecture. One bounded property per class plus one for the progress class.

Why per-class bounds rather than one. A single generous bound would pass for a design that starves the CPU for thousands of cycles — within the bound, and useless. The CPU's bound encodes its latency requirement; the GPU's encodes only that it must not starve. Writing them separately is what makes the requirement reviewable.

DV. Prove all four. Then set every class_bound equal and confirm the CPU property fails, verifying that the per-class distinction is load-bearing.

25. Cross-Agent Barriers

§4's row 3 says the agents synchronise differently. A cross-agent phase barrier is therefore an explicit package-level structure, not something any single agent provides.

The failure mode is not a hang — it is a phase advancing while a participant is still writing, which corrupts the next phase's inputs silently. 18.1 §32 makes the general argument; what heterogeneity adds is that the participants complete at wildly different times, so the barrier is held by the slowest agent for most of every phase.

26. Barrier RTL — a Bitmap Across Unlike Agents

// ILLUSTRATIVE. A BITMAP, for the fifth time in this curriculum — and here the
// participants are unlike, so their completion times differ by orders of
// magnitude and duplicates are more likely, not less.
logic [NUM_AGENTS-1:0] phase_pending_q;
logic [PHASE_W-1:0]    phase_q;
logic [AGE_W-1:0]      agent_wait_q [NUM_AGENTS];   // who is the critical path?
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    phase_pending_q <= '0;
    phase_q         <= '0;
  end else if (phase_start) begin
    phase_pending_q <= participants_mask;
    phase_q         <= phase_q + 1'b1;
    for (int a = 0; a < NUM_AGENTS; a++) agent_wait_q[a] <= '0;
  end else begin
    // GUARDED clear, and the phase must match — a completion from a previous
    // phase, arriving late after a recovery, must not clear a current bit.
    if (phase_complete_valid
        && (phase_complete_phase == phase_q)
        && phase_pending_q[phase_complete_agent])
      phase_pending_q[phase_complete_agent] <= 1'b0;
 
    // Diagnostic: how long has each still-pending agent been holding the phase?
    for (int a = 0; a < NUM_AGENTS; a++)
      if (phase_pending_q[a])
        agent_wait_q[a] <= (agent_wait_q[a] == AGE_MAX) ? AGE_MAX : agent_wait_q[a] + 1'b1;
  end
 
assign barrier_satisfied = (phase_pending_q == '0);

Architecture. One bit per agent plus a per-agent wait counter. The wait counter is the heterogeneous addition — with unlike agents, which agent is the critical path varies by phase, and without measuring it the partitioning cannot be tuned (§40).

State. NUM_AGENTS bits, a phase counter, and NUM_AGENTS saturating ages.

Cycle behaviour. Set as a whole at phase_start; cleared one bit at a time under a guard and a phase match. A completion for a previous phase is ignored entirely.

Contract. Everything downstream waits on barrier_satisfied. A false assertion corrupts the next phase's inputs, and there is no recovery — the corrupted values are consumed as legitimate data.

Failure. §27. Also omitting the phase check, which lets a stale completion satisfy a current barrier (18.1 §42's second symptom).

DV. §28's properties; cover a duplicate completion and a stale-phase completion, both injected.

27. Wrong RTL — a Scalar Barrier Counter

// WRONG — a counter cannot distinguish a new completion from a repeat.
always_ff @(posedge clk)
  if (phase_start)                pending_cnt_q <= NUM_AGENTS[CNT_W-1:0];
  else if (phase_complete_valid)  pending_cnt_q <= pending_cnt_q - 1'b1;

Illustrative, three agents, with the GPU's completion transported twice.

Four properties.

The next phase reads a mixture of the CPU's new and old outputs. Silent corruption whose magnitude depends on how far the CPU had progressed.

The duplicate is not a bug anywhere else — it is a transport retry doing exactly what 14.3 specifies. The counter is the only incorrect component.

And heterogeneity makes it more likely, not less. The agents complete at very different times, so a phase spends most of its duration with one or two bits still set — a wider window in which a duplicate can land and matter.

The bitmap fix is free: NUM_AGENTS bits instead of log2, plus one term in the clear condition. This is the fifth appearance of this argument (16.2 §21, 18.1 §35, 18.2 §26, 18.2 §52, and here) — it recurs because a counter looks like the obvious way to track "how many are left".

28. SVA — Barrier Safety

// MANDATORY. Only a pending participant can clear its bit.
property p_only_pending_clears;
  @(posedge clk) disable iff (!rst_n)
    (phase_complete_valid && !phase_pending_q[phase_complete_agent])
      |=> $stable(phase_pending_q);
endproperty
a_only_pending_clears: assert property (p_only_pending_clears);
 
// A completion for a non-current phase changes nothing.
property p_stale_phase_ignored;
  @(posedge clk) disable iff (!rst_n)
    (phase_complete_valid && (phase_complete_phase != phase_q))
      |=> $stable(phase_pending_q);
endproperty
a_stale_phase_ignored: assert property (p_stale_phase_ignored);
 
// No participant is still executing when the next phase begins — the EFFECT,
// checked against a model of what each agent is actually doing.
property p_no_execution_across_barrier;
  @(posedge clk) disable iff (!rst_n)
    phase_start |-> (tb_agents_still_running_in($past(phase_q)) == '0);
endproperty
a_no_execution_across_barrier:
  assert property (p_no_execution_across_barrier);

Architecture. Three properties: the guard, the phase check, and the end-to-end guarantee.

Why the third is not implied by the first two. They check the bitmap; the third checks reality. A design whose completion signal is asserted before the agent's last write drains satisfies both and violates the third — which is §10's visibility problem appearing inside the barrier.

DV. The third needs a verification model of each agent's actual activity, which is §41's Layer 1.

29. Dependency Bitmaps

// ILLUSTRATIVE. A job advances only when every predecessor it depends on has
// resolved — which is what lets ONE agent's failure not fail the package (§30).
logic [NUM_AGENTS-1:0] dependency_pending_q [MAX_JOBS];
 
always_ff @(posedge clk)
  if (create_fire[i])
    dependency_pending_q[i] <= required_predecessors[i];
  else if (predecessor_resolved_valid
           && (resolved_job == i[JOB_ID_W-1:0])
           && dependency_pending_q[i][resolved_agent])          // GUARDED
    dependency_pending_q[i][resolved_agent] <= 1'b0;
 
assign job_dependencies_met = (dependency_pending_q[i] == '0);

Architecture. Per-job, per-agent dependency tracking. The distinction from a barrier is scope: a barrier is per phase across all participants; this is per job, so two independent jobs can proceed while a third waits.

State. MAX_JOBS × NUM_AGENTS bits — small, and it is what makes partial progress possible.

Cycle behaviour. Set at creation from the job's declared predecessors; cleared under a guard, so a duplicate resolution is idempotent.

Contract. The scheduler dispatches only when job_dependencies_met. A design without this must use a global barrier for everything, which serialises independent work — 18.2 §40's global-ordering mistake at job scope.

Failure. Building the mask from the agent set rather than the actual predecessors, which makes every job depend on every agent and reduces the bitmap to a global barrier.

DV. Cover jobs with zero, one and several dependencies; cover a duplicate resolution.

30. One Agent's Failure Is Not the Package's

A UCIe recovery on the GPU's link, or a fault in the GPU itself, affects the GPU's path.

The dependency bitmap is what makes this expressible. Without it, a design must either fail everything or continue everything — and both are wrong.

31. Recovery Semantics

1. The GPU accepts a job and begins executing.
2. UCIe on its path enters recovery.
3. THE GPU KEEPS COMPUTING. It has its operands; the link is irrelevant
   to its progress.
4. It completes. The completion cannot be delivered until the link returns.
5. The scheduler's timeout expires somewhere between steps 2 and 4.

A timeout tells the scheduler that no completion arrived. It never says the job did not execute (12.4 §16).

32. Wrong Recovery — Blind Re-Dispatch

// WRONG — a timeout is treated as evidence the job did not run.
always_ff @(posedge clk)
  if (job_q[id].valid && (age_q[id] >= JOB_TIMEOUT))
    redispatch(id);            // ← it may already have executed. Once.

Illustrative, with a job that accumulates into a shared output range.

Four properties.

Every component behaved correctly. The GPU executed the job it was given, twice, because it was given it twice.

And in a heterogeneous pipeline the error propagates immediately. The AI stage consumes V + 2Δ as its input — so the wrong value is baked into everything downstream before anyone notices.

A second symptom appears in the barrier. The re-dispatched job may complete in a later phase, so its completion carries a stale phase and is correctly rejected (§26) — leaving the current phase's bit set and the barrier hung. One bug, two symptoms, in different runs.

The correct response is to query, not re-dispatch (§34), using {job_id, generation} — and re-dispatch only if the agent confirms it never accepted the job.

33. Generation Identities

// ILLUSTRATIVE. Identity plus generation — 12.2 §24's quarantine, applied to a
// cross-agent semantic job.
logic [GEN_W-1:0] job_gen_q [MAX_JOBS];
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)
    for (int i = 0; i < MAX_JOBS; i++) job_gen_q[i] <= '0;
  else if (create_fire)
    job_gen_q[create_job_id] <= job_gen_q[create_job_id] + 1'b1;

Architecture. A counter per identity, incremented at creation. {job_id, generation} is unique over a far longer window than job_id alone.

State. One counter per identity. GEN_W is derived from the maximum time a completion can be delayed — which after a recovery can be very long — not chosen for convenience.

Contract. The completion matcher compares both. A design that carries the generation and does not check it has paid the cost and kept none of the benefit.

Failure. A one-bit toggle, which fails on the third use. Or resetting generations on a recovery, which destroys the history exactly when stale completions are most likely.

DV. Cover a completion arriving with a stale generation and confirm it is reported as an orphan.

34. SVA — Exactly Once, and Survival

// MANDATORY. Semantic execution at most once, per agent, for a non-idempotent
// job. Reference-model property — "this is the same work" is testbench knowledge.
int unsigned tb_executions [int][int];      // [job key][agent]
 
property p_at_most_one_execution(int unsigned key, int agent);
  @(posedge clk) disable iff (!rst_n)
    (tb_executions[key][agent] <= 1);
endproperty
a_at_most_one_execution: assert property (p_at_most_one_execution(KEY_UT, AGENT_UT));
 
// A completion must name a live job AND its current generation.
property p_completion_matches_generation;
  @(posedge clk) disable iff (!rst_n)
    completion_valid |-> (job_q[comp_id].valid
                       && (job_q[comp_id].generation == comp_gen));
endproperty
a_completion_matches_generation:
  assert property (p_completion_matches_generation);
 
// Semantic job state survives a transport recovery.
property p_job_survives_recovery;
  @(posedge clk) disable iff (!rst_n)
    ucie_recovery_event |=> ($stable(job_q[ID].valid)
                          && $stable(job_q[ID].owner)
                          && $stable(job_q[ID].generation)
                          && $stable(job_q[ID].dependency_pending));
endproperty
a_job_survives_recovery: assert property (p_job_survives_recovery);
 
// A failure on one agent's path does not retire an independent job.
property p_independent_job_unaffected;
  @(posedge clk) disable iff (!rst_n)
    (agent_path_fault[FAULTY] && !job_q[ID].dependency_pending[FAULTY])
      |=> $stable(job_q[ID].valid);
endproperty
a_independent_job_unaffected: assert property (p_independent_job_unaffected);

Architecture. Four properties: exactly-once, generation matching, recovery survival, and fault independence.

Why the fourth is a heterogeneous-specific property. In a homogeneous system every agent is doing the same kind of work, so a blast-radius question rarely arises. Here the CPU's jobs are genuinely independent of the GPU's path, and asserting that independence is what proves §30's design is real rather than aspirational.

DV. The first needs a recovery, a timeout and a re-dispatch in sequence (§42). The fourth needs a fault injected on a path some jobs depend on and others do not.

35. Locality — Not All Accessible Memory Is Equal

An idle agent is not free if the data has to move to it. §36 is the scheduler that forgets this, and §37 quantifies when it is wrong.

36. Wrong Scheduler — the Fastest Idle Agent

// WRONG — pick the fastest agent that is idle, regardless of where the data is.
always_comb begin
  best = AGENT_CPU;
  for (int a = 0; a < NUM_AGENTS; a++)
    if (agent_idle[a] && (agent_speed[a] > agent_speed[best])) best = a[1:0];
end

Illustrative arithmetic. A job whose operands are 16 MiB, currently resident behind the GPU's memory.

OPTION A — run it on the GPU (data is local)
  compute time      = 2000 cycles
  data move         = 0
  total             = 2000 cycles
 
OPTION B — run it on the idle, faster AI engine
  compute time      = 1200 cycles          (illustratively 1.67× faster)
  data move         = 16 MiB across the package
                    = illustratively 3500 cycles
  total             = 4700 cycles
 
  -> the "faster" agent is 2.35× SLOWER for this job.

Three readings.

The compute saving is 800 cycles and the movement cost is 3500 — a factor of more than four. The scheduler optimised the smaller term.

And it makes the system worse twice. The 16 MiB also consumes fabric bandwidth that other jobs needed, so the cost is not confined to this job (18.2 §29's hotspot, created by a scheduling decision).

The break-even is computable and worth stating. Moving is worth it when compute_saving > move_cost — here, only if the AI engine were more than about 4.4× faster, which is a very different threshold from "faster".

37. A Cost Model

job_cost(agent) =   compute_time(agent, job)         [cycles]
                  + data_move_time(agent, job)       [cycles]
                  + synchronisation_time(agent, job) [cycles]
                  + expected_queue_time(agent)       [cycles]

Worked, with §36's illustrative numbers plus queueing and synchronisation:

The GPU still wins by 1800 cycles, despite being busy and slower per unit of work.

38. A Locality-Aware Scheduler

// ILLUSTRATIVE. Eligibility first, then a cost comparison. Deliberately simple:
// the point is WHICH signals participate, not the sophistication of the policy.
logic [NUM_AGENTS-1:0] eligible;
logic [COST_W-1:0]     est_cost [NUM_AGENTS];
 
always_comb
  for (int a = 0; a < NUM_AGENTS; a++)
    eligible[a] = agent_up[a]                       // not failed
               && agent_can_run[a][job_opcode]      // capability
               && agent_has_resources[a]            // Section 23's admission
               && domain_permits[a][job_domain];    // Section 39
 
always_comb
  for (int a = 0; a < NUM_AGENTS; a++)
    est_cost[a] = compute_est_q[a]
                + (data_local[a] ? '0 : move_cost_q[a])     // Section 36
                + sync_est_q[a]
                + (queue_occ_q[a] * service_est_q[a]);      // expected queue time
 
always_comb begin
  best = NO_AGENT;
  for (int a = 0; a < NUM_AGENTS; a++)
    if (eligible[a] && ((best == NO_AGENT) || (est_cost[a] < est_cost[best])))
      best = a[AG_W-1:0];
end

Architecture. Eligibility as a hard gate, then cost as a soft comparison. Separating them matters: an ineligible agent must never be chosen regardless of cost, and a cheap agent that cannot run the job is not a candidate.

State. Four per-agent estimates plus a queue occupancy. service_est_q converts occupancy into time — which is 18.1 §29's lesson: queue depth alone inverts precisely when it matters.

Cycle behaviour. Combinational, evaluated at dispatch. data_local[a] is a per-agent, per-job fact the scheduler must be told — it cannot be inferred from anything local.

Contract. The dispatcher trusts eligible. A stale agent_up sends work to a failed agent; a stale domain_permits is an isolation hole (§39).

Failure. §36 — omitting the movement term. Also omitting service_est_q and comparing raw occupancy across unlike agents, which compares a CPU's queue of 4 small commands against a GPU's queue of 4 large ones as if they were equal.

DV. Inject a job whose data is local to a busy agent and confirm the busy agent is chosen; inject a failed agent and confirm it is never chosen.

39. A Valid Job Is Not Blanket Authorisation

Different agents may run work in different access domains.

// ILLUSTRATIVE. Two independent decisions, both required — 17.5 §26's rule at
// agent scope.
assign job_admit =
      job_valid                                        // the descriptor is well-formed
   && domain_permits[target_agent][job_domain]         // WHETHER, owned elsewhere
   && (job_perm_epoch == domain_epoch_q[job_domain]);  // and not stale

A well-formed job descriptor names a range. It does not establish that this agent may access that range — and a design that treats descriptor validity as authorisation grants every agent everything any job ever named.

Two consequences, stated concisely because 17.5 §25 covers the general case.

The epoch term is what makes revocation enforceable. Without it, a permission granted under one configuration remains indistinguishable from a current one.

And the check belongs at the resource, not only at the scheduler. A scheduler-side check is an optimisation; the agent or the memory side is where a stale or malformed descriptor must be refused.

40. Instrumentation

// Diagnostic only. PER AGENT, because the agents are unlike and an aggregate
// describes none of them.
logic [63:0] jobs_accepted_q  [NUM_AGENTS];
logic [63:0] jobs_completed_q [NUM_AGENTS];
logic [63:0] queue_stall_q    [NUM_AGENTS];   // work offered, no queue space
logic [63:0] memory_stall_q   [NUM_AGENTS];   // waiting on memory
logic [63:0] fabric_stall_q   [NUM_AGENTS];   // waiting on the fabric
logic [63:0] sync_stall_q     [NUM_AGENTS];   // waiting at a barrier  <- Section 25
logic [63:0] vis_stall_q      [NUM_AGENTS];   // waiting for visibility <- Section 11
logic [63:0] barrier_critical_q[NUM_AGENTS];  // phases where this agent was last
logic [63:0] remote_data_bytes_q[NUM_AGENTS]; // bytes moved TO this agent  <- Section 36

Two properties.

barrier_critical_q is the heterogeneous-specific counter. With unlike agents, which one holds each phase varies — and knowing the distribution is what turns "rebalance the work" from a guess into a measurement.

And vis_stall_q makes the invisible visible. The gap between completion and visibility (§9) has no other observable. Without this counter, a design that spends 30% of its time there cannot know it.

41. The Heterogeneous Scoreboard

// Verification-only. FOUR models — the second is the one no other layer has.
class heterogeneous_scoreboard;
 
  // ---- Layer 1: JOB / OWNERSHIP model.
  typedef struct {
    int  job_id, generation;
    int  producer, consumer;
    int  current_owner;              // MUST be exactly one
    bit [63:0] dependency_pending;
    bit [63:0] agents_still_running; // TRUTH, for Section 28's third property
    int  executions [int];           // [agent] -> count, MUST be <= 1 if non-idempotent
    bit  idempotent;
    bit  retired;
  } job_model_t;
  job_model_t jobs [int];            // keyed by {job_id, generation}
 
  // ---- Layer 2: VISIBILITY model. WHEN a value became observable to whom.
  //      No other layer can see Section 10.
  typedef struct {
    bit [DATA_W-1:0] value;
    int              written_by;
    bit [63:0]       visible_to;     // one bit per agent
  } visibility_model_t;
  visibility_model_t vis [bit [ADDR_W-1:0]];
 
  // ---- Layer 3: MEMORY CONTENT model — what a correct read returns.
  bit [DATA_W-1:0] expected [bit [ADDR_W-1:0]];
 
  // ---- Layer 4: PERFORMANCE model, per agent.
  typedef struct {
    int accepted, completed;
    longint remote_bytes_moved;
    int     barrier_critical_count;
  } perf_model_t;
  perf_model_t perf [int];
 
  // ---- Catches Section 10 — and only this layer can.
  function void check_visibility(int agent, bit [ADDR_W-1:0] a, bit [DATA_W-1:0] got);
    if (!vis[a].visible_to[agent])
      $error("PREMATURE READ agent %0d addr %0h: value written by %0d is not yet "
           , "visible to this agent (Section 10)", agent, a, vis[a].written_by);
    else if (got !== expected[a])
      $error("STALE READ agent %0d addr %0h: got %0h expected %0h", agent, a, got, expected[a]);
  endfunction
 
  // ---- Catches Section 15.
  function void check_single_owner(int key);
    if ($countones(jobs[key].current_owner) != 1)
      $error("JOB %0d has %0d owners", key, $countones(jobs[key].current_owner));
  endfunction
 
  // ---- Catches Section 27.
  function void check_barrier(int phase, bit [63:0] still_running, bit advanced);
    if (advanced && (still_running != '0))
      $error("BARRIER ADVANCED EARLY phase %0d, still running %0h", phase, still_running);
  endfunction
 
  // ---- Catches Section 32.
  function void check_exactly_once(int key, int agent);
    if (!jobs[key].idempotent && (jobs[key].executions[agent] > 1))
      $error("JOB %0d executed %0d times on agent %0d (must be <= 1)",
             key, jobs[key].executions[agent], agent);
  endfunction
 
  // ---- Catches Section 36 — a PERFORMANCE bug no correctness model sees.
  function void note_unnecessary_movement(int key, int chosen, int local_agent);
    if ((chosen != local_agent) && (jobs[key].executions[chosen] > 0))
      $display("NOTE: job %0d ran on agent %0d while its data was local to %0d "
             , "(Section 36)", key, chosen, local_agent);
  endfunction
 
endclass

Architecture. Four models keyed by job-and-generation, by address, by address, and by agent.

Layer 2 is what makes this chapter's flagship bug detectable. It tracks when a value became visible to which agent — not what memory contains. A model of memory contents alone cannot see §10, because memory may well hold the right value while the consumer's read does not yet return it.

Layer 1's agents_still_running is truth, separate from the design's bitmap. §27's bug is precisely those two diverging, and a model that mirrors the design reproduces the bug and passes.

And Layer 4 exists because §36 is a correctness-clean performance disaster. No safety model detects it; only a note comparing the chosen agent against data locality does.

42. Flagship Trace — CPU → GPU → AI, With a Recovery

Illustrative. Cycle numbers illustrative.

Seven readings.

Cycles 52 to 55 are the whole chapter. The CPU completed at 52 and the handoff happened at 56 — after visibility was confirmed at 55. §10's design hands off at 52 and the GPU reads a mixture.

Cycle 91: the GPU keeps computing through the recovery. Its operands are local; the link is irrelevant to its progress.

Cycle 120: it completes during the recovery. The scheduler knows nothing.

Cycle 140: the timeout fires against a job that already succeeded, and cycle 141 is the decision that matters — query, not re-dispatch. §32's design executes the accumulate twice here.

Cycle 160: the link returns narrower. Latency and bandwidth change; not one semantic field does.

The barrier bits clear at 52, 163 and 210 — three agents, wildly different times, and the GPU held the phase for over 100 cycles. barrier_critical_q[GPU] increments (§40), which is what tells the next tuning pass where the imbalance is.

And nothing about the CPU's or AI's independent jobs was affected by the GPU recovery (§30), because their dependency bitmaps did not include the GPU.

43. Coverage

covergroup cg_heterogeneous @(posedge clk);
  option.per_instance = 1;
 
  // --- Agent participation (Sections 4, 8).
  cp_agents_active : coverpoint num_agents_with_work {
    bins one = {1}; bins two = {2}; bins all = {3};
  }
  cp_pipeline : coverpoint pipeline_shape {
    bins cpu_only = {0}; bins gpu_only = {1}; bins ai_only = {2};
    bins cpu_gpu  = {3}; bins gpu_ai   = {4}; bins cpu_gpu_ai = {5};
  }
 
  // --- Ownership and visibility (Sections 9-18).
  cp_handoff_model : coverpoint ownership_model {
    bins coherent_shared = {0}; bins explicit_handoff = {1};
  }
  cp_vis_gap : coverpoint completion_to_visibility_gap {
    bins zero = {0}; bins short = {[1:16]}; bins long = {[17:$]};   // Section 10
  }
  cp_dirty_handoff : coverpoint handoff_with_dirty_producer_cache;   // Section 18
  cp_state : coverpoint job_state_q_ut { bins each[] = {[0:8]}; }
  cp_simul_events : coverpoint simultaneous_job_events {
    bins none = {0}; bins complete_and_admit = {1}; bins complete_and_fault = {2};
  }
 
  // --- Classes and fairness (Sections 19-24).
  cp_class_mix : coverpoint classes_requesting {
    bins one = {1}; bins two = {2}; bins all = {3};
  }
  cp_cpu_wait : coverpoint cpu_class_wait {
    bins none = {0}; bins some = {[1:CPU_BOUND-1]}; bins at_bound = {CPU_BOUND};
  }
  cp_progress_override : coverpoint progress_override_fired;
 
  // --- Barriers and dependencies (Sections 25-29).
  cp_barrier : coverpoint barrier_event {
    bins none = {0}; bins satisfied = {1};
    bins duplicate_completion = {2};      // Section 27
    bins stale_phase = {3};
  }
  cp_barrier_critical : coverpoint barrier_critical_agent {
    bins cpu = {0}; bins gpu = {1}; bins ai = {2};   // all three must occur
  }
  cp_deps : coverpoint dependency_count {
    bins none = {0}; bins one = {1}; bins several = {[2:$]};
  }
 
  // --- Failure and recovery (Sections 30-34).
  cp_link_event : coverpoint link_event_during_job {
    bins none = {0};
    bins retry = {1};
    bins recovery_before_accept = {2};    // re-dispatch IS safe here
    bins recovery_after_accept  = {3};    // THE case — Section 32
    bins degraded_return = {4};
  }
  cp_idempotent : coverpoint job_idempotence { bins idem = {0}; bins non_idem = {1}; }
  cp_stale_gen : coverpoint stale_generation_completion;
  cp_independent : coverpoint fault_on_path_with_independent_jobs_live;  // Section 30
 
  // --- Scheduling (Sections 35-38).
  cp_locality : coverpoint dispatch_locality {
    bins data_local  = {0};
    bins data_remote_won  = {1};          // moving was actually correct
    bins data_remote_lost = {2};          // Section 36's mistake
  }
  cp_eligibility : coverpoint dispatch_refusal_reason {
    bins none = {0}; bins agent_down = {1}; bins capability = {2};
    bins resources = {3}; bins domain = {4};                       // Section 39
  }
 
  // --- Crosses that carry the information.
  x_recovery_idem  : cross cp_link_event, cp_idempotent;     // Section 32
  x_vis_pipeline   : cross cp_vis_gap, cp_pipeline;          // Section 10
  x_barrier_agents : cross cp_barrier, cp_agents_active;
  x_locality_cost  : cross cp_locality, cp_agents_active;    // Section 36
endcovergroup

Seven bins worth calling out:

cp_vis_gap.long crossed with cp_pipeline.cpu_gpu_ai. §10's precondition — a substantial gap between completion and visibility in a multi-stage pipeline. A zero-gap environment cannot distinguish a correct gate from an absent one.

cp_dirty_handoff. §18 — an explicit handoff performed with the producer's cache still dirty. Never occurs spontaneously.

cp_link_event.recovery_after_accept crossed with cp_idempotent.non_idem. §32's flagship failure, and the contrast bin recovery_before_accept where re-dispatch is safe.

cp_barrier_critical — all three agents. With unlike agents the critical path varies, and an environment where only one agent is ever last has not exercised the imbalance (§40).

cp_simul_events. §15's precondition — two job events on one cycle.

cp_locality.data_remote_lost. §36's mistake, deliberately produced, to prove the scheduler avoids it.

And cp_eligibility.domain. §39's isolation check, exercised alone.

44. Debug Taxonomy

Row 2 is the one that wastes the most time. Nearly-right results, worse under load is read as a numerical-precision problem for weeks before anyone suspects a visibility gap — and the completion-to-visibility distribution answers it in one measurement.

45. Debug Checklist

1. Which job — identity and generation? (§33)
2. Which agent owns it right now, and is that exactly one? (§16)
3. What state is it in, and how long has it been there? (§13)
4. Did the committed state equal the next-state function's output? (§16)
5. Which producer, which consumer? (§7)
6. Did the producer complete? (§9)
7. Was visibility confirmed, and how many cycles after completion? (§11, §40)
8. Which handoff model is in use — coherent, or explicit ownership? (§17)
9. If explicit: did the producer flush before handing over? (§18)
10. Which traffic class, and what is its wait against its own bound? (§22, §24)
11. Which barrier phase, and which agents are still pending? (§26)
12. Is the barrier a bitmap or a counter? (§27)
13. Did any completion arrive twice, or with a stale phase? (§26, §28)
14. Which agent was the barrier's critical path this phase? (§40)
15. What are this job's dependencies, and which are unresolved? (§29)
16. Did a link retry or recovery occur — and before or after acceptance? (§31, §32)
17. Did the scheduler re-dispatch, or query? (§32)
18. Is this job idempotent — judged from its operands? (§34)
19. Where was the data when the job was dispatched? (§35, §36)
20. Which cost terms did the scheduler use? (§37, §38)
21. Which eligibility term refused the other agents? (§38)
22. Was the access domain checked, and against the current epoch? (§39)
23. Which of the four scoreboard layers diverged first? (§41)

46. Common Misconceptions

"Shared memory automatically makes heterogeneous compute coherent." Shared addressability says the agents can name the same location. Whether a producer's write is visible to a consumer, and when, is answered by the coherence protocol or by an explicit ownership handoff — and by nothing in the completion path (§9, §11).

"Job completion means result visibility." They are different events with a variable, load-dependent gap. Launching a consumer on completion produces a partially-stale read that is nearly right, passes tolerance checks, and gets blamed on precision (§10).

"The fastest accelerator should receive the job." Only when the data is already there. In the worked example an agent 1.67× faster loses by 2.35× because 16 MiB had to move — and the break-even required it to be more than 4.4× faster (§36, §37).

"A barrier is just a counter." A duplicated completion — a correct transport retry — decrements it and advances the phase while a participant is still writing. A bitmap with a guarded clear is idempotent by construction, and heterogeneity widens the window in which the duplicate matters (§27).

"Transport recovery means re-dispatch." The agent has its operands locally and does not stop when the link does; it may have completed already. For a non-idempotent job the correct action is to query using identity and generation, and re-dispatch only if the agent confirms it never accepted it (§31, §32).

"CPU, GPU and AI can share one queue without consequences." The agent least able to tolerate latency offers the least traffic and therefore waits behind the most — an illustrative 43× inflation from queueing alone (§19, §20).

"Local atomicity is enough." An update serialised only against other jobs of the same class is not atomic with respect to the other agents. Genuine atomicity requires one serialisation point every agent goes through (16.1 §19), and this chapter does not invent one (§3).

"All accessible memory is equivalent." Local SRAM, memory behind the agent's own link, memory behind another agent's link and pooled memory differ by orders of magnitude, and the difference dominates the scheduling decision (§35).

"QoS priority guarantees progress." Priority is a comparison; a bound is a promise. A class can hold the highest priority and still starve behind a continuously-requesting one — and the class that starves is usually the one that releases resources, which is why it becomes a deadlock (§21, §23).

"Heterogeneous compute is mostly software scheduling." The failures in this chapter are a missing visibility term, two writers to one state register, a counter where a bitmap was needed, a cost model missing its largest term, and a timeout mistaken for evidence. All five are hardware architecture, and none is detectable by transport verification (§10, §15, §27, §36, §32).

47. Understanding Check

48. Summary and What Comes Next

Heterogeneous compute is the controlled transfer of ownership between agents with different execution models — different latency tolerance, access shape, synchronisation style, queue depth, granularity and completion semantics.

Four paths, four meanings. Work, data, completion and visibility are distinct, and the third does not imply the fourth — which is the chapter's flagship failure and produces a nearly-right result under load with a perfectly clean transport.

One job, one owner, one next-state function. Two writers lose a transition to a last-write-wins race that may behave differently in simulation and synthesis.

Both handoff models work and both have a step people skip — coherence traffic per line, or a flush the software must perform.

The agent least able to tolerate latency waits behind the most traffic, so per-class queues and per-class bounds are structural, and priority is a comparison while a bound is a promise.

A barrier needs a bitmap, and heterogeneity widens the window in which a duplicated completion matters.

A per-agent fault needs a per-agent blast radius, which requires a dependency bitmap, and a timeout never authorises re-dispatching non-idempotent work.

And locality beats speed. An agent 1.67× faster lost by 2.35× because the data had to move — the break-even needed 4.4×.

This chapter composed unlike agents inside one package and assumed the package itself was flat. The next chapter removes that assumption: when a package grows to several substrates and reticles, it becomes a hierarchy of local fabrics joined by constrained cuts — with routing levels, clock domains, reset domains, configuration hierarchy and failure containment that a flat package never had.

• 18.4 — Large-Package Systems — multi-reticle and multi-substrate package architectures.

Browse the full path on the UCIe tutorials index.