Arbitration Failures

Add a second master and a new class of question appears, one that single-master debugging never has to ask:

This transfer completed. Whose was it?

Modules 16–18 built the arbiters and the interconnects. This chapter does not rebuild them. It is about three words that get used interchangeably and mean entirely different things.

1. Contention, Starvation, Corruption

The first two are properties of a policy. The third is a defect. Treating the first as a bug wastes a week; treating the third as contention ships silicon that occasionally hands one CPU another CPU's data.

2. The Five Questions

A debugger who can answer only "did it complete" cannot work here. Five separate facts are needed, and a system that conflates any two of them is unfixable:

// ── THE FIVE QUESTIONS ──────────────────────────────────────────────────
//     WHO REQUESTED?          both masters may be asserting [CYC_O]
//     WHO OWNED?              exactly one may be granted
//     WHO WAS PRESENTED?      the request the slaves actually saw
//     WHO TERMINATED?         which target answered
//     WHO RECEIVED IT?        which master the answer was delivered to

Ownership itself is not a local invention. RULE 3.25 makes CYC_O the marker for an entire cycle, so a fabric that re-arbitrates inside one has handed a master's transfer to somebody else — while every individual port stays perfectly protocol-shaped.

3. The Ownership Timeline

Nine columns, one row per clock, reduced to the clocks where something happened:

=== SIM H - ARBITRATION: OWNERSHIP TIMELINE ===

  the correct system, first 40 clocks
  cyc  req0 req1  owner  target  presented  term  response to
    2   Y  Y   --     --      no         no    --
    3   Y  Y   M0     S0      YES        no    --
    4   Y  Y   M0     S0      YES        no    --
    5   Y  Y   M0     S0      YES        no    --
    6   Y  Y   M0     S0      YES        YES   M0
    7   .  Y   M0     --      no         no    --
    8   .  Y   M1     S1      YES        YES   M1
   10   Y  .   --     --      no         no    --
   11   Y  .   M0     S0      YES        no    --
   12   Y  Y   M0     S0      YES        no    --
   13   Y  Y   M0     S0      YES        no    --
   14   Y  Y   M0     S0      YES        YES   M0
   15   .  Y   M0     --      no         no    --
   16   .  Y   M1     S1      YES        YES   M1

Read what healthy looks like. M1 requests from cycle 2 and does not own the bus until cycle 8 — six clocks of pure waiting on a correct system. At cycle 7 M0 has released CYC_O and the fabric has not yet re-granted; at cycle 8 M1 owns, is presented, terminates and receives its own response on the same clock.

Every row where presented is YES has exactly one owner, one target, and — where it terminated — a response delivered to that same owner. That coherence is the property; the timeline is how you check it.

4. The Four Rigs

=== SIM H - THE FIVE QUESTIONS ===

  rig  phases  owner moved  response misdelivered  first moved  first misdeliver
  HA      12            0                      0            0                 0
  HB      12            6                      0            5                 0
  HC      99            0                     99            0                 7
  HD      12            0                      0            0                 0

  rig  protocol P0  protocol P2  P2 ctx moved  scoreboard  M0 retired  M1 retired
  HA            0            0             0           0           6           6
  HB            6            6             6           0           6           6
  HC            0            0             0           0           0           6
  HD            0            0             0           0           6           6

• HA is the correct system. Twelve phases, six per master, everything coherent.
• HB changes the owner with a transfer in flight.
• HC delivers every termination to the master that does not own the bus.
• HD is an intentionally unfair arbiter — not a defect, a policy.

Two rows in that table are worth stopping on.

HC's scoreboard reports zero mismatches. So does HA's. The scoreboard is not broken; it has nothing to compare. The monitor watches the owner's port, nothing ever terminates there, and a scoreboard with an empty input agrees with everything. Zero mismatches is not a pass. Zero mismatches with zero comparisons is an absence of evidence.

HB's protocol violations are 6 at P0. In every other rig in this module P0 stays clean while P2 reports the breakage. Here both report it, and the reason is a property of the probe rather than of the design — Section 7 below.

5. Corruption

  CORRUPTION  HC delivered 99 terminations to the master
              that did not own the bus, first at cycle 7.
              Protocol violations at P0: 0. Every port was
              shaped correctly. The answers went to the
              wrong engineer, and the arithmetic of that is
              the whole symptom:
                M0 issued 1 and retired 0
                M1 issued 6 and retired 6
              M1 completed operations using acknowledgements
              earned by M0's transfers. M0 is still waiting.
              Note the monitor saw 0 transactions: it
              watches the owner's port, and nothing ever
              terminated there. A SCOREBOARD WITH NOTHING TO
              COMPARE REPORTS ZERO MISMATCHES (0).

M0 issued one operation and retired none. M1 issued six and retired six — using acknowledgements generated in response to M0's requests, aimed at M0's addresses, answered by M0's target.

There is no Wishbone rule against this. B3's model is point-to-point; it has exactly one master per interface and therefore nothing to say about which of several masters an answer belongs to. Response provenance is this system's rule, and the executable assertions record it as such:

  rig  A1   A2   A3   A4   A5   SPEC  LOCAL
  HA    0    0    0    0    0     0      0
  HB    0    0    0    0    6     6      0
  HC    0    0    0   99    0     0     99
  HD    0    0    0    0    0     0      0

  HC breaks A4 99 times and NOT ONE SPECIFICATION CHECK
  (SPEC total 0). Response provenance is this system's own
  rule; B3 has nothing to say about which master an answer

Ninety-nine local-policy breaches, zero specification breaches. Reporting HC as a Wishbone violation would be factually wrong, and the conversation with the IP vendor would end there.

6. The Checker That Could Not See It

The ownership checker did not work the first time, and the way it failed is the chapter's sharpest lesson.

Its first version watched the owner's own port for terminations — the obvious place, since that is where a master's answer arrives. On HC it reported zero phases, zero owner moves and zero misdeliveries — on a rig built to misdeliver every single response. That output is not reproduced below, because the wiring that produced it no longer exists; the comment the fix left behind is the record. The checker was not wrong about its input. It was pointed at the one boundary the defect removes the evidence from:

  // The termination is taken from P4 - the fabric's return path - and
  // NOT from the owner's own port. The first version of this wiring
  // watched the owner's port, which is where a misdelivered response by
  // definition never appears: the checker reported zero phases and zero
  // misdeliveries on a rig built to misdeliver every single one. A
  // CHECKER CANNOT DETECT A DEFECT AT A BOUNDARY THE DEFECT REMOVES IT
  // FROM.

Moved to P4 — the return path, before the destination steering — it reports 99 and a first occurrence at cycle 7.

7. The Probe That Moved

HB reports six context moves at P0, which contradicts every other experiment in this module. It is not a second defect:

  OWNERSHIP   HB moved the owner inside 6 open phases,
              first at cycle 5, and the request seen by the
              slaves changed 6 times as a result.
              P0 reports the SAME 6 context moves - and
              that is not a second bug, it is a property of
              the probe. P0 is defined as THE OWNER'S PORT.
              When ownership moves, P0 moves with it, so a
              probe that was a fixed observation point in
              every other rig stops being one here.
              A PROBE DEFINED RELATIVE TO A SELECTOR IS ONLY
              AS STABLE AS THE SELECTOR.

P0 is a multiplexer output, selected by owner. In eleven of twelve rigs the selector is constant for the whole phase and P0 behaves exactly like a physical pin. In HB the selector changes mid-phase, so P0 jumps from M0's pins to M1's pins and reports a context move that neither master committed.

Both checkers are telling the truth about the signals they see. Neither is telling you what you assumed it would.

8. Starvation, And An Experiment That Said No

HD's arbiter never hands the bus to a waiting M1 when M0 releases it — a deliberately unfair local policy, armed to produce starvation. It did not.

  rig  M0 requested  M0 owned  M1 requested  M1 owned  M1 share
  HA            30        30            32        12      28%
  HD            30        30            33        12      28%

  STARVATION  HD's arbiter never hands the bus to a waiting
              M1 on release - a deliberately unfair local
              policy - and M1 still retired 6 operations.
              THE EXPERIMENT REFUTES THE HYPOTHESIS. An
              unfair arbiter does not starve a requester
              that competes against a master which RELEASES
              [CYC_O] between transfers. Starvation needs an
              owner that does not let go, not merely a
              priority order.

Twenty-eight per cent of ownership clocks in both rigs, six of six operations retired in both. The unfairness is real and is worth nothing measurable here, because M0 drops CYC_O between transfers and a registered arbiter re-evaluates in that gap.

This is published because it is true, and because the alternative is worse. A tutorial that arms a "starvation defect", reports the number it wanted, and never checks whether the mechanism actually operated has taught a plausible story instead of a measurement. Starvation requires an owner that holds CYC_O across many transfers — a block cycle — not merely a priority order.

9. The Debug Record

10. What To Carry Forward

• Contention is not starvation is not corruption. A master that waited is not a master that was robbed.
• Answer the five questions separately. Who requested, who owned, who was presented, who terminated, who received it — a system that conflates any two cannot be debugged.
• Fairness is local policy. B3 mandates no arbitration algorithm, so "M1 waited too long" is a statement about your arbiter and your requirements.
• A checker cannot see a defect at a boundary the defect removes the evidence from. Place it where the evidence still exists.
• A probe selected by a signal is only as stable as that signal. P0 is a pin in eleven rigs and a multiplexer in the twelfth.
• Zero mismatches with zero comparisons is not a pass.
• Arm your defect and then check that it armed. An unfair arbiter that starves nobody is a finding, not an embarrassment.

Chapter 27.5 takes the case where everything about the transport is right and only the payload is wrong.