Scalability

Chapter 18.3 finished the correctness question for both topologies. The comparison between them has so far been made with two words: "serialises" and "overlaps".

What actually grows when a system gets bigger, and along which dimension does a topology "scale"?

1. What Grows, Counted

For M masters and S slaves, the two topologies grow differently and the difference is countable without building either.

Read the middle rows as the trade. A shared bus has one of everything; a crossbar has S arbiters and M decoders, and its return side needs M muxes of S inputs each where the shared bus needed one.

Two of those rows are the reason "crossbars scale" needs qualifying. The arbiter count grows with S and the decoder count with M, so the structure grows as M + S — while the relationships grow as M × S in both. A full M × S crossbar is not free, and the specification's own glossary reaches for hierarchy rather than width as systems get large: "Switched Fabric Interconnection — A type of interconnection that uses large numbers of crossbar switches. These are organized into arrays..."

And the growth is not only in gates. LiteX's Wishbone Decoder carries a comment about exactly the timing consequence:

# register adds flip-flops after the address comparators. Improves timing,
# but breaks Wishbone combinatorial feedback.

Registering the decode buys timing and costs a clock of latency — and the "combinatorial feedback" it breaks is the single-cycle STB_O → ACK_I path that OBSERVATION 3.50 warns about in the other direction: "In large high speed designs the asynchronous assertion of [ACK_O], [ERR_O], and [RTY_O] could lead to unacceptable delay times, caused by the loopback delay from the MASTER to the SLAVE and back to the MASTER." Both ends of that loop get longer as the interconnect gets wider, and that is a real scaling pressure this module can name without measuring.

2. Scalability Is Not One Number

Ask "scalable along which dimension?" and the answer is usually several, with different answers.

Two rows deserve emphasis because they are the ones that surprise people.

Verification grows fastest. A shared bus has one forward path and one return path to get right. A crossbar has a return path per master, each selecting among S sources, and Chapter 18.3's gate needed five checkers and seven interconnects to establish that a 2 × 2 was correct. At 4 × 8 the same argument is not five checkers — it is the same five checkers over thirty-two routes.

And hotspots do not appear in the topology column at all, because they are not a property of the topology. Section 5.

3. Simulation — SIM G: The Same Work, Two Destination Patterns

Twelve reads either way: six from each master, two wait states at each slave, and the identical stimulus driven into a shared bus and a crossbar simultaneously.

  === SIM G - destination diversity, both topologies ===
    twelve reads either way: six from each master, two wait
    states at each slave. The testbench waits for BOTH rigs
    before issuing the next pair, so neither is given a head
    start and the comparison is of behaviour, not of speed.

    workload 1 - both masters read S0
      topology     transfers  overlap clocks  contention clocks
      shared bus      12             0               24
      crossbar        12             0               24

    workload 2 - M0 reads S0, M1 reads S1
      topology     transfers  overlap clocks  contention clocks
      shared bus      12             0                0
      crossbar        12            18                0

    -> the crossbar's overlap is zero on workload 1 and
       non-zero on workload 2. The topology supplies the
       CAPABILITY; the workload decides whether it is used.

Reading it

Workload 1, both masters reading S0:

Identical. Every number. The crossbar's second path exists and contributes nothing, because nothing is addressed to it.

Workload 2, M0 reading S0 and M1 reading S1:

Eighteen clocks of genuine overlap, and zero contention on both topologies — because with different destinations the masters were never competing for the same thing even on the shared bus. What the shared bus lacked was not arbitration capacity; it was a second path.

So the same hardware pair produces identical behaviour on one workload and completely different behaviour on the other.

The topology supplies the CAPABILITY. The workload decides whether it is used.

One measurement note, because it is the kind of thing that quietly invalidates comparisons. The testbench waits for both rigs to finish each pair before issuing the next, so neither topology is given a head start and the run lengths are not a race. These are counts of behaviour, not of speed — and turning them into a throughput claim would need a different experiment, which is Module 22's.

4. Simulation — SIM H: A Hotspot

Twelve reads on the crossbar, all of them to S0. S1 is present, correct, and idle.

  === SIM H - a crossbar with a destination hotspot ===
    both masters read S0 repeatedly; S1 is idle and available.

    crossbar, twelve reads, all to S0
      S0 path active clocks        36
      S1 path active clocks        0
      clocks with both paths busy  0
      contention clocks at S0      24
      contention clocks at S1      0
      S0 reads 12   S1 reads 0
      elapsed clocks               66

    -> a second path exists, is correct, and is unused. The
       contention is caused by where the traffic is going,
       and no amount of extra paths changes where it is
       going.

Reading it

S1 path active clocks: 0. S1 reads: 0. clocks with both paths busy: 0. contention clocks at S0: 24.

A correct second path, wired, verified, and never used once. The crossbar behaved exactly like Chapter 18.1's shared bus for the entire run, and it cost an extra arbiter, an extra decoder and an extra return mux to do so.

The contention is caused by where the traffic is going. Adding paths cannot move traffic. A hotspot is a property of the address map and the workload, and the only fixes are on that side: split the hot slave, give it more ports, cache in front of it, or move the traffic.

This is the measurement that makes the earlier caution concrete. "This makes the crossbar switches inherently faster than traditional bus schemes" is the specification's glossary describing crossbar switches in general. Here is a Wishbone crossbar, correct in every respect, that is exactly as fast as a shared bus — and the reason has nothing to do with the interconnect.

Say it as the diagnostic rule:

Before blaming an interconnect for lack of concurrency, count per-slave activity. If one slave's active-clock count is zero, the conversation is not about the interconnect.

5. Where the Growth Actually Is

Put the two topologies' defining lines next to each other. Both are verbatim from the modules Chapters 18.1 and 18.2 published and this module compiled.

Shared bus — one decode, gating two slaves:

  assign s0_cyc_o = s_cyc && !dest;
  assign s0_stb_o = s_stb && !dest;
  assign s1_cyc_o = s_cyc &&  dest;
  assign s1_stb_o = s_stb &&  dest;

Crossbar — one decode per master, producing per-destination eligibility:

  assign m0_w0 = m0_cyc_i && !m0_dest;
  assign m0_w1 = m0_cyc_i &&  m0_dest;
  assign m1_w0 = m1_cyc_i && !m1_dest;
  assign m1_w1 = m1_cyc_i &&  m1_dest;

Four lines each, and the whole structural difference is in the subscripts. The shared bus decodes once, after arbitration, and the result names a slave. The crossbar decodes per master, before arbitration, and the result names a queue.

Now extend both to M masters and S slaves in your head. The first block stays four lines and becomes S lines. The second becomes M × S lines — and behind them, S arbiters and M return muxes of S inputs each.

That is what "the crossbar costs more structure" means, stated as something you can count rather than as an adjective. It is also why the specification reaches for hierarchy at scale rather than for a single enormous crossbar.

6. Failure Modes and Discriminating Evidence

These are the defects that only appear once a system is big enough to have a topology worth arguing about.

SYMPTOM — false parallelism: two masters, two slaves, and no overlap ever.

Candidates. The design is a shared bus. Or it is called a crossbar and has a shared internal path — a single arbiter feeding per-slave muxes, or one shared address/data bus behind the per-slave gating.

Discriminating evidence. Per-slave active clocks with both non-zero, and a both-active count of zero. That combination cannot happen in a real crossbar under an overlapping workload. If it does, look for a signal in the interconnect that only one transfer can occupy — an owner register with global scope is the usual one.

Where to look. Count the arbiters. A crossbar has one per destination; if there is one, it is not one.

SYMPTOM — hotspot misdiagnosed as an interconnect problem.

Candidates. All traffic targets one slave. The address map concentrates two engines on one memory.

Discriminating evidence. Per-slave active clocks. SIM H's S1 shows zero. Zero is not a performance problem in the interconnect; it is an absence of traffic.

Where to look. The address map and the software, not the fabric.

SYMPTOM — a route works in the 2 × 2 bring-up and fails after slaves are added.

Candidates. A decode that was unambiguous with two ranges overlaps with five. A return mux whose default arm was harmless with two sources. An OR-reduced response that was safe because only one slave existed in a cycle.

Discriminating evidence. Chapter 18.3's eight counters, run on every route rather than on the ones bring-up exercised. The route set is M × S; a test plan that grew with M only will have a blind spot the size of S.

Where to look. Chapter 12.7's sparse-map reasoning, applied to the destination decode rather than to a slave's offsets.

SYMPTOM — timing closure degrades when the interconnect widens.

Candidates. A combinational path from a master's STB_O through the decode, the mux, the slave and back to ACK_I — the loop OBSERVATION 3.50 describes — now crosses more logic.

Discriminating evidence. Whether the decode is registered. LiteX's Decoder exposes this as a parameter and states the trade in its own comment: "Improves timing, but breaks Wishbone combinatorial feedback."

Caution. This module measured nothing about timing. The failure mode is named because it is real and structural; the fix and its cost belong to a synthesis run this module did not perform.

7. Common Mistakes

"Shared bus is for small systems, crossbar is for large ones."

Why it is too coarse: the useful variable is destination diversity, not size. A twelve-master system whose traffic all goes to one DRAM controller gains nothing from a crossbar; a two-master system with two independent memories gains a lot. SIM G measured both cases on the same pair of designs.

"Crossbar scalability means unlimited scalability."

Why it is wrong: arbiters grow with S, decoders with M, return muxes with M × S inputs, and verification with M × S routes. The specification's own answer at scale is switched fabrics — arrays of crossbars — which is a hierarchy, not a wider crossbar.

"More paths remove hotspots."

Why it is wrong: SIM H. Twelve transfers, one destination, a second path idle throughout. Paths cannot move traffic.

"Crossbar is always faster."

Why it is unsupportable: SIM G workload 1 and SIM H are both crossbars performing identically to a shared bus. And no timing measurement of any kind was taken in this module.

"Shared bus is obsolete."

Why it is unsupportable: one arbiter, one decoder, one return mux, one route set to verify. For a workload with a single dominant destination it is the correct answer, and it is one of five interconnection means B3 lists without ranking.

"We changed to a crossbar and throughput did not improve, so the crossbar is broken."

Why it is a misdiagnosis: check per-slave activity first. SIM H is exactly this system, and everything in it is correct.

"Scalable" without a dimension.

Why it is not a claim: Section 2 has eight dimensions and the two topologies win different ones. A statement that does not name the dimension cannot be checked and cannot be wrong.

8. Interview Reasoning

"How would you compare two topologies without making unsupported PPA claims?"

Count structure and measure behaviour. Structure: arbitration domains, arbiters, decoders, mux inputs, retained state, route sets to verify. Behaviour: per-slave active clocks, overlap clocks, contention clocks, under a stated workload. Neither needs a synthesis run, and neither is a performance claim.

"What does 'scalable' mean for an interconnect?"

Nothing, until a dimension is named. Master count, slave count, concurrency, routing structure, arbitration domains, hotspot behaviour and verification effort all scale differently, and a crossbar improves some of them while making others worse.

"What is a destination hotspot, and why does a crossbar not remove it?"

A workload in which most traffic targets one slave. A crossbar gives independent paths to different destinations; a hotspot has only one destination. The contention is at the slave, and extra paths to elsewhere do not lead anywhere useful.

"Your crossbar shows no concurrency. What do you check, in order?"

Per-slave active clocks. If one is zero, it is the workload. If both are non-zero but the both-active count is zero, there is a shared resource inside the design — false parallelism. Only then is it worth reading the arbiters.

"What grows fastest as a crossbar gets bigger?"

Verification, in my experience of what this module measured. Establishing a 2 × 2 was correct took five checkers and seven interconnects, and the route set is M × S with a return path each. The gates grow as M + S blocks; the argument grows as M × S.

"When is a shared bus still the right answer?"

When destination diversity is low, when one place to look matters, or when the verification budget is the binding constraint. Chapter 18.5 shows a real library that ships both and lets the integrator choose.

9. Understanding Check

SIM G workload 1 produced identical numbers for both topologies. Was the crossbar wasted?

On that workload, yes — and it was still correct. The structure cost was paid and the capability was not used. That is the definition of a topology mismatched to a workload, and it is a design review finding rather than a bug.

In SIM H, S1 was idle for twelve transfers. What would you change?

The address map or the traffic, not the interconnect. Split the hot data across both slaves, or put the second memory where the DMA is actually reading. Nothing in the interconnect column can help.

Which dimension does a crossbar make worse, and by how much?

Routing structure and verification. Arbiters go from 1 to S, decoders from 1 to M, return muxes from 1 to M of S inputs — and the route set to verify goes from one path pair to M × S path pairs.

Why does this chapter refuse to state a frequency or area comparison?

Because no synthesis tool was run. Elaboration is not synthesis, and a number nobody measured is not evidence. The structural counts are what can be stated honestly without one.

10. What's Next

Two topologies, correct, compared structurally, and measured under two workloads.

What do real open-source Wishbone systems actually build — and what can be read off one of their block diagrams?

Chapter 18.5 — Open-Source SoC Examples looks at three verified projects that made three different choices, and closes the module with a complete interconnect audit: every route, both topologies' behaviours, wait states, contention, an ERR, and eight provenance counters at zero.