Open-Source SoC Examples

Chapter 18.4 counted what grows. Everything in this module so far has been built to be measured rather than found in the world.

What do real open-source Wishbone systems build — and what can be read off one of their designs?

1. LiteX — Three Topologies as Three Classes

litex/soc/interconnect/wishbone.py defines the three interconnection means this module has been discussing, as three classes with the same names.

InterconnectPointToPoint — a single master to a single slave. No arbitration, no decode, which is B3's glossary definition of point-to-point made literal.

InterconnectShared is one arbiter and one decoder:

self.arbiter = Arbiter(masters, shared, mode=arbiter)
self.decoder = Decoder(shared, slaves, register)
if timeout_cycles is not None:
    self.timeout = Timeout(shared, timeout_cycles)

That is Chapter 18.1's topology in three lines — arbitrate onto one shared interface, then decode from it. And the third line is RECOMMENDATION 3.10 shipped as a default: "Design INTERCON modules to prevent deadlock conditions. One solution to this problem is to include a watchdog timer function..." LiteX's Timeout is that watchdog, on by default.

Crossbar is the same two components in the opposite order and in more copies:

# decode each master into its access row
for row, master in zip(access, masters):
    self.submodules += Decoder(master, row, register)
# arbitrate each access column onto its slave
for column, bus in zip(zip(*access), busses):
    self.submodules += Arbiter(column, bus, mode=arbiter)

A Decoder per master and an Arbiter per slave — Chapter 18.2's structural claim, in somebody else's codebase, written before this module existed.

What the learner should notice: the shared bus and the crossbar are not different designs. They are the same two blocks, composed differently. That is why Chapter 18.2's crossbar could reuse Chapter 17.1's arbiter unchanged — and why "should we use a crossbar" is a composition question rather than a rewrite.

Two more details from the same file are worth carrying away. The Arbiter's default mode instantiates a round-robin — "if mode == "cycle": self.rr = roundrobin.RoundRobin(len(masters))" — which is Module 17's policy as a default rather than a rule. And the Decoder documents its own timing trade in a comment: "register adds flip-flops after the address comparators. Improves timing, but breaks Wishbone combinatorial feedback."

2. ZipCPU wbxbar — One Crossbar, Done Rigorously

rtl/wbxbar.v in the wb2axip repository describes itself as "A Configurable wishbone cross-bar interconnect, conforming to the WB-B4 pipeline specification".

Read that last clause carefully, because it matters for this course. This is WB-B4 pipelined, not the B3 Classic profile taught here. The pipelined profile allows a master to issue further requests before earlier ones are answered, which changes the return-path problem substantially. What transfers across the profile boundary is the structure, and the structure is instructive.

Per-slave grant state. Each slave has its own sgrant[M] and sindex[M] recording which master currently holds it — Chapter 18.2's sown0_q/sown1_q, at N × M.

A retained per-master destination. mindex[N] stores the slave channel index for each master, and the source states why: "faster/cheaper logic on the return path, since we can now use a fully populated LUT rather than a priority based return scheme." That is Chapter 18.3 §8's invariant with the opposite encoding — this module keeps the grant per slave, wbxbar keeps the destination per master, and both are answering "where does this master's response come from".

A formal assertion that reads like this module's exclusivity checker: "Can't grant the same channel to two separate masters."

And a timeout that is RECOMMENDATION 3.10 again, with the abort spelled out: "If set to a non-zero value, is a number of clock periods to wait for a slave to respond. Should the timeout expire and the slave not respond, a bus error will be returned and the slave will be issued a bus abort signal (CYC will be dropped)."

What the learner should notice: a serious crossbar carries exactly the state this module identified as necessary, and nothing exotic. Per-destination ownership, a retained destination for the return path, an exclusivity property, and a watchdog. There are no transaction IDs and no reorder buffers — the structural identity of a transfer is still origin, destination and grant.

3. OpenCores wb_conmax — A Fixed Matrix From 2002

rtl/verilog/wb_conmax_top.v, headed "WISHBONE Connection Matrix Top Level / Author: Rudolf Usselmann".

The top level carries eight master ports, m0 through m7, and sixteen slave ports, s0 through s15, each with the Wishbone signal set including ack, err and rty. Sixteen per-slave priority-select parameters, pri_sel0 through pri_sel15, each two bits.

What the learner should notice is the shape of the parameter list. The arbitration configuration is per slave, not global — sixteen separate two-bit choices. That is the contention-domain structure of Chapter 18.2 visible in a parameter list, and it dates from 2002.

What is deliberately not claimed: the fetched top-level file did not expose the address-decode mechanism, so nothing is said about it here. Nor about the project's current status, adoption or suitability. One verified observation is worth more than three unverified ones.

4. Reading Any Interconnect

The three examples are different, and the same nine questions answer all of them. This is the transferable result of Module 18, and it is worth more than the topology names.

Question 4 and question 5 together give you the topology, without anybody having to use the word. Question 8 is the one people forget to ask, and it is the one Chapter 18.3 showed can be wrong while everything else is right.

The top row is the one every block diagram draws. The bottom row is the one that gets checked last.

5. Simulation — SIM I: A Small SoC-Like Workload

A teaching system built from this module's concepts. It does not reproduce any named project and no such claim is made. A CPU-like master mixes RAM traffic with peripheral register traffic; a DMA-like master streams from RAM throughout. The peripheral's upper registers are read-only, so a configuration mistake produces a real ERR that has to be routed like anything else.

  === SIM I - a small SoC-like workload ===
    M0 behaves like a CPU: it reads RAM, writes a peripheral
    control register, reads a status register and finally
    writes one it is not allowed to write. M1 behaves like a
    DMA engine streaming from RAM throughout.
    RAM has one wait state, the peripheral has two.

    what each requester did
      CPU RAM accesses        3
      CPU peripheral accesses 3
      CPU accesses refused    1  (write to a read-only reg)
      DMA RAM reads           6

    what the slaves saw
      S0 (RAM)        reads 9   writes 0
      S1 (peripheral) reads 1   writes 1
      terminations    S0 ack 9 err 0   S1 ack 2 err 1

    what the interconnect did
      clocks with both paths busy   6
      clocks with one path busy     15
      S0 contention clocks          9
      S1 contention clocks          0
      elapsed clocks                48

    provenance audit
      dest 0  fwd 0  no-route 0  excl 0  stable 0
      foreign termination 0  data source 0  signature 0

    -> the DMA's RAM traffic and the CPU's peripheral traffic
       overlap because they go to different destinations. The
       CPU's own RAM reads do not, because they meet the DMA
       at S0. Both facts come from the address map, not from
       the topology.

Reading it

Six clocks with both paths busy, and nine contention clocks at S0. Both numbers come from the address map rather than from the topology.

The overlap is the CPU at the peripheral while the DMA is at the RAM. Different destinations, different queues, no meeting. The contention is the CPU's own three RAM reads meeting the DMA's six — same destination, same queue.

So a single workload contains both of this module's cases at once, which is what a real SoC looks like: some of the traffic is separable by destination and some of it is not, and the ratio is a property of the software and the memory map.

S1 ack 2 err 1. Three phases at the peripheral: a control-register write that landed, a status read, and a write to a read-only register that was refused. The ERR was routed to M0 and to nobody else — foreign termination 0 — and the CPU's client saw an error rather than a success, which is Chapter 10.2's contract surviving two topology changes and an arbiter.

S1 writes 1. The refused write changed nothing. The slave's own counter is what says so, not the bus.

And all eight provenance counters are zero, on a workload with mixed destinations, mixed directions, two wait-state profiles, contention, overlap and an error. That is the point of running it after Chapter 18.3 rather than before.

6. Simulation — SIM J: The Complete Audit

One deterministic trace through everything this module has discussed, on the crossbar, with two wait states at S0 and one at S1: the four routes individually, simultaneous different destinations, simultaneous same destination, a refused write, an accepted write.

  === SIM J - the complete interconnect audit ===
    crossbar, S0 with two wait states, S1 with one and its
    upper registers read-only.

    ten clocks inside the simultaneous-different segment
      clk  m0_dest m1_dest  S0 act  S0 own  S1 act  S1 own  ACK
       31     1       1       0      -       0      -          -
       32     1       1       0      -       0      -          -
       33     0       1       0      -       0      -          -
       34     0       1       1      M0      1      M1         -
       35     0       1       1      M0      1      M1        S1
       36     0       1       1      M0      0      M1        S0
       37     0       1       0      M0      0      -          -
       38     0       1       0      -       0      -          -

    segment                    clocks  overlap  contention
      M0 -> S0                     8        0         0
      M0 -> S1                     7        0         0
      M1 -> S0                     8        0         0
      M1 -> S1                     7        0         0
      simultaneous, different      8        2         0
      simultaneous, same slave    12        0         4
      a refused write (ERR)        7        0         0
      an accepted write            7        0         0

    ROUTE PROVENANCE AUDIT
      request reached the wrong slave        0
      forward context was not the owner's    0
      a phase presented with no owner        0
      a master owned two slaves at once      0
      owner moved under an open phase        0
      a master saw a foreign termination     0
      data did not come from the granter     0
      data was not the intended slave's      0

    CONCURRENCY AND CONTENTION
      clocks with both paths busy  2
      clocks with one path busy    21
      clocks with neither          42
      contention at S0 4   at S1 0

    TERMINATIONS AND EFFECTS
      S0 ack 5 err 0   S1 ack 4 err 1
      delivered to M0 6   to M1 4
      S0 reads 5 writes 0   S1 reads 3 writes 1
      elapsed clocks 64

Reading it

The ten-clock window is the destination-coherence evidence. At clock 32 m0_dest is 1; at clock 33 it is 0, because M0 has started a new request with a new address. At clocks 34 to 36 S0 is owned by M0 and S1 by M1 simultaneously, and the two slaves answer on different clocks — S1 at 35 with one wait state, S0 at 36 with two.

That difference is worth a sentence. The two transfers overlapped and were not synchronised: two independent paths means two independent durations. A return path keyed on a live decode would have to be right on both of those clocks, and Chapter 18.3 measured what happens when it is not.

The segment table separates topology from workload one last time. Four single-master segments: zero overlap, zero contention — nothing to overlap with. "Simultaneous, different": 2 overlap clocks, 0 contention. "Simultaneous, same slave": 0 overlap, 4 contention clocks. Same hardware, opposite results, and the only variable is the address.

The provenance audit is eight zeros across a trace containing both topologies' characteristic behaviours, both termination classes, two wait-state profiles and a destination change under an open phase.

And the last check is the one worth adopting. Deliveries 6 + 4 = 10; terminations 5 + 4 + 0 + 1 = 10. Client completions balance slave terminations exactly, which is two counters, needs no model of the topology, and is the single cheapest thing you can add to an interconnect you did not write.

7. What Module 18 Established

18.1 — four agreements, and a topology that serialises. An interconnect is ownership, destination, forward route and return route, and a transaction is correct only when all four describe the same one. Two free slaves, two independent requests, zero clocks of overlap — and the specification attributes that to the architecture, in those words, rather than to Wishbone.

18.2 — contention moves. Decode before arbitration, an arbiter per destination. Two slaves acknowledging two different masters on the same clock — and the same crossbar, given the same destination twice, behaving exactly like a shared bus with an idle second path.

18.3 — five things that must agree. Origin, intended, actual, source, recipient. Five broken interconnects, every master and every slave in all of them obeying every rule in B3. And the defect a structural check missed and a known-answer check caught.

18.4 — counting instead of adjectives. Arbiters grow with S, decoders with M, return muxes with M × S inputs, and the verification argument grows with M × S routes. A crossbar with twelve transfers to one destination and a second path idle throughout.

18.5 — the same nine questions answer every diagram. Three verified projects that chose differently, and a full audit closing at eight zeros.

The through-line is one sentence. Wishbone specifies what happens at a port and names the module that connects ports. Everything between two ports is yours, and there is no rule you can break by getting it wrong.

Across Modules 9 to 18 the count of published defects a bus-level protocol checker would catch remains one — Chapter 10.2's ACK+ERR double termination.

8. Common Mistakes

"Real systems use crossbars; shared buses are for tutorials."

Why it is unsupportable: LiteX ships InterconnectPointToPoint, InterconnectShared and Crossbar, and lets the integrator choose. A library that shipped only one would be making the claim; one that ships three is not.

"An open-source project using X proves X is the right choice."

Why it is wrong: a project's choice is evidence about that project's constraints. This chapter claims three architectural facts and nothing about popularity, deployment or performance, because only the facts were verified.

"The crossbar in project Y works, so I can copy its return path."

Why it is dangerous: wbxbar conforms to WB-B4 pipelined, not B3 Classic. The profile changes what a return path has to handle. Copy the invariant — a retained destination, a per-destination grant — not the code.

"Wishbone prescribes an interconnect architecture."

Why it is wrong: it names the INTERCON, lists five interconnection means after the word "including", and hands arbitration to the end user. There is no prescribed routing architecture and no prescribed arbitration architecture.

"If I can name the topology, I understand the system."

Why it is insufficient: Section 4's nine questions. Question 8 — how the response is routed back — is not answered by any topology name, and it is where the expensive defects live.

9. Interview Reasoning

"Show me how you would read an unfamiliar SoC block diagram."

Nine questions, in order — initiators, responders, shared paths, arbitration points, decode position, overlappable pairs, contention domains, return routing, retained state. Questions 4 and 5 give you the topology without needing its name; question 8 is the one nobody puts on the diagram.

"What does a real Wishbone crossbar keep that a shared bus does not?"

Per-destination ownership, and a retained association between a master and the slave answering it. wbxbar keeps a per-master slave index and says why: the return path. This module's crossbar keeps a per-slave owner. Same invariant, opposite encoding.

"Does a library shipping a crossbar mean crossbars are better?"

No — it means the library declined to choose. LiteX ships point-to-point, shared and crossbar. The choice belongs to the system, and destination diversity is the variable.

"What would you verify first on an interconnect you inherited?"

Client completions against slave terminations, then response provenance per master, then forward destination. None of those is a protocol check, and a protocol checker passes all five of Chapter 18.3's broken designs.

"Why is a watchdog in an interconnect a specification-sanctioned idea?"

RECOMMENDATION 3.10 suggests exactly that — a timer on STB_O that returns ERR or RTY if a cycle exceeds a limit. LiteX's Timeout and wbxbar's timeout parameter are both that recommendation, and it is the only behaviour B3 asks an INTERCON for beyond wiring.

10. Understanding Check

LiteX builds its Crossbar from Decoder and Arbiter instances. What does that tell you about the two topologies?

That they are compositions of the same parts. One arbiter then one decoder is a shared bus; a decoder per master then an arbiter per slave is a crossbar. "Move to a crossbar" is a re-composition, not a rewrite.

wbxbar keeps mindex[N] per master; this module keeps sown0_q/sown1_q per slave. Are these different designs?

Same invariant, different encoding. Both answer "which slave is answering this master". One indexes by master, the other by slave, and the cost difference is in the return-path logic — which is the reason wbxbar's own comment gives.

In SIM I, six clocks overlapped and nine contended. Which was the topology's doing?

Neither, on its own. The topology permitted the overlap; the address map decided that the CPU's peripheral traffic was separable from the DMA's RAM traffic and its own RAM traffic was not.

SIM J's deliveries balance its terminations exactly. Why is that the cheapest useful check?

Because it needs no model of the interconnect — two counters and an equality. It catches broadcast returns, phantom completions and lost terminations, and it is true regardless of topology, policy or address map.

A colleague proposes adopting wbxbar into this course's B3 Classic system. What do you flag?

The profile. It conforms to WB-B4 pipelined, which permits requests to be outstanding. The structure transfers; the timing assumptions do not, and a Classic master driving it is a different integration question.

11. What's Next

The fabric is built, routed, measured and read. Every system in this module has been two masters and two slaves with addresses chosen to make a point.

What does a real processor bring to this, and how does an SoC get built around it?

Module 19 — Wishbone in RISC-V Systems takes up what this module deliberately stayed out of: Wishbone as the bus in RISC-V SoCs, LiteX as a system builder rather than as a source of two class definitions, attaching a CPU core as a master, and attaching peripherals as slaves. The full path is on the Wishbone curriculum index.