LiteX Systems

Chapter 29.1 read a system somebody wrote by hand. This one is assembled by a framework, and the interesting question is not how does the Python work — it is:

What hardware did those five lines just build, and which of its behaviours are Wishbone's?

1. What We Inspected

This is a framework, not a chip. Everything below describes what the inspected source constructs. It says nothing about any particular SoC built with it.

2. Generator Abstraction Is Not Protocol Abstraction

A LiteX user writes Python and receives Verilog. The abstraction hides the wires; it does not — and cannot — hide the obligations. Whatever the generator emits still has to decide, on every clock:

• which master owns the shared interface;
• which slave the address belongs to;
• which slave's read data reaches the master;
• who the acknowledgement is for.

The inspected file answers all four in three small classes. Here is the assembly, in full:

class InterconnectShared(LiteXModule):
    def __init__(self, masters, slaves, register=False, timeout_cycles=1e6, arbiter="cycle"):

ORIGINAL SOURCE EXCERPT — enjoy-digital/litex, litex/soc/interconnect/wishbone.py, commit dce79bf9.

        self.arbiter = Arbiter(masters, shared, mode=arbiter)
        self.decoder = Decoder(shared, slaves, register)
        if timeout_cycles is not None:
            self.timeout = Timeout(shared, timeout_cycles)

ORIGINAL SOURCE EXCERPT — same file.

N masters arbitrate onto one shared interface. That one interface is decoded to M slaves. A watchdog watches the shared interface. Three sentences, and the whole topology follows from them — including the fact that two transfers can never be in flight at once, because there is exactly one shared interface for them to be in flight on.

3. The Decoder Gates Exactly One Wire

        # connect master->slaves signals except cyc
        for slave in slaves:
            for name, size, direction in _layout:
                if direction == DIR_M_TO_S and name != "cyc":
                    self.comb += getattr(slave[1], name).eq(getattr(master, name))

ORIGINAL SOURCE EXCERPT — same file.

        # combine cyc with slave selection signals
        self.comb += [slave[1].cyc.eq(master.cyc & slave_sel[i])
            for i, slave in enumerate(slaves)]

ORIGINAL SOURCE EXCERPT — same file.

Address, data, byte selects, write-enable and the strobe reach every slave unconditionally. Only CYC carries the selection.

That is legal, and the reason it is legal is a rule from Chapter 4.9:

RULE 3.30 — SLAVE interfaces MAY NOT respond to any SLAVE signals when CYC_I is negated.

Because a deselected slave is obliged to ignore everything, gating one wire deselects it completely. The generator exploits that to produce less logic — every other signal is a plain fan-out with no per-slave gating at all.

DERIVED INFERENCE: a slave that ignores CYC_I and acts on STB_I alone will misbehave in a LiteX system even though its own bus interface looks reasonable in isolation, because it will see every transfer in the SoC. That is a concrete integration hazard produced by a legal optimisation.

The terminations come back the same economical way:

        self.comb += [
            master.ack.eq(Reduce("OR", [slave[1].ack for slave in slaves])),
            master.err.eq(Reduce("OR", [slave[1].err for slave in slaves]))
        ]

ORIGINAL SOURCE EXCERPT — same file.

An OR across all slaves, with no provenance latch — safe for the same reason, since exactly one CYC is ever asserted.

The register option, and the trade-off its authors wrote down

    # register adds flip-flops after the address comparators. Improves timing,
    # but breaks Wishbone combinatorial feedback.

ORIGINAL SOURCE EXCERPT — same file.

That comment is Chapter 28.4's registered-versus-combinational scenario, stated by the framework's own authors, about their own decoder. The specification frames the same tension in OBSERVATION 3.50, which warns that asynchronous termination "could lead to unacceptable delay times, caused by the loopback delay from the MASTER to the SLAVE and back to the MASTER."

4. The Arbiter Steers By The Live Grant

        # connect slave->master signals
        for name, size, direction in _layout:
            if direction == DIR_S_TO_M:
                source = getattr(target, name)
                for i, m in enumerate(masters):
                    dest = getattr(m, name)
                    if name == "ack" or name == "err":
                        self.comb += dest.eq(source & (self.rr.grant == i))
                    else:
                        self.comb += dest.eq(source)

ORIGINAL SOURCE EXCERPT — same file.

ack and err are ANDed with the current grant. Every other slave-to-master signal, read data included, is broadcast to all masters unsteered.

Chapter 27.4 measured what happens when the grant can move mid-transfer: 99 terminations delivered to a master that did not own the bus, with zero specification violations. Chapter 28.4 made "latch the owner, do not mux by live grant" the finding of a design review.

And the inspected source contains the mitigation, as a parameter:

        elif mode == "transaction":
            self.rr = roundrobin.RoundRobin(len(masters), roundrobin.SP_CE)
            cycs = Array(m.cyc for m in masters)
            self.comb += self.rr.ce.eq(target.ack | target.err | ~cycs[self.rr.grant])

ORIGINAL SOURCE EXCERPT — same file.

In mode="transaction" the round-robin is clock-enabled only when the granted transaction terminates, or when the granted master releases CYC. The grant cannot move underneath a live transfer.

SOURCE FACT: the default is mode="cycle". NOT VERIFIED: whether any particular LiteX SoC configuration overrides that default. We inspected the interconnect module, not a generated SoC.

Both modes complete every operation in our reconstruction; they differ in how often ownership moves:

  the same contended traffic through the same interconnect
  mode           M0 ops  M1 ops  grant handovers
  cycle                4       4                8
  transaction          4       4                7

OUR MEASUREMENT, from a VLSI Mentor reconstruction.

5. The Timeout Is Policy, And An Unusual One

        timer = WaitTimer(cycles)
        self.submodules += timer
        self.comb += [
            timer.wait.eq(master.stb & master.cyc & ~master.ack),
            If(timer.done,
                master.dat_r.eq((2**len(master.dat_w))-1),
                master.ack.eq(1),
                self.error.eq(1)
            )
        ]

ORIGINAL SOURCE EXCERPT — same file.

Five lines, and every one is a decision rather than a requirement:

This is not "Wishbone has a timeout". B3 bounds no latency and defines no timeout. The nearest it comes is:

RECOMMENDATION 3.10 — Design INTERCON modules to prevent deadlock. One solution is a watchdog timer function that monitors the MASTER's STB_O signal.

A recommendation, about an interconnect, watching STB_O. The inspected code is one implementation of it, installed by InterconnectShared by default with timeout_cycles=1e6.

Answering with ACK rather than ERR is the choice worth thinking about. It un-hangs a master that does not implement ERR_I at all — which matters, because OBSERVATION 3.35 warns that a slave supporting ERR_O against a master that does not may deadlock. The price is that the transfer looks successful on the bus, and the only things distinguishing it are the all-ones data and a wire that is not part of the interface.

Reconstruction, beside the source

Every measurement in this chapter comes from a compact Verilog model, not from running LiteX. The distinction matters enough to show it:

  assign wait_now = cyc_i && stb_i && !ack_i;
  assign done     = (cnt_q >= CYCLES[15:0]);

  assign ack_o      = done ? 1'b1 : ack_i;
  assign dat_o      = done ? {DW{1'b1}} : dat_i;
  assign error_o    = done;

VLSI MENTOR RECONSTRUCTION — wb_lx_timeout.sv. Preserves the count condition, the synthesised ACK, the all-ones data and the out-of-band error wire. Omits LiteX's WaitTimer structure, its signal layout, and everything else in the file. No byte-equivalence to LiteX source is claimed or intended.

Measured against a target that never answers:

=== LAB C - THE FRAMEWORK TIMEOUT POLICY ===

  one master, two targets. The second never answers.
  operations retired      2 of 2
  watchdog expiries       2
  data returned to the master on the second operation
    0xffffffff

OUR MEASUREMENT. Both operations "completed". One of them never reached a slave.

6. Shared Or Crossbar Is Nine Lines Apart

        # decode each master into its access row
        for row, master in zip(access, masters):
            row = list(zip(matches, row))
            self.submodules += Decoder(master, row, register)
        # arbitrate each access column onto its slave
        for column, bus in zip(zip(*access), busses):
            self.submodules += Arbiter(column, bus, mode=arbiter)

ORIGINAL SOURCE EXCERPT — same file, class Crossbar.

Same two building blocks as InterconnectShared. Opposite order, and one instance each versus N and M. That is the entire structural difference, and it is why a crossbar can have two transfers in flight: there is no single shared interface for them to queue on.

What that is worth is a measurement, not a property:

  DISJOINT destinations: M0 -> S0 only, M1 -> S1 only
  topology   M0 ops  M1 ops  concurrent clocks  clocks to finish both
  shared          4       4                  0                    25
  crossbar        4       4                  4                    21

  SAME destination: both masters -> S0
  topology   M0 ops  M1 ops  concurrent clocks  clocks to finish both
  shared          4       4                  0                    25
  crossbar        4       4                  0                    25

OUR MEASUREMENT. Chapter 29.3 takes that number apart.

7. The Local Policies, Listed

Nothing in this list is Wishbone. Every one is a decision the inspected framework made, and every one would be a legitimate thing for a different framework to decide differently:

• that a deselected slave is deselected by gating CYC alone;
• that terminations are ORed across slaves with no provenance latch;
• that read data is a one-hot AND-OR mux, registered or not;
• that the arbiter is round-robin rather than priority;
• that its grant may move mid-transfer by default;
• that a watchdog is installed by default at all;
• that the watchdog answers ACK rather than ERR;
• that it returns all ones rather than zeros or the last value;
• that the failure is reported on a wire outside the bus.

The last three together are the sharpest example of local policy shaping what a bug looks like. A system built on this interconnect that ignores the error wire will see timed-out accesses as successful reads of 0xFFFFFFFF — which is a perfectly plausible register value, and therefore a silent failure that looks like data.

8. What Not To Generalize

Do not say "LiteX uses a timeout so Wishbone has one." The framework implements RECOMMENDATION 3.10. The protocol has no timeout.

Do not assume the defaults. mode="cycle" and timeout_cycles=1e6 are what the inspected constructor signature says; a given SoC may pass something else, and we did not inspect one.

Do not assume a slave written for another system drops in. The decoder's CYC-only gating means every slave sees every strobe in the SoC. That is legal and it is an integration requirement.

Do not read this as a statement about deployment. We inspected a framework's interconnect module. That is evidence about the code, and about nothing else.

9. What To Carry Forward

• A generator still has to answer every question a hand-written interconnect answers. Find where it answers them.
• CYC-only gating is a legal optimisation with an integration obligation attached to every slave.
• Live-grant response steering appears in real frameworks — together with the parameter that fixes it.
• A watchdog that answers ACK un-hangs the master and hides the failure in the same stroke. Know which wire tells you.
• Shared and crossbar are the same two blocks in the opposite order.

Chapter 29.3 sets three projects side by side, including one that is not Classic at all.