Firmware-Driven Tests

Part II · Module 18 · Chapter 18.4 · Firmware-Driven Tests

1. Software as the Stimulus

A firmware-driven test is a test written as the software the product will run, executing on the processor inside the design — or on a model of it — while the simulation runs.

The testbench may not be involved at all. It sets up the memory image, releases reset, and waits; the design's own core fetches instructions and issues bus transactions, and those transactions are the stimulus.

Chapter 18.1 §4 showed the shape against the peripheral model from Chapter 16.1:

after init : CTRL = 0x00203e8b
firmware read : "Hi" (2 bytes)
pending before ack : 0x00000005
pending after  ack : 0x00000004   <- bit 2 preserved

bus transactions issued by the firmware: 6 reads, 2 writes

Every line of that firmware is Part II material used unchanged — fixed-width types, field accessors, read-modify-write, a single read of a read-to-clear register, a write-one-to-clear acknowledgement. The code is not special. What is special is who runs it and what it is trying to do.

2. What Firmware Does That a Test Does Not

A directed test is written to exercise a feature. It sets up a condition, performs an operation, checks a result, and finishes.

Firmware is written to achieve something — open a port, transfer a buffer, handle an interrupt, reconfigure on the fly. It does whatever that takes, in whatever order the software's own logic requires, and it keeps running afterwards.

That difference produces different stimulus, and the difference is not realism in the vague sense. It is specific:

Firmware produces sequences. It configures, then uses, then reconfigures, then uses again — because that is what software does. A directed test usually configures once.

Firmware leaves state behind. The next operation starts from wherever the last one finished, not from reset.

Firmware crosses blocks. A single transfer may touch a clock controller, a pin multiplexer, a DMA engine and a UART, in an order the software's architecture dictates and that nobody enumerated.

Firmware repeats. The same operation happens hundreds of times with different data, and the hundredth time starts from the state the ninety-ninth left.

3. The Measurement, and the Version That Was Backwards

Here is a design defect with no special properties except one: it is invisible to any test that performs a single configuration.

The specification says the baud divisor may be changed at any time. The design latches it only on the first write while the device is enabled; later writes are silently ignored until it is disabled again.

Two directed tests, each configuring from reset:

=== a directed test: configure once, check ===
  set divisor 1000 -> 1000   PASS (bug invisible)

=== a second directed test: a DIFFERENT value, from reset ===
  set divisor 2400 -> 2400   PASS (bug still invisible)

Both pass. Between them they have written two different values and read back exactly what was written. A coverage report would show the field exercised.

Now firmware, doing what a driver does when an application changes the baud rate:

=== firmware: open the port, then change the baud rate ===
  1000 then 2400   -> 1000   FAIL  <- the bug, reached only by a SEQUENCE

The second write did nothing. The port is still running at the first rate, and nothing reported an error — Chapter 16.6 §2 measured that a register need not read back what was written, and here it reads back the old value, which is worse than either.

No single operation in that sequence is unusual. Write CTRL. Write CTRL again. What exposes the bug is that the second write follows the first.

4. The Class, Precisely

Firmware-driven tests reach bugs whose trigger is a relationship between operations, rather than a property of any single operation.

Four shapes, all of which fit that description and none of which a per-operation test enumerates:

Ordering. Operation B behaves differently depending on whether A happened first. §3's divisor is this.

Repetition. The second occurrence differs from the first — a write-once latch, a FIFO that does not reset a pointer, an interrupt that only arms once.

Residual state. An operation leaves something behind that the next one inherits. A directed test starting from reset never sees it.

Cross-block interleaving. Two blocks interact through an order that only a real software flow produces — a clock enabled after a peripheral was configured, a DMA descriptor written while a transfer is in flight.

The reason a directed test misses these is not carelessness. A directed test enumerates operations; the number of operations is manageable and the number of sequences is not. With fifty registers and two operations each there are a hundred directed tests and ten thousand ordered pairs. Firmware does not enumerate sequences either — it performs the specific ones that real software performs, which is a much smaller and much better-chosen set than random ordering would give.

That is the whole argument, and it is an argument about which sequences, not about quantity.

5. What Firmware-Driven Tests Need

Three things, and the third is the one people underestimate.

A way to run the software. The processor must be in the simulation — a real core, a model of one, or a bus master that replays transactions. Chapter 18.5 is how C gets into a simulation at all.

A way to see what happened. Firmware running inside a simulation has no printf. The usual mechanisms are a memory-mapped output register the testbench monitors, a known memory location the testbench reads at the end, or a simulator interface call. Choosing one is an early decision and changing it later is painful.

A way to decide pass or fail. This is the hard part. A directed test knows what it expects; firmware is trying to achieve something and may not notice that it failed. §3's divisor bug produced no error at all from the firmware's point of view — the write returned, the device kept running, and the baud rate was wrong. Something outside the firmware had to check.

That third point is the honest limit of the technique, and it shapes how firmware tests are used in practice: the firmware provides the stimulus, and a monitor, a scoreboard or a reference model provides the check. Chapter 18.2's model is the natural partner.

6. What They Are Worse At

Four costs, stated plainly because the technique is usually oversold.

A failure is a weak signal. A directed test that fails names the operation that failed. A firmware test that fails tells you something went wrong somewhere in a long sequence, and localising it is a separate investigation. Chapter 13.9's bisection method applies, and it costs time a directed test would not have cost.

They are slow. Executing real software in a simulation means simulating the processor as well as the design. A sequence that takes microseconds on silicon can take hours.

They are poor at corner values. Firmware uses the values the product uses. A directed test can set a field to zero, to its maximum, and to maximum plus one; firmware sets it to 115200 baud.

They need the software to exist. Early in a project it does not, and writing firmware purely as a test is a different activity from reusing the product's own code — and loses the main advantage, which is that the code is the code that ships.

So the sensible position is not a replacement:

Directed tests enumerate operations; firmware exercises sequences; a reference model checks both. Each is bad at what the others are for.

7. The Misconception: Firmware Tests Are More Realistic, So They Are Better

"Running the real software against the RTL is the most realistic test there is, so it is the best test."

Realism and diagnostic value are different properties, and the technique trades one for the other.

A firmware test is more realistic and a worse signal when it fails — §6's first cost. It is more realistic and worse at corners, because the product does not use corner values. It is more realistic and slower by orders of magnitude.

The thing it is genuinely better at is the one §4 named, and it is worth being precise rather than enthusiastic: sequences that real software performs. Not all sequences — real ones. That is a narrow and valuable claim, and it does not generalise into "therefore the best test".

The failure mode of believing the broad version is a project that defers directed testing in favour of bringing up firmware, discovers integration bugs late, and has no fast way to localise any of them.

The accurate statement:

Firmware-driven tests find a class of bug that per-operation tests structurally cannot reach, and they are the worst available tool for diagnosing anything. Use them for the first property and keep the other tools for the second.

8. What You Should Be Able to Say Now

1. Firmware-driven testing uses the product's own software as the stimulus, executed by the design's own processor. The testbench may not be involved.
2. The code is not special — 18.1's firmware used Part II's types, accessors, read-modify-write and access-type discipline unchanged.
3. Firmware produces sequences, leaves state behind, crosses blocks and repeats, because it is trying to achieve something rather than to test something.
4. Measured: two directed tests set a divisor to 1000 and 2400 from reset and both passed; firmware setting 1000 then 2400 left it at 1000.
5. My first demonstration was backwards — that defect was caught by the directed test and missed by firmware. "Sequence-dependent" is not the same as "only firmware finds it".
6. The class, precisely: bugs whose trigger is a relationship between operations — ordering, repetition, residual state, cross-block interleaving.
7. The reason is combinatorial: operations are enumerable and sequences are not. Firmware performs the real ones, which is a better-chosen set than random ordering.
8. They need three things: a way to run the software, a way to observe it, and — hardest — a way to decide pass or fail. §3's bug produced no error from the firmware's point of view.
9. Four costs: a weak failure signal, slow execution, poor corner coverage, and the software has to exist.
10. Not a replacement. Directed tests enumerate operations, firmware exercises sequences, a model checks both.

9. Exercises

1. Reproduce §3's three runs. Then add a fourth: disable, re-enable, and set the divisor again. Predict the result from the defect's description before running.
2. Write a directed test that would have caught §3's bug. How many such tests would you need to cover the ordered pairs of two operations on five registers?
3. Take the firmware from Chapter 18.1 §4 and extend it to re-configure the device mid-run. Confirm against the Chapter 16.1 model what happens, and count the transactions.
4. For each of §4's four shapes, write a one-sentence example from a block you have worked on.
5. §5 says deciding pass or fail is the hard part. For §3's bug, describe two mechanisms that would have caught it — one inside the firmware and one outside — and say which you would build.
6. A project proposes replacing its directed register tests with firmware tests. Using §6, write the three-sentence reply you would send.
7. §3's callout reports an experiment that contradicted its own premise. Describe how you would have noticed if the chapter had been written first.

10. Summary

• Firmware-driven testing = the product's software as stimulus, run by the design's processor.
• The C is ordinary — Part II's discipline, unchanged.
• Firmware sequences, repeats, inherits state and crosses blocks because it is achieving something, not testing something.
• Measured: two directed tests passed (1000, 2400 from reset); firmware's 1000-then-2400 left 1000.
• My first demo was backwards — the directed test caught that defect and firmware missed it. Run the experiment before writing the claim.
• The class: a trigger that is a relationship between operations — ordering · repetition · residual state · cross-block interleaving.
• Operations are enumerable; sequences are not. Firmware performs the real ones.
• Needs: a way to run it · a way to observe it · a way to decide pass/fail — the hard one.
• Costs: weak failure signal · slow · poor corners · the software must exist.
• Not a replacement — three tools, each bad at what the others are for.

All three categories so far — a model, an adopted oracle, firmware — have assumed the C can be reached from, or can reach, the simulation. That is a separate problem with several answers, and the one everybody names first is neither the simplest nor the most common.