fork-join

Module 14 · Page 14.1

fork-join is the fundamental tool for running multiple threads in parallel and waiting for all of them to finish. It turns a sequential testbench into a concurrent one — cutting simulation time and enabling stimulus patterns that are simply impossible without parallelism. The trap is the loop-variable capture bug: silently produces wrong results in every parallel loop a junior writes. This page walks the basic semantics, the three join variants, the variable-scope rules, the automatic-task discipline, and the canonical patterns.

1. Engineering Problem — Why fork-join Exists

A real DUT has 4 AXI channels (AW / W / B / AR / R), 8 PCIe lanes, 16 DDR banks, and at least one debug interface — all alive at the same instant. A testbench that drives them sequentially does not exercise the design; it drives one channel, idles 3, drives the next, idles 3, and so on. Coverage stays at single digits; cross-channel arbitration is never exercised; protocol races between concurrent channels never fire.

Without fork, you have two options, both bad:

• One initial block per channel — works but cannot coordinate end-of-test, cannot share IPC, cannot launch dynamically based on the test scenario.
• An always block per channel — works for steady-state stimulus but cannot model finite-length scenarios, cannot return, cannot be disabled cleanly.

fork-join is the IEEE 1800 answer: spawn an arbitrary set of named threads from inside a procedural block, run them all at the same simulation time, and block the parent until every one of them finishes. The total wall-clock time of the parent becomes max(thread_durations), not their sum.

2. Mental Model — Barrier-Launch, Barrier-Wait

The picture every engineer carries:

fork is a barrier-launch: every statement inside it springs into a parallel thread at the same simulation instant. join is a barrier-wait: the parent thread suspends and waits for every spawned thread to finish before crossing the barrier and resuming.

Three invariants this picture preserves:

• Same start, different finish. All threads begin at the same simulation time. They run independently — different delays, different events, different external signals — and finish whenever their own work completes.
• Parent is paused, not killed. The parent suspends at join and consumes no scheduling work while waiting. When the last thread finishes, the parent resumes in the same simulation time the last thread completed in.
• Local variables are shared. Every thread sees the same enclosing-scope variables. This is the source of the loop-capture bug in §6 — the rule that catches every junior writing their first parallel loop.

3. Visual Explanation — Three Threads, One Barrier

The figure captures the canonical pattern. All three threads start at the same instant (the fork bar); each runs independently; the parent waits at the join bar until every thread crosses it. The JOIN variant is the strictest of the three — join_any (next page 14.2) and join_none (page 14.3) relax that wait in different ways but share the same fork-bar semantics.

4. Syntax & Semantics — fork, join, Named Blocks

4.1 The basic form

// ── Syntax ────────────────────────────────────────────────────
fork
    statement_or_block_1 ;
    statement_or_block_2 ;
    // ... any number of threads ...
join
// parent resumes HERE — after ALL threads are done
 
// ── Three concurrent tasks, parent waits for all ──────────────
$display("[MAIN] Launching 3 threads at t=%0t", $time);
fork
    begin : thread_a
        #20;
        $display("[A] done at t=%0t", $time);
    end
    begin : thread_b
        #30;
        $display("[B] done at t=%0t", $time);
    end
    begin : thread_c
        #10;
        $display("[C] done at t=%0t", $time);
    end
join
$display("[MAIN] All threads done at t=%0t", $time);
// Output:
//   [MAIN] Launching 3 threads at t=0
//   [C] done at t=10
//   [A] done at t=20
//   [B] done at t=30
//   [MAIN] All threads done at t=30   ← resumes after the LAST thread (B)

4.2 The three join variants

SystemVerilog has three forms — they differ only in when the parent resumes. The other two are covered in their own pages.

4.3 Single-statement vs multi-statement threads, and named blocks

// ── Single-statement threads: no begin-end needed ─────────────
fork
    drive_reset();          // task call — one thread
    init_memory();          // task call — another thread, starts at the same time
    #100 check_clocks();    // statement with delay — starts after 100 ns
join
// All three run in parallel; parent waits for all three.
 
// ── Multi-statement blocks ────────────────────────────────────
fork
    begin
        $display("[CH0] Start");
        drive_channel_0();
        $display("[CH0] Done");
    end
    begin
        $display("[CH1] Start");
        drive_channel_1();
        $display("[CH1] Done");
    end
join
 
// ── Named blocks — required for selective disable ─────────────
fork
    begin : blk_a
        @(posedge clk); drive_a();
    end
    begin : blk_b
        @(posedge clk); @(posedge clk); drive_b();
    end
join
// Named blocks (blk_a, blk_b) allow `disable blk_a` to kill just one thread

Name every thread you might want to disable, debug, or reference from wait fork — the names show up in simulator logs and in Verdi's thread browser, and they make disable targeted instead of nuclear.

5. Simulation View — How the Scheduler Handles fork-join

At the fork keyword, the simulator:

1. Spawns each statement inside the block as a separate process, sharing the parent's local variables.
2. Schedules all spawned threads at the current simulation time in the Active region. Their relative order in the Active region is not specified by IEEE 1800 — see 13.1 — IPC and the scheduling regions for the consequences.
3. Suspends the parent process at the join keyword. The parent consumes no scheduler work while suspended.
4. Resumes the parent in the Active region of the time slot in which the last thread completes. "Last" is by $time, not by source order.

A fork-join with no threads (fork join) is a legal no-op — the parent passes through without suspending. A fork-join with one thread blocks the parent for that thread's duration; semantically equivalent to the thread body inline, but with a separate scheduler frame.

disable fork is executed by a process and terminates that process's active descendants — the threads it spawned, and their descendants in turn. Its scope is the calling process; it is not a statement you aim at someone else's fork from outside. disable <named_block> instead terminates exactly the named thread. Both statements are developed in 14.4 — disable fork & wait fork.

6. Waveform — Parent Blocked Until the Last Thread

The waveform below shows the three-thread example from §4.1. Threads A, B, C all start at cycle 0. C finishes at cycle 10, A at cycle 20, B at cycle 30. The parent is blocked through the entire window — it consumes no scheduler work, fires no $display, drives no signals — until cycle 30, the instant B finishes. The parent then resumes in the same time slot.

7. Synthesis — Not Applicable

fork-join is a procedural simulation construct. It has no hardware footprint and no place in synthesisable RTL — synthesis tools reject it outright. This section is intentionally omitted; the topic does not warrant it.

The corresponding hardware idiom for "do these N things in parallel" is just an always @(*) or always_ff per concurrent block — the synthesisable equivalent is built into the RTL semantics, not a procedural primitive. fork-join is a testbench-only tool.

8. Verification View — Where fork-join Earns Its Keep

8.1 The loop-variable capture bug — the canonical first-fork failure

When you fork inside a loop, the spawned threads are scheduled to run concurrently — they do not necessarily execute immediately. And there is exactly one i for the whole loop, not one per iteration: the loop declares the variable once and every iteration writes that same storage. The forked thread captures a reference to that variable, not a snapshot of its value at spawn time. By the time the threads execute, the loop has run to completion and i holds its final value — so every thread reads that final value.

The "one variable, many threads" fact is the whole bug, and it does not hinge on lifetime. In a static scope (a module initial block) i is static; inside a class method or a task automatic it is automatic — but automatic lifetime is per call, not per iteration, so a single call's loop still shares one i across all of its iterations and the bug reproduces identically. Lifetime is covered in static vs automatic.

for (int i = 0; i < 4; i++) begin
    fork
        begin
            #10;
            $display("Thread %0d done", i);
        end
    join_none
end
// Output:
//   Thread 4 done
//   Thread 4 done
//   Thread 4 done
//   Thread 4 done
// Every thread reads i=4 (the post-loop value).

for (int i = 0; i < 4; i++) begin
    automatic int idx = i;   // NEW per iteration
    fork
        begin
            #10;
            $display("Thread %0d done", idx);
        end
    join_none
end
// Output:
//   Thread 0 done
//   Thread 1 done
//   Thread 2 done
//   Thread 3 done

8.2 The automatic task discipline

When a forked thread calls a task, the task's local variables must be automatic if the same task may be called from multiple concurrent threads. Without automatic, all concurrent calls share the same storage — a race.

// ── WRONG: static task — all concurrent calls share local variables
task drive_channel(int ch);     // static by default in a module
    int count = 0;              // SHARED by ALL concurrent calls!
    repeat (10) begin
        @(posedge clk);
        count++;
        $display("ch=%0d count=%0d", ch, count);
    end
endtask
 
// ── CORRECT: automatic — each call has its own locals ─────────
task automatic drive_channel(int ch);
    int count = 0;              // PRIVATE to this call instance
    repeat (10) begin
        @(posedge clk);
        count++;
        $display("ch=%0d count=%0d", ch, count);
    end
endtask
 
initial fork
    drive_channel(0);   // ch=0, count is private
    drive_channel(1);   // ch=1, count is private
    drive_channel(2);   // ch=2, count is private
join

8.3 Nested fork-join for hierarchical parallelism

A fork-join block can appear inside another fork-join. Inner threads must all complete before the inner join releases control back to the outer thread that launched it.

fork
    begin : channel_a_init
        fork
            configure_phy_a();
            configure_mac_a();
        join                       // inner join: wait for both PHY and MAC
        enable_channel_a();        // runs only after both sub-tasks complete
    end
    begin : channel_b_init
        fork
            configure_phy_b();
            configure_mac_b();
        join
        enable_channel_b();
    end
join                               // outer join: wait for both channels
$display("[MAIN] Both channels initialised");

Both channels initialise in parallel. Within each channel, PHY and MAC configure simultaneously. enable_channel_* runs only after both sub-tasks of that channel finish. The outer parent waits until both channels are fully enabled.

8.4 Parallel multi-channel stimulus — the canonical pattern

parameter N_CH = 8;
 
initial begin
    for (int i = 0; i < N_CH; i++) begin
        automatic int ch = i;          // MUST be automatic
        fork
            drive_channel(ch);
        join_none                       // spawn without waiting
    end
 
    // ... other test setup ...
 
    wait fork;                         // wait for ALL spawned threads here
    $display("[TB] All channels done at t=%0t", $time);
end

fork ... join_none inside the loop spawns each channel-driving thread and returns immediately. wait fork later in the parent waits for every previously-spawned thread to finish — even threads that were join_none-ed earlier.

8.5 A runnable proof — all three facts, checked by the simulator

Everything above is checkable. The module below is self-contained and compilable; it reproduces the capture bug deterministically, shows the loop-body automatic fixing it, and measures the join barrier — then reports [PASS] or [FAIL] on its own.

// ─────────────────────────────────────────────────────────────────────────
//  Proves three fork-join facts by construction rather than by assertion:
//    1. a fork body that reads the loop variable sees its FINAL value
//    2. an `automatic` declared in the LOOP BODY captures per iteration
//    3. `join` releases the parent at max(thread durations), not the sum
//
//  vcs -sverilog fork_join_proof.sv && ./simv
//  xrun -sv fork_join_proof.sv
//  vlog fork_join_proof.sv && vsim -c fork_join_proof -do "run -all; quit"
// ─────────────────────────────────────────────────────────────────────────
module fork_join_proof;
 
  timeunit      1ns;
  timeprecision 1ps;
 
  localparam int N = 4;
 
  int reported_shared [N];   // what each thread read from the SHARED loop var
  int reported_copy   [N];   // what each thread read from its OWN copy
  int errors = 0;
 
  // ── 1. The bug: the thread body reads the loop variable itself ─────────
  //  `slot` is a per-iteration copy used ONLY to choose an array index, so the
  //  array writes cannot race and the single variable under test is the one
  //  the thread reports. That isolates the defect to exactly one expression.
  task automatic spawn_reading_shared();
    for (int i = 0; i < N; i++) begin
      automatic int slot = i;
      fork
        begin
          #10;                        // runs after the loop has finished
          reported_shared[slot] = i;  // BUG: reads the shared loop variable
        end
      join_none
    end
    wait fork;   // waits for the children THIS process spawned
  endtask
 
  // ── 2. The fix: capture into an `automatic` in the LOOP BODY ───────────
  task automatic spawn_reading_copy();
    for (int i = 0; i < N; i++) begin
      automatic int slot = i;
      automatic int idx  = i;         // fresh copy — one per iteration
      fork
        begin
          #10;
          reported_copy[slot] = idx;  // reads its OWN copy
        end
      join_none
    end
    wait fork;
  endtask
 
  // ── 3. `join` releases the parent at max(durations), not the sum ───────
  task automatic three_threads_then_join();
    fork
      #20;   // thread A
      #30;   // thread B — the longest
      #10;   // thread C
    join
  endtask
 
  initial begin : proof
    int t_start, t_elapsed;
 
    // ---- fact 1: every thread reports the post-loop value ---------------
    spawn_reading_shared();
    for (int k = 0; k < N; k++) begin
      $display("[%0d ns] spawned as %0d, thread read shared i = %0d",
               $time, k, reported_shared[k]);
      if (reported_shared[k] !== N) begin
        errors++;
        $display("   ** CHECK FAILED: expected the post-loop value %0d", N);
      end
    end
 
    // ---- fact 2: each thread reports its own captured index -------------
    spawn_reading_copy();
    for (int k = 0; k < N; k++) begin
      $display("[%0d ns] spawned as %0d, thread read its own copy = %0d",
               $time, k, reported_copy[k]);
      if (reported_copy[k] !== k) begin
        errors++;
        $display("   ** CHECK FAILED: expected %0d", k);
      end
    end
 
    // ---- fact 3: the join barrier costs max(20,30,10) = 30, not 60 ------
    t_start   = $time;
    three_threads_then_join();
    t_elapsed = $time - t_start;
    $display("[%0d ns] fork-join of #20/#30/#10 took %0d ns (the max, not the 60 sum)",
             $time, t_elapsed);
    if (t_elapsed !== 30) begin
      errors++;
      $display("   ** CHECK FAILED: expected 30");
    end
 
    if (errors == 0) $display("\n[PASS] all fork-join checks held");
    else             $display("\n[FAIL] %0d check(s) failed", errors);
    $finish;
  end
 
endmodule

Expected output.

[10 ns] spawned as 0, thread read shared i = 4
[10 ns] spawned as 1, thread read shared i = 4
[10 ns] spawned as 2, thread read shared i = 4
[10 ns] spawned as 3, thread read shared i = 4
[20 ns] spawned as 0, thread read its own copy = 0
[20 ns] spawned as 1, thread read its own copy = 1
[20 ns] spawned as 2, thread read its own copy = 2
[20 ns] spawned as 3, thread read its own copy = 3
[50 ns] fork-join of #20/#30/#10 took 30 ns (the max, not the 60 sum)
 
[PASS] all fork-join checks held

What the log proves. The first four lines are the capture bug, and note that they are not flaky — every thread reads 4 on every run and every simulator, because they are all reading one variable that the loop left at its final value. That is what makes the bug so dangerous: it is perfectly reproducible and perfectly wrong, so it never looks like a race. The next four lines show the loop-body automatic restoring per-thread identity. The last line measures the barrier: 30 ns of wall-clock for 60 ns of thread work — the whole reason fork-join exists.

9. Industry Usage — Where fork-join Lands in Real Verification

• UVM phase machinery — uvm_test::run_phase() runs as a thread; each agent's run_phase() spawns its driver / monitor / sequencer threads via fork-join_none internally; the test waits on phase.raise_objection / drop_objection rather than join directly, but the underlying spawn mechanism is the same.
• UVM uvm_sequence::do_pre_post — pre / post / body fork patterns inside the sequencer use fork-join_any and fork-join_none heavily.
• Multi-protocol VIPs — every AXI / AHB / APB / DDR / PCIe / Ethernet agent spawns separate driver / monitor threads; the test's run_phase fork-joins the whole agent tree.
• Reset / clock-gating tests — the test's fork-join runs stimulus, reset injector, and gating sequencer in parallel; the test waits for all three to complete before scoring.
• Coverage-closure tests — multiple seeds run as parallel fork-join_none threads inside a single simulation when the verification environment supports it.
• Regression infrastructure — when one simulation runs N tests sequentially (e.g. checkpoint / restart flows), each test is a fork ... join block with a watchdog timeout in the parent.

10. Design Review Notes — What a Senior Will Flag

The single highest-value rule: automatic is not optional for fork-loops. Every shop's coding-style guide mandates automatic int idx = i; (or for (automatic int i = ...)) — and most ship a lint check for the violation.

11. Debugging Guide — Real Failures, Real Fixes

for (int i = 0; i < 4; i++) begin
    fork
        begin
            drive_channel(i);    // reads i when it runs, not when forked
        end
    join_none
end

task drive_channel(int ch);          // static by default!
    int count = 0;                   // SHARED by all concurrent calls
    repeat (10) begin
        @(posedge clk);
        count++;
        $display("ch=%0d count=%0d", ch, count);
    end
endtask
 
initial fork
    drive_channel(0);
    drive_channel(1);
    drive_channel(2);
join

initial begin
    for (int i = 0; i < 8; i++) begin
        automatic int ch = i;
        fork
            drive_channel(ch);       // spawned, not waited on
        join_none
    end
    #1000;                           // arbitrary
    $finish;                         // kills every still-running thread
end

fork
    begin : channel_work
        for (int i = 0; i < N_CH; i++) begin
            automatic int ch = i;
            fork drive_channel(ch); join_none   // children of channel_work
        end
        wait fork;                              // waits for THOSE children
    end
    begin : watchdog
        #100_000;
        $fatal(1, "[TB] channels did not finish within 100us");
    end
join_any
disable fork;    // kill whichever branch is still alive
$finish;

initial begin
    fork
        drive_stimulus();
        wait_for_dut_done();        // never asserts if DUT is broken
        sample_coverage();
    join                            // blocks forever on wait_for_dut_done
end

Txn shared_txn = new();
fork
    begin
        shared_txn.randomize();
        drive_to_dut(shared_txn);     // reads shared_txn
    end
    begin
        shared_txn.randomize();       // ALSO writes shared_txn concurrently!
        push_to_scoreboard(shared_txn);
    end
join

12. Interview Insights — What Interviewers Actually Probe

13. Exercises

1. Design — N parallel channel drivers (Foundation)
Write a for loop over N_CHANNELS = 8 that spawns one drive_channel(ch) thread per channel. Use the correct automatic capture pattern. Add a wait fork; $finish; at the end of the parent to ensure no channel is killed mid-flight.

2. Debug — the channel-4-only bug (Intermediate)
A teammate's parallel-channel test only drives stimulus on channel 4 of 4 — channels 0–3 never see traffic. The code is for (int i = 0; i < 4; i++) fork drive(i); join_none end. Walk through the diagnostic, name the bug, and write the one-line fix.

3. Code review — the static task (Intermediate)
A teammate submits a task drive_protocol(int ch); with a local int seq_num = 0; and uses it as initial fork drive_protocol(0); drive_protocol(1); drive_protocol(2); join. Identify the bug, predict the symptom, and write the fix.

4. Trade-off — fork-join everywhere vs fork-join_none + wait fork (Advanced)
A junior teammate argues "always use fork-join — it's the safest because the parent blocks until everything finishes; you never lose work." Argue the case against the blanket rule. When does fork-join_none + wait fork legitimately beat the all-join style, and what specifically does it enable that pure join cannot?

14. Summary

fork spawns parallel threads from inside a procedural block; join blocks the parent until every spawned thread completes. The parent's wall-clock duration is the longest single thread, not the sum. The single failure mode every junior ships once is the loop-variable capture bug — fix with automatic int idx = i; before the fork. Tasks called from concurrent fork branches must be declared automatic. Always name every fork branch you might want to disable or debug.

Defaults to memorise. fork ... join waits for all. automatic is mandatory for fork-loops and for tasks called from forks. wait fork is the parent-side counterpart to fork ... join_none — without it, threads get killed mid-flight by $finish. Wrap the test-level fork-join in fork ... join_any with a watchdog to prevent silent hangs.

Next up: 14.2 — fork-join_any — the variant where the parent resumes as soon as any one thread completes. The basis for the canonical timeout pattern and the race-detect pattern.

15. Where This Is Specified

Parallel blocks are defined in IEEE Std 1800 (SystemVerilog), clause 9 — Processes: fork ... join, join_any, and join_none in §9.3.2 (parallel blocks), and the process-control statements wait fork and disable fork in §9.6. The variable-lifetime rules that make the loop-capture bug possible — and that make the loop-body automatic the fix — are in §6.21 (scope and lifetime); the static-vs-automatic default for tasks and functions is in clause 13. The IEEE Standards Association listing is the primary source.

Two clauses are worth reading directly rather than reconstructing from memory, because both are commonly misremembered: §9.6, for the fact that wait fork waits on the calling process's children (which is why a wait fork placed in a sibling fork branch silently waits for nothing), and §6.21, for the fact that automatic lifetime is per call, not per iteration.

Related lessons. The other two join flavours are 14.2 — fork-join_any and 14.3 — fork-join_none; the process-tree statements get their own treatment in 14.4 — disable fork & wait fork. The lifetime rules behind the automatic discipline are in static vs automatic. For the shared-state races that concurrency creates once the threads are running, see semaphores for mutual exclusion, mailboxes for handing data between threads, and 13.1 — IPC and the scheduling regions for why same-region ordering is unspecified in the first place.