Why Coherent Attach Matters

Chapter 1.6 ended on a precise boundary: the device-I/O model stops where two agents need a shared, hardware-maintained view of the same state rather than a sequence of explicit transfers between two private ones.

This chapter is about providing that view. It is the payoff of Module 1, and it would be easy to write badly — as an advertisement for coherence. Coherence is not free, and an engineer who leaves thinking it is will design a system that deadlocks. So the chapter has two halves of equal weight: what agreement buys, and what maintaining it costs.

1. The One-Sentence Model

Coherence is a hardware-maintained agreement about the current value of an address, enforced by serialising every transition through one point and by ensuring that before any agent may write, every other cached copy has been invalidated and that invalidation acknowledged. It removes an entire class of software obligation, and it pays for that with state, traffic, protocol complexity, latency, and new failure modes that are hangs rather than wrong answers.

The mechanism reduces to one invariant, and the whole chapter orbits it:

SINGLE WRITER:  at most one agent may hold a writable copy of a line,
                and it may not begin writing until every other copy is gone

2. What This Chapter Owns

Nothing here is a CXL protocol claim. The RTL is a pedagogical model of a mechanism class, not of any specification.

3. Start With the Failure

Forget protocols. Here is a program.

1.  CPU reads X          → CPU caches X = 5
2.  Device reads X       → device holds its own copy, X = 5
3.  Device writes X = 9  → device's copy is now 9
4.  CPU reads X          → CPU hits its cache and reads 5

Step 4 is wrong, and nothing anywhere reported an error. The CPU did not misbehave — it read its cache, which is what a cache is for. The device did not misbehave — it wrote the value it was asked to write. Every component behaved correctly and the program got a wrong answer, which is the signature of a missing agreement rather than a missing feature.

Ask the question that organises everything: which copy is authoritative? In step 4 there are two values for one address and no rule that says which one a reader should get. Coherence is exactly the rule, plus the machinery that enforces it.

Without such machinery, correctness has to be manufactured in software — flush before the device reads, invalidate before the CPU reads back, and a fence at every ownership transition. That works, it is what a great deal of production software does, and Chapter 1.6 §14 is what it looks like when one of the steps is missing. The PCIe track has measured the cost of exactly that omission in PCIe vs CXL.

4. Three Words That Get Conflated

This distinction is interview-critical and routinely fumbled. Keep the three separate.

Coherence is per-address. It says that all agents observing a single location see a single sequence of values for it, and that a write eventually becomes visible. It says nothing whatever about two different addresses.

Ordering is about relationships between operations. Whether a write to A becomes visible before a write to B is an ordering question, and it is not answered by coherence — a perfectly coherent system may make them visible in either order.

A consistency model is the contract software programs against: which orderings the hardware guarantees, and which the programmer must enforce with fences or atomics. It is the system-level rule that ordering guarantees add up to.

The compressed version worth memorising: coherence is about one address; a consistency model is about the relationship between many. A system can be fully coherent and still require fences, because coherence never promised anything about A relative to B.

That is also why the misconception in Section 17 — "coherence means no more synchronisation" — is wrong. Coherence removes explicit flush and invalidate. It does not remove the need to know that the producer has finished.

5. What Coherence Guarantees, Precisely

Three properties, stated carefully because over-claiming here is the most common error.

Write propagation. A write by one agent eventually becomes visible to the others. "Eventually" is doing real work: coherence bounds the outcome, not the latency.

Write serialisation. All agents observe writes to a single location in the same order. Two agents cannot disagree about whether X became 9 before or after it became 7.

A single-writer, multiple-reader discipline. At any moment a line has either one agent that may write it, or any number that may read it — never both. This is the invariant the RTL in Section 9 enforces, and it is what makes the other two achievable.

What coherence does not guarantee: that a read is fast, that a remote copy is as cheap as a local one, that operations to different addresses are ordered, or that software needs no synchronisation. Each of those is a separate concern, and conflating them with coherence is how systems get designed on false expectations.

6. What Coherent Attach Buys

Return to Module 1's accumulated list and apply the mechanism.

Copies stop being mandatory. Chapter 1.3 separated copies that the algorithm needs from copies forced by two domains that cannot address each other. If a device can address host memory coherently, the second category disappears — not because the bytes stop moving, but because the duplication stops being required.

A device may cache what it re-reads. Chapter 1.3 §13 derived that whether reaching across beats copying depends on locality at the far end. A device that can hold coherent cached copies gets that locality safely, which moves the crossover substantially in favour of sharing.

Fine-grained sharing becomes practical. The mismatch Chapter 1.6 §7 tabulated — cache-line units, implicit initiation, high frequency — is exactly what a coherence protocol is built for and exactly what a descriptor model is not.

An entire class of software obligation is deleted. No flush before the device reads. No invalidate before the host reads back. No ownership bit that both sides must honour by discipline. Three of the four Debug Labs in Chapter 1.6 were driver bugs of precisely this kind, and they stop being possible rather than becoming less likely.

Shared data structures become expressible. A work queue, a flag, a lock, a pointer-rich structure that both agents traverse — all impractical when every transition costs a maintenance operation on a whole buffer.

7. What Coherence Costs

Equal weight, because this is where designs fail.

State. Somebody must track which agents may hold which lines. Snoop filters and directories are large structures whose size scales with the memory being tracked and the number of agents tracking it.

Traffic. Invalidations and their acknowledgements are messages that would not otherwise exist, and they scale with sharing and with agent count. A widely-shared line is a fan-out event on every write.

Latency on the transition. A write to a line held elsewhere cannot begin until the invalidation round trip completes. Section 11's waveform shows that wait explicitly, and it is the reason a coherent write to a contended line is dramatically more expensive than an uncontended one.

Protocol complexity. States, transitions, races between concurrent requests to the same line, and the ordering rules that make them resolvable. This is where the specification pages go.

New failure modes, and they are a different kind. The bugs in Chapter 1.6 were wrong answers. The bugs here include hangs — an acknowledgement that never arrives blocks a writer indefinitely, as Debug Lab 2 measures. Deadlock and livelock become design concerns in a way they simply are not for a descriptor ring.

Verification cost. The state space is the product of per-agent states across agents, times the interleavings of concurrent requests. This is why coherence verification is a specialism.

8. The Mechanism, Minimally

Everything above reduces to three requirements, and they are what Section 9 builds.

A serialisation point. Concurrent requests for the same line must be ordered somewhere. Without one, two agents can both believe they are next, which is how two writers happen.

Invalidation fan-out. Before a write is permitted, every other agent holding a copy must be told to drop it.

Acknowledgement collection. The writer may not proceed until every one of those agents has confirmed. This is the part that is tempting to skip and catastrophic to skip.

request write  →  serialise  →  snoop all holders  →  collect every ack  →  grant
                                                            ↑
                                        the write waits HERE, and this wait
                                        is the latency cost of coherence

9. RTL — A Coherence Point

// A DELIBERATELY SIMPLIFIED coherence point for ONE address, shared by N
// agents. Per-agent state, two bits:
//   00 INVALID    the agent holds nothing
//   01 SHARED     the agent may read
//   10 EXCLUSIVE  the agent may write; at most ONE agent may be here
//
// The whole design exists to enforce one sentence: a write is granted only
// after every other cached copy has been invalidated and acknowledged.
module coherence_point #(
  parameter int unsigned N = 4
) (
  input  logic          clk,
  input  logic          rst_n,
  input  logic [N-1:0]  req_shared,
  input  logic [N-1:0]  req_excl,
  output logic [N-1:0]  grant_shared,
  output logic [N-1:0]  grant_excl,
  output logic [N-1:0]  snoop_inv,
  input  logic [N-1:0]  inv_ack,
  output logic [1:0]    state_q [N],
  output logic          busy_q,
  output logic [N-1:0]  awaiting_ack_q,
  output logic          two_writers_err,
  output logic          ack_without_snoop_err
);
  localparam logic [1:0] INVALID = 2'b00, SHARED = 2'b01, EXCLUSIVE = 2'b10;
  localparam int unsigned AW = (N > 1) ? $clog2(N) : 1;
 
  logic [AW-1:0] winner_q;
  logic          pending_q;
  logic [N-1:0]  excl_pick, holders;
  logic          any_exclusive;
 
  // One winner per cycle. Serialising requests to the same line is the whole
  // reason a coherence point exists -- two agents may not both be "next".
  assign excl_pick = req_excl & (~req_excl + 1'b1);
 
  always_comb begin
    holders = '0;
    any_exclusive = 1'b0;
    for (int unsigned i = 0; i < N; i++) begin
      holders[i] = (state_q[i] != INVALID);
      if (state_q[i] == EXCLUSIVE) any_exclusive = 1'b1;
    end
  end
 
  assign grant_shared = (busy_q || any_exclusive) ? '0 : req_shared;
 
  // THE invariant, in one expression: exclusive access is granted only when no
  // acknowledgement is still outstanding -- INCLUDING the ones arriving this
  // cycle. Qualifying on the registered vector alone never grants at all,
  // because the last acknowledgement and the promotion happen on one edge.
  assign grant_excl = (pending_q && ((awaiting_ack_q & ~inv_ack) == '0))
                        ? (N'(1) << winner_q)
                        : '0;
 
  assign snoop_inv = awaiting_ack_q;
  assign busy_q    = pending_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int unsigned i = 0; i < N; i++) state_q[i] <= INVALID;
      pending_q <= 1'b0; winner_q <= '0; awaiting_ack_q <= '0;
      two_writers_err <= 1'b0; ack_without_snoop_err <= 1'b0;
    end else begin
      // An acknowledgement from an agent that was never snooped means the
      // fan-out and the collection disagree about who was asked.
      if ((inv_ack & ~awaiting_ack_q) != '0) ack_without_snoop_err <= 1'b1;
 
      // Detect the state this design exists to prevent.
      begin
        int unsigned nexcl;
        nexcl = 0;
        for (int unsigned i = 0; i < N; i++)
          if (state_q[i] == EXCLUSIVE) nexcl = nexcl + 1;
        if (nexcl > 1) two_writers_err <= 1'b1;
      end
 
      if (!pending_q) begin
        if (!any_exclusive)
          for (int unsigned i = 0; i < N; i++)
            if (req_shared[i] && (state_q[i] == INVALID)) state_q[i] <= SHARED;
 
        // Start an exclusive transition: snoop every OTHER holder.
        if (req_excl != '0) begin
          pending_q <= 1'b1;
          for (int unsigned i = 0; i < N; i++)
            if (excl_pick[i]) winner_q <= AW'(i);
          awaiting_ack_q <= holders & ~excl_pick;
        end
      end else begin
        for (int unsigned i = 0; i < N; i++)
          if (awaiting_ack_q[i] && inv_ack[i]) state_q[i] <= INVALID;
        awaiting_ack_q <= awaiting_ack_q & ~inv_ack;
 
        // When the last acknowledgement lands, the writer is promoted.
        if ((awaiting_ack_q & ~inv_ack) == '0) begin
          state_q[winner_q] <= EXCLUSIVE;
          pending_q         <= 1'b0;
        end
      end
    end
  end
endmodule

Purpose, architectural role, state

The serialisation point of Section 8. State is two bits per agent, a pending flag, the winning agent's index, and the outstanding-acknowledgement vector — and that acknowledgement vector is the module. Everything else arranges for it to be correct.

Contract

At most one agent in EXCLUSIVE, ever. No exclusive grant while any acknowledgement is outstanding. A shared grant never occurs while an agent holds the line exclusively or while a transition is in flight.

Simulation evidence

Three agents read the line, then a fourth asks to write. Verbatim from the Icarus run, with per-agent state shown as I/S/E:

=== EXP1: three agents read the line (SHARED is not exclusive) ===
  after 3 shared reads       states=SSSI busy=0 snoop=0000 awaiting=0000
=== EXP2: agent 3 asks to WRITE -- invalidate then grant ===
  exclusive requested        states=SSSI busy=0 snoop=0000 awaiting=0000
  snoops issued              states=SSSI busy=1 snoop=0111 awaiting=0111
  agent0 acked               states=ISSI busy=1 snoop=0110 awaiting=0110
  agent1 acked               states=IISI busy=1 snoop=0100 awaiting=0100
  agent2 acked (last)        states=IIIE busy=0 snoop=0000 awaiting=0000
=== EXP3: a second writer must wait for the first to be invalidated ===
  agent0 requests write      states=IIIE busy=0 snoop=0000 awaiting=0000
  snooping the current owner states=IIIE busy=1 snoop=1000 awaiting=1000
  owner acked                states=EIII busy=0 snoop=0000 awaiting=0000
RESULT: PASS (single-writer held every cycle)

Read the state column down the page. SSSI — three readers, no writer. Then awaiting=0111 while the acknowledgements are collected, with the writer still I. Only when the last acknowledgement lands does it become IIIE: one writer, and every other copy gone. Then EXP3 moves ownership again, IIIE → EIII, by the same mechanism.

Synthesis

2N state flops, an N-bit outstanding vector, a pending flag, a winner index, a lowest-set-bit encoder and comparison logic. Small — because this models one address. The reason real coherence hardware is large is that it does this for every address, which is what a directory or snoop filter is.

A bug that only simulation found

The first version of this module computed the grant from the registered acknowledgement vector alone:

assign grant_excl = (busy_q && (awaiting_ack_q == '0)) ? ... : '0;   // never fires

That expression never asserts. The last acknowledgement and the promotion happen on the same clock edge, so by the time awaiting_ack_q reads zero, pending_q has already cleared. The design was functionally correct in its state transitions and its grant output was dead — a fault that reading the code does not reveal and a single simulated transition does.

10. The Ownership Transfer, Cycle by Cycle

What changes, and why. At t1 agent 3 requests exclusive access. At t2 the coherence point issues snoop_inv = 0111 to the three sharers and records awaiting = 0111. Each acknowledgement clears its bit: 0111 → 0110 → 0100 → 0000. At t5 the final acknowledgement arrives, grant_excl asserts in that same cycle, and at t6 the owner's state reads E.

Where the latency is. The four cycles from t2 to t5 are not overhead in the design's own logic — they are the round trip to three agents and back. This is the price of the agreement, it scales with how widely the line is shared, and it is why a contended coherent write costs dramatically more than an uncontended one. Section 14 is what that looks like at scale.

What a bug would look like. If grant_excl asserted at t2 rather than t5, the writer would proceed while three stale readers still held copies — Debug Lab 1, where the simulation shows states=SSSE with awaiting=0111. If one acknowledgement never arrived, awaiting would stop at a non-zero value and the grant would never come — Debug Lab 2, a hang rather than a wrong answer.

11. Assertions

Bind-ready properties. Icarus does not support concurrent assertions, so these were not executed; the table gives the procedural check that verified each.

// C1 — SINGLE WRITER. The invariant the whole design exists for. Everything
// else in this file is a way of protecting it.
a_single_writer: assert property (@(posedge clk) disable iff (!rst_n)
  $countones(excl_vector) <= 1);
 
// C2 — no exclusive grant while any invalidation is unacknowledged. This is
// the property Debug Lab 1 violates, and it is the one worth writing first.
a_grant_after_all_acks: assert property (@(posedge clk) disable iff (!rst_n)
  (grant_excl != '0) |-> ((awaiting_ack_q & ~inv_ack) == '0));
 
// C3 — a shared grant never coexists with an exclusive holder.
a_no_share_while_exclusive: assert property (@(posedge clk) disable iff (!rst_n)
  (grant_shared != '0) |-> !any_exclusive);
 
// C4 — acknowledgements only come from agents that were snooped. A violation
// means the fan-out and the collection disagree about who was asked, and the
// count can then reach zero early.
a_ack_only_if_snooped: assert property (@(posedge clk) disable iff (!rst_n)
  (inv_ack & ~awaiting_ack_q) == '0);
 
// C5 — the outstanding set only shrinks during a transition. If it grows, a
// second transition started before the first finished.
a_awaiting_monotonic: assert property (@(posedge clk) disable iff (!rst_n)
  busy_q |=> ((awaiting_ack_q & ~$past(awaiting_ack_q)) == '0));
 
// C6 — LIVENESS, and it needs a bound. Without a timeout a lost
// acknowledgement is an indefinite wait, which is Debug Lab 2.
a_transition_terminates: assert property (@(posedge clk) disable iff (!rst_n)
  $rose(busy_q) |-> ##[1:MAX_SNOOP_LATENCY] !busy_q);

C6 is the one that distinguishes this from Chapter 1.6's property set. Every assertion in the device-I/O chapter was a safety property — nothing bad happens. A coherence mechanism additionally needs liveness — something good eventually happens — because its characteristic failure is a wait that never ends. Safety properties cannot express that, and a design verified only with safety properties can hang while passing every check.

12. Debug Lab

Each failure was produced by injecting the bug into the real module and re-running the testbench. The output is actual simulator output.

// The snoops have been sent, so the writer can proceed.
end else begin
  state_q[winner_q] <= EXCLUSIVE;      // BUG: promoted without waiting
  pending_q         <= 1'b0;
end

  three sharers      : states=SSSI
  writer requested   : states=SSSI awaiting=0111
  writer PROMOTED    : states=SSSE awaiting=0111  <-- 3 stale sharers still hold the line
  second writer      : states=ESSE
  detector           : two_writers_err=1  <-- two agents in EXCLUSIVE

if ((awaiting_ack_q & ~inv_ack) == '0) begin
  state_q[winner_q] <= EXCLUSIVE;
  pending_q         <= 1'b0;
end

// Collect acknowledgements until the outstanding set is empty.
awaiting_ack_q <= awaiting_ack_q & ~inv_ack;
// ... and nothing bounds how long that takes.

  agent2 never acked  : awaiting=0100 busy=1 grant_excl=0000
  cycles stuck waiting: 30  (no timeout exists, so this is forever)

if (busy_q) snoop_timer_q <= snoop_timer_q + 1'b1;
if (snoop_timer_q == MAX_SNOOP_LATENCY) snoop_timeout_err <= 1'b1;

// Any acknowledgement clears its bit.
awaiting_ack_q <= awaiting_ack_q & ~inv_ack;   // no check that the agent was asked

  ack_without_snoop_err=1

if ((inv_ack & ~awaiting_ack_q) != '0) ack_without_snoop_err <= 1'b1;

// Snoop everyone who held the line when the request was queued.
awaiting_ack_q <= holders_at_request_time & ~excl_pick;

assign grant_shared = (busy_q || any_exclusive) ? '0 : req_shared;

13. Where the Costs Land at Scale

The waveform showed one transition costing an invalidation round trip. Multiply that by real sharing patterns and the pathologies appear.

False sharing deserves its own sentence because it is the one that surprises people: two agents writing two different variables that happen to occupy one cache line will invalidate each other continuously. The program has no shared state and the hardware sees nothing but sharing. It is a layout bug that presents as a coherence cost.

The general shape is worth carrying: coherence costs scale with sharing, not with data volume. A workload that moves enormous amounts of unshared data pays almost nothing; one that shares a few hot lines among many agents can spend most of its time on agreement.

14. Verifying a Coherence Mechanism

Three properties of this problem make it different from verifying a data mover.

The state space is a product. With N agents each in one of three states plus the transition machinery, the reachable space grows quickly, and the interesting states are the concurrent ones. Directed tests reach almost none of them.

Liveness matters as much as safety. Section 11's C6 is not optional. A design that never violates single-writer and occasionally hangs has passed every safety check and is unshippable.

The bugs are races. Debug Lab 4 is the shape: a request granted in the window between sampling a set and acting on it. The stimulus that finds these is concurrent requests to the same line with randomised relative phase, run for a long time.

The corner list worth writing directly: simultaneous shared and exclusive requests to one line; two exclusive requests in the same cycle; an acknowledgement in the same cycle as the snoop; an acknowledgement withheld indefinitely; an acknowledgement from an unsnooped agent; reset with a transition pending; and every agent requesting exclusive access in turn, repeatedly, to exercise ownership migration.

15. How This Appears in Real Engineering

CPU and coherency architect

The questions are about who participates and at what cost. Which agents may cache, and which addresses? Where is the ordering point for a given address, and how many can exist without creating ordering problems between them? Is tracking a directory or a snoop filter, and how does its size scale with memory and agent count? What is the invalidation fan-out for a widely shared line, and is the resulting traffic affordable at the target agent count?

Accelerator architect

Whether the device should cache host memory at all is a genuine decision rather than an obvious yes. Caching buys the locality that moves Chapter 1.3's crossover in favour of sharing; it costs protocol participation, state, and a much larger verification burden. A device whose access pattern is streaming and unshared gains little and pays fully.

RTL and microarchitecture engineer

The structures in Section 9, scaled: state per tracked line, outstanding-transition tables, snoop request and response paths that must stay consistent, and the timers that bound every wait. The discipline that matters most is that every wait needs a bound, because the characteristic failure here is indefinite rather than incorrect.

Verification engineer

Section 14 in full. The distinctive additions relative to every other chapter in this module are liveness properties and long randomised concurrent stimulus. A coherence DUT that passes a directed suite has been barely tested; the state space is in the interleavings.

Performance engineer

Coherence costs scale with sharing, so the counters that matter are per-line rather than aggregate: invalidations issued, snoop round-trip latency, transitions per line, and time spent waiting at the ordering point. A high invalidation rate against a low logical sharing rate is the false-sharing signature, and it is a layout fix rather than a hardware one.

Firmware and system software engineer

Less to do, which is the point — no flush before the device reads, no invalidate before reading back, no ownership bit maintained by discipline. What remains is real: producer-consumer ordering, completion signalling, and mutual exclusion are still the program's job, because coherence is per-address and says nothing about the relationship between addresses.

System architect

The trade in one line: exchange software's explicit maintenance discipline for hardware's agreement machinery. Worth it when agents genuinely share fine-grained state, and not worth it when they exchange large private buffers. Getting this classification right per device is the decision; applying either answer universally is the mistake.

16. Common Misconceptions

17. Interview Reasoning

18. Summary

Coherence is a hardware-maintained agreement about the current value of an address. It guarantees three things — a write eventually becomes visible, all agents see writes to one location in the same order, and a line is either writable by one agent or readable by many — and it guarantees nothing about the relationship between two different addresses, which is what a consistency model is for.

The mechanism reduces to one invariant and three requirements. Single writer: at most one agent may hold a writable copy, and it may not begin writing until every other copy is gone. Enforcing it needs a serialisation point so concurrent requests are ordered, invalidation fan-out so every holder is told, and acknowledgement collection so the writer proceeds only when every holder has confirmed. Section 9 builds exactly that, and the simulated transition reads SSSI → awaiting=0111 → IIIE: three readers, an invalidation round trip, then one writer with every other copy gone.

What it buys is the deletion of an obligation. Forced copies disappear, devices can cache what they re-read, fine-grained sharing becomes practical, and the flush-invalidate-fence discipline that three of Chapter 1.6's four Debug Labs violated stops being possible rather than becoming less likely.

What it costs is real and lands on a different team. State that scales with tracked memory and agent count. Traffic that scales with sharing. Latency on every contended write — visible directly as the gap from t2 to t5 in Figure 2. Protocol complexity, a verification state space that is a product across agents, and a new failure class: the four Debug Labs here produce three stale readers, two simultaneous writers, an inconsistency between fan-out and collection, and a writer blocked forever by one missing acknowledgement.

That last one is the difference worth carrying. Chapter 1.6's bugs were wrong answers; these include hangs. Which is why a coherence design needs liveness properties, not only safety ones — and why every wait in it needs a bound, and every bound needs to escalate rather than proceed.

The durable form: coherence buys agreement and pays for it in state, traffic, latency and complexity. It is a transfer of obligation from software to hardware, not the elimination of one.

19. What Comes Next

Module 1 is complete, and it has been an argument with a shape.

Memory could not supply what compute demanded (1.1). One owner of memory, coherence and translation did not scale to a machine full of peers (1.2). Moving bytes between those peers had a price nobody was counting (1.3). The peers multiplied and specialised (1.4), each acquired a bounded island of local memory (1.5), and the attachment model carrying all of it defines transfers rather than shared state (1.6). This chapter named what would close the gap, and priced it honestly.

The requirement that emerges is specific enough to build to:

Keep what the I/O model does well — discovery, configuration, bulk movement, completion — and add the ability for a device to hold coherent cached copies of host memory, and for device-attached memory to be part of the system's memory rather than a private resource. Do it over infrastructure the ecosystem already has, so devices that need none of it are unaffected.

Chapter 2.1 names the architecture built to exactly that specification, defines it precisely against current consortium material, and introduces the three protocols and three device classes that Module 2 develops. Everything in Module 1 was the question; Module 2 is the answer.

For adjacent material: PCIe vs CXL compares the two contracts from the PCIe side and measures what a missing invalidation costs, AMBA CHI covers coherence as a distributed-state problem on an on-chip interconnect, and cache coherency over CXL sits in the UCIe track. The path is on the CXL tutorials index.