Merging Multiple Reports

Part II · Module 7 · Chapter 7.8 · Merging Multiple Reports

1. Why This Matters

No single EDA report answers a signoff question. "Is this block ready?" needs its timing margin, its coverage, and often its area — written by three tools into three files with three formats, agreeing only on the name of the block.

Combining them is a join, and awk does joins well: read the first report into an array keyed on the shared name, then walk the second and look each key up. Chapter 7.5 built that machinery; this chapter uses it and concentrates on the part that goes wrong.

The part that goes wrong is always the same. A join has to decide what to do about a key that is not in both inputs, and every form of the command makes that decision implicitly. Section 5 is about making it explicitly, because the implicit answer drops rows and says nothing.

2. Two Reports, One Key

alu,0.5
fifo,-0.2
uart,0.1

alu,95
uart,88
ddr,70

Three keys each, and they do not agree. alu and uart are in both. fifo is only in the timing report — and it is the one with negative slack. ddr is only in the coverage report.

That overlap pattern is not contrived; it is what real reports look like. A timing report lists paths the tool analysed, a coverage report lists blocks the testbench exercised, and the two sets are never identical.

3. The Inner Join

Only the keys present in both:

awk -F, 'FNR == NR { slack[$1] = $2; next } $1 in slack { print $1, $2, slack[$1] }' timing.csv cov.csv

alu 95 0.5
uart 88 0.1

Two rows. FNR == NR loads timing.csv into slack[], then each cov.csv record is emitted only if its key was found — Chapter 7.5's idiom with a lookup instead of a membership filter.

This is the honest join when the question genuinely requires both numbers. "Which blocks meet both timing and coverage targets?" cannot be answered for a block with no coverage data, so excluding it is correct.

It is the wrong join when the question is "what is the state of the design?", because two of the four keys are missing from the answer.

4. The Left Join

Keep every record of the driving file, filling in a placeholder where the lookup fails:

awk -F, 'FNR == NR { slack[$1] = $2; next } { print $1, $2, ($1 in slack ? slack[$1] : "NA") }' timing.csv cov.csv

alu 95 0.5
uart 88 0.1
ddr 70 NA

Three rows now. ddr appears with NA for slack, which is right — it has coverage data and no timing data, and the report says so.

The lookup uses ($1 in slack ? slack[$1] : "NA") rather than slack[$1], which matters for the reason Chapter 6.1 section 5 gave: a bare slack[$1] would create an entry for every missing key, so a later count of slack[] would be wrong. It also distinguishes a missing key from a key whose recorded slack is the empty string.

5. What the Left Join Drops

Compare the three keys in that output against the six distinct keys across the two files.

fifo is not there. It is in timing.csv with a slack of -0.2 — a violation — and the merged report does not mention it.

awk -F, 'FNR == NR { s[$1]; next } { c[$1] } END { ni = 0; for (x in c) if (x in s) ni++; nl = 0; for (x in c) nl++; nf = 0; for (x in s) nf++; tot = 0; for (x in s) tot++; for (x in c) if (!(x in s)) tot++; printf "inner=%d  left(cov-driven)=%d  timing keys=%d  full=%d\n", ni, nl, nf, tot }' timing.csv cov.csv

inner=2  left(cov-driven)=3  timing keys=3  full=4

Four distinct keys exist. The left join emitted three. The inner join emitted two.

6. The Full Outer Join

Every key from either file, with a placeholder wherever a value is absent:

awk -F, 'FNR == NR { s[$1] = $2; k[$1]; next } { c[$1] = $2; k[$1] } END { printf "%-6s %8s %6s\n", "key", "slack", "cov"; for (x in k) printf "%-6s %8s %6s\n", x, (x in s ? s[x] : "-"), (x in c ? c[x] : "-") | "sort" }' timing.csv cov.csv

key       slack    cov
alu         0.5     95
ddr           -     70
fifo       -0.2      -
uart        0.1     88

Four rows for four keys. fifo's violation is present. ddr's missing timing is marked. Nothing is dropped and every gap is visible.

The mechanism is a third array. k[] collects keys from both passes — k[$1] with no assignment, the set idiom from Chapter 6.1 section 6 — so the END loop iterates the union rather than either input. s[] and c[] hold the values, and each is tested with in before being read.

The | "sort" is Chapter 7.7's requirement: for (x in k) has no defined order, so a report meant to be diffed between runs must impose one.

This is the join to reach for by default in a signoff context. It is three characters longer than the left join and it cannot silently omit a row.

7. A Duplicate Key in the Loaded File

The second edge. The loading pass assigns s[$1] = $2, and a repeated key overwrites:

alu,0.5
alu,-0.9
uart,0.1

awk -F, 'FNR == NR { slack[$1] = $2; next } { print $1, ($1 in slack ? slack[$1] : "NA") }' dup.csv cov.csv

alu -0.9
uart 0.1
ddr NA

alu reports -0.9. The 0.5 row is gone — last one wins, and nothing says a value was replaced.

Here the surviving value happens to be the worse one, which is lucky. Reverse the file order and the merged report would show 0.5 and hide a violation.

Detecting it costs one test in the loading rule:

awk -F, 'FNR == NR { if ($1 in slack) { printf "DUPLICATE key in %s: %s (had %s, now %s)\n", FILENAME, $1, slack[$1], $2 > "/dev/stderr"; dups++ } slack[$1] = $2; next } END { if (dups) printf "%d duplicate key(s)\n", dups > "/dev/stderr" }' dup.csv cov.csv 2>&1 >/dev/null

DUPLICATE key in dup.csv: alu (had 0.5, now -0.9)
1 duplicate key(s)

The key, the file, and both values — enough to decide what to do.

And if duplicates are legitimate, keep them all rather than choosing:

awk -F, 'FNR == NR { slack[$1] = ($1 in slack ? slack[$1] ";" $2 : $2); n[$1]++; next } { printf "%-6s %-12s (%d timing row(s))\n", $1, ($1 in slack ? slack[$1] : "NA"), n[$1] + 0 }' dup.csv cov.csv

alu    0.5;-0.9     (2 timing row(s))
uart   0.1          (1 timing row(s))
ddr    NA           (0 timing row(s))

Both slacks for alu, with the row count beside each key. ddr shows 0 timing row(s), which states the gap rather than leaving NA to be interpreted.

A timing report legitimately contains many paths per block, so this is usually the right reading — and the aggregate you then want is the worst of them, which is Chapter 6.3's per-key extremum with its per-key seed.

8. Merging Three Reports

With three inputs, infer nothing from counters — name the files, per Chapter 7.5 section 9:

awk -F, 'FILENAME == ARGV[1] { s[$1] = $2; k[$1]; next } FILENAME == ARGV[2] { c[$1] = $2; k[$1]; next } { a[$1] = $2; k[$1] } END { printf "%-6s %8s %6s %6s\n", "key", "slack", "cov", "area"; for (x in k) printf "%-6s %8s %6s %6s\n", x, (x in s ? s[x] : "-"), (x in c ? c[x] : "-"), (x in a ? a[x] : "-") | "sort" }' timing.csv cov.csv area.csv

alu,1820
fifo,640
ddr,2410

key       slack    cov   area
alu         0.5     95   1820
ddr           -     70   2410
fifo       -0.2      -    640
uart        0.1     88      -

Four keys, three sources, every gap marked. Each row now says something a single report could not: fifo violates timing, has area, and was never covered. uart has timing and coverage and no area row.

The pattern scales — one array per source plus one union array — and the - placeholders are what make it readable. A table of this shape is the usual deliverable of a signoff script.

9. Common Mistakes

Using a left join when the question is about the design. It is complete with respect to the driving file only — section 5.

Letting the command-line order decide which keys are dropped. Swap the files and a different set vanishes.

Reading s[$1] without in. It creates a key for every miss — Chapter 6.1 section 5.

Assuming the loaded file has unique keys. A repeat overwrites silently — section 7.

Reporting a single value for a key that had several rows. Keep them, or aggregate deliberately with a seeded extremum.

Leaving the union implicit. Collect keys from every pass into one array — section 6.

Not imposing an order on the output. for (x in k) has none — Chapter 7.7.

Using FNR == NR for three files. It separates the first from all the rest — Chapter 7.5 section 9.

Printing a bare placeholder with no count. NA and 0 rows carry different information.

10. Exercises

Exercise 1 — Predict which keys appear

awk -F, 'FNR == NR { s[$1]; next } $1 in s { print $1 }' timing.csv cov.csv

alu
uart

The inner join: only the two keys in both files.

Exercise 2 — Left join with a placeholder

Emit one row per coverage record, with the slack where it exists.

alu,0.5
fifo,-0.2
uart,0.1

alu,95
uart,88
ddr,70

awk -F, 'FNR == NR { slack[$1] = $2; next } { print $1, $2, ($1 in slack ? slack[$1] : "NA") }' timing.csv cov.csv

alu 95 0.5
uart 88 0.1
ddr 70 NA

Exercise 3 — Debug a missing path

A signoff table built from the two reports does not mention fifo, which has a timing violation. Explain and fix.

awk -F, 'FNR == NR { slack[$1] = $2; next } { print $1, $2, ($1 in slack ? slack[$1] : "NA") }' timing.csv cov.csv

alu 95 0.5
uart 88 0.1
ddr 70 NA

The command emits one row per record of cov.csv, and fifo has no coverage record. A left join can only report keys the driving file contains.

awk -F, 'FNR == NR { s[$1] = $2; k[$1]; next } { c[$1] = $2; k[$1] } END { for (x in k) printf "%-6s %8s %6s\n", x, (x in s ? s[x] : "-"), (x in c ? c[x] : "-") | "sort" }' timing.csv cov.csv

alu         0.5     95
ddr           -     70
fifo       -0.2      -
uart        0.1     88

fifo is present with its violation, and - marks the missing coverage.

Exercise 4 — Count what each join loses

Report the row count of the inner, left and full joins from one command.

awk -F, 'FNR == NR { s[$1]; next } { c[$1] } END { ni = 0; for (x in c) if (x in s) ni++; nl = 0; for (x in c) nl++; tot = 0; for (x in s) tot++; for (x in c) if (!(x in s)) tot++; printf "inner=%d  left=%d  full=%d\n", ni, nl, tot }' timing.csv cov.csv

inner=2  left=3  full=4

Three numbers, three different reports from the same two files. Printing them beside a merged table is the cheapest way to show a reader what the join excluded.

Exercise 5 — Detect a duplicate key

Show the overwrite happening, then detect it.

alu,0.5
alu,-0.9
uart,0.1

awk -F, 'FNR == NR { slack[$1] = $2; next } { print $1, ($1 in slack ? slack[$1] : "NA") }' dup.csv cov.csv

alu -0.9
uart 0.1
ddr NA

awk -F, 'FNR == NR { if ($1 in slack) { printf "DUPLICATE key in %s: %s (had %s, now %s)\n", FILENAME, $1, slack[$1], $2 > "/dev/stderr"; dups++ } slack[$1] = $2; next } END { if (dups) printf "%d duplicate key(s)\n", dups > "/dev/stderr" }' dup.csv cov.csv 2>&1 >/dev/null

DUPLICATE key in dup.csv: alu (had 0.5, now -0.9)
1 duplicate key(s)

alu reported -0.9 and the 0.5 row was discarded. The detector names both values so a reviewer can see which survived.

Exercise 6 — Engineering application

Merge three reports into a signoff table: every key, every column, gaps marked, a count of what each source contributed, and a non-zero status if any key is missing data from any source.

alu,1820
fifo,640
ddr,2410

awk -F, 'FILENAME == ARGV[1] { s[$1] = $2; k[$1]; ns++; next } FILENAME == ARGV[2] { c[$1] = $2; k[$1]; nc++; next } { a[$1] = $2; k[$1]; na++ } END { nk = 0; gaps = 0; printf "%-6s %8s %6s %6s  %s\n", "key", "slack", "cov", "area", "status"; for (x in k) { nk++; miss = (x in s ? 0 : 1) + (x in c ? 0 : 1) + (x in a ? 0 : 1); if (miss) gaps++; printf "%-6s %8s %6s %6s  %s\n", x, (x in s ? s[x] : "-"), (x in c ? c[x] : "-"), (x in a ? a[x] : "-"), (miss ? "INCOMPLETE" : "complete") | "sort" } close("sort"); printf "(%d key(s); timing %d, cov %d, area %d row(s); %d key(s) incomplete)\n", nk, ns, nc, na, gaps; if (gaps) exit 1 }' timing.csv cov.csv area.csv
echo "status: $?"

key       slack    cov   area  status
alu         0.5     95   1820  complete
ddr           -     70   2410  INCOMPLETE
fifo       -0.2      -    640  INCOMPLETE
uart        0.1     88      -  INCOMPLETE
(4 key(s); timing 3, cov 3, area 3 row(s); 3 key(s) incomplete)
status: 1

Four keys from three three-row reports, every gap marked, and a status a pipeline can branch on.

Each row now says something no single report could. fifo violates timing, has area, and was never covered — a block that was synthesised and not exercised. uart has timing and coverage and no area row, which usually means it was not in the synthesis run at all. alu is the only complete row.

Three mechanisms worth naming. The phases are keyed on FILENAME == ARGV[n] rather than on counters, because FNR == NR only separates the first file from the rest — Chapter 7.5 section 9. The close("sort") is what puts the trailer after the sorted rows instead of before them, per Chapter 7.3 section 7. And the miss arithmetic sums three in tests, so a key absent from two sources is still counted once as one incomplete key — which is why 3 key(s) incomplete matches the three INCOMPLETE rows rather than counting four missing cells.

The engineering caveat: INCOMPLETE means this merge had no row for that key in that file, which is not the same as the quantity being absent from the design. A path the timing tool did not analyse and a path it analysed and found fine are both reported as - here if the report simply lacks the row — so the placeholder marks a gap in the data, and distinguishing a missing measurement from a measurement of nothing needs the tool's own unconstrained-path report alongside.

Exercise 7 — Challenge: a merge that cannot silently drop a key

Write a merge that asserts its own completeness — the union key count must equal the number of rows emitted — and show the assertion firing when a join type is wrong.

awk -F, 'FNR == NR { s[$1] = $2; k[$1]; next } { c[$1] = $2; k[$1] } END { nk = 0; for (x in k) nk++; rows = 0; for (x in k) { rows++; printf "%-6s %8s %6s\n", x, (x in s ? s[x] : "-"), (x in c ? c[x] : "-") | "sort" } close("sort"); printf "(%d union key(s), %d row(s) emitted)\n", nk, rows; if (rows != nk) { print "ASSERTION FAILED: rows != keys" > "/dev/stderr"; exit 3 } }' timing.csv cov.csv
echo "status: $?"

alu         0.5     95
ddr           -     70
fifo       -0.2      -
uart        0.1     88
(4 union key(s), 4 row(s) emitted)
status: 0

Four keys, four rows, assertion satisfied.

The same assertion applied to a left join catches the omission:

awk -F, 'FNR == NR { s[$1] = $2; k[$1]; next } { c[$1] = $2; k[$1]; rows++; printf "%-6s %8s %6s\n", $1, ($1 in s ? s[$1] : "-"), $2 | "sort" } END { close("sort"); nk = 0; for (x in k) nk++; printf "(%d union key(s), %d row(s) emitted)\n", nk, rows; if (rows != nk) { printf "ASSERTION FAILED: %d key(s) never emitted\n", nk - rows > "/dev/stderr"; exit 3 } }' timing.csv cov.csv 2>/dev/null
echo "status: $?"

alu         0.5     95
ddr           -     70
uart        0.1     88
(4 union key(s), 3 row(s) emitted)
status: 3

Three rows against four union keys, status 3. The assertion does not say which key was dropped, and it says a key was — which is enough to stop a signoff script from proceeding on an incomplete table.

Naming the missing key as well:

awk -F, 'FNR == NR { s[$1] = $2; k[$1]; next } { c[$1] = $2; k[$1]; seen[$1]; rows++ } END { nk = 0; for (x in k) nk++; for (x in k) if (!(x in seen)) printf "never emitted: %s (slack %s)\n", x, (x in s ? s[x] : "-"); printf "(%d union key(s), %d row(s) emitted)\n", nk, rows }' timing.csv cov.csv

never emitted: fifo (slack -0.2)
(4 union key(s), 3 row(s) emitted)

fifo, with its violation, named explicitly.

This is the pattern every chapter in Modules 5 to 7 has converged on: derive the same quantity two ways and compare them. Here the two derivations are the union of all keys and the count of rows written — independent enough that a dropped row cannot affect both, which is precisely what Chapter 7.5 section 8 said a guard requires.

11. Summary

• A join is an array lookup built on Chapter 7.5's two-file idiom: load the first report keyed on the shared name, then walk the second.
• Every join form decides what to do with a key in only one input, whether or not the author decided.
• An inner join keeps only shared keys — correct when the question needs both values.
• A left join emits one row per driving-file record, so it always looks complete and silently omits every key the driving file lacks.
• Which keys are dropped depends on the command-line order, usually chosen for convenience.
• A full outer join collects keys from both passes into a union array and marks every gap. Three characters longer, and it cannot lose a row.
• Test with in before reading, or the lookup creates a key for every miss.
• A duplicate key in the loaded file silently overwrites. Detect it, or concatenate and report the row count.
• For three or more reports, name the files and keep one array per source plus a union array.
• Impose an output order — the union loop has none.
• Assert that rows emitted equals union keys, and the merge cannot drop a row quietly.

That completes Module 7 and Part II. You can read many files, join them, compute defensible numbers from them, and emit the result in a shape and an order that something else can consume.

Module 8 — Writing Reliable AWK Scripts is the last of Part II and is about the habits that make a program a team can depend on: moving it out of the shell's quotes into a -f script file, passing values in with -v, validating that the input looks the way the program assumes, handling the missing field that becomes a confident zero, forcing the numeric comparison you meant, knowing which features are POSIX and which are gawk, and keeping a test input beside the script so a format change is caught by a diff rather than by a wrong nightly report.