Blog
Coverage before clean reports
I treat a quiet security report as a question about observation, not an assurance of safety.
Ask what was observed
A blank findings list can mean that checks ran and found nothing, or that inputs were missing, skipped, or unsupported. I want a report to name what it attempted, what it actually inspected, and what it could not inspect. Without that distinction, the neatest summary may be the least informative one. The absence of an alert is evidence only inside the boundaries of a completed, relevant check.
I also want the report to say whether a check was applicable. A dependency check on a project without dependencies has nothing to inspect, and that is different from a check that could not read its input. Separating not applicable from not scanned keeps the summary honest without turning every quiet result into an alarm.
Aggregation is where this discipline usually breaks down. A summary that counts only findings will happily compress a partial run into a clean-looking page. I prefer a summary that carries a count per state and a short reason for every skip, so a reader can see how much of the intended ground was actually covered.
{
"checks": [
{ "name": "dependencies", "state": "inspected", "findings": 0 },
{ "name": "permissions", "state": "not-scanned", "reason": "input missing" },
{ "name": "licenses", "state": "not-applicable", "reason": "no bundled assets" }
],
"summary": { "inspected": 1, "notScanned": 1, "notApplicable": 1 }
}Make gaps visible
I look for a separate state for not scanned, rather than folding it into pass. If a check cannot read its input, its scope changes, or it stops early, I want that condition to survive into the human-facing summary. This costs a little clarity in presentation: a report becomes less reassuring at first glance. In return, someone can decide whether to rerun it, supply missing context, or seek another kind of review.
In practice I model a check as a small state machine: attempted, inspected, and then an outcome such as clean, finding, or skipped with a reason. The states are not decoration; they decide what the next reader should do. A skipped check invites supplying missing input or choosing another kind of review, while a clean one invites moving on.
The same distinction should survive into automation. If a pipeline treats incomplete coverage as success, the report's careful wording is wasted. I map the states to conservative exit codes: one code for findings, a different one for an incomplete run, and success only when every eligible check actually inspected its input.
#!/usr/bin/env bash
set -euo pipefail
report="report.json"
gaps=$(jq '[.checks[] | select(.state == "not-scanned")] | length' "$report")
findings=$(jq '[.checks[].findings // 0] | add' "$report")
# An incomplete run must not be reported as success.
if [ "$gaps" -gt 0 ]; then
echo "coverage incomplete: $gaps check(s) did not inspect their input" >&2
exit 2
fi
if [ "$findings" -gt 0 ]; then
echo "$findings finding(s) need review" >&2
exit 1
fi
echo "all checks inspected their input"
Verify the reporting path
I would test the reporter with a known eligible input, an ineligible one, and an intentionally unavailable input. The result should distinguish no finding from no inspection without inventing a vulnerability. That exercise does not prove a scanner detects every problem; it proves that a coverage failure is not mistaken for a clean result. I would still ask which risks lie outside the checks altogether.
A useful exercise is to run the reporter against a small set of fixtures: one eligible input, one that does not apply, and one that cannot be read. Then I check the summary rather than the checker. Does the missing input appear as a gap, or does it vanish? Does the inapplicable check stay neutral? Those assertions are cheap and they protect the most important property of the report.
I keep the claim modest. This proves that the reporting path distinguishes absence of findings from absence of inspection; it does not prove that the checks find everything, or that the list of checks covers the risks that matter. I still ask what lies outside the checks, because coverage begins with choosing what to look at.
type CheckState = "inspected" | "not-scanned" | "not-applicable";
interface CheckResult {
name: string;
state: CheckState;
findings: number;
reason?: string;
}
// The summary keeps gaps visible instead of folding every
// quiet result into "pass".
export function summarize(results: readonly CheckResult[]) {
const inspected = results.filter((r) => r.state === "inspected");
const notScanned = results.filter((r) => r.state === "not-scanned");
return {
findings: inspected.reduce((sum, r) => sum + r.findings, 0),
inspected: inspected.length,
gaps: notScanned.map((r) => r.reason ?? r.name),
};
}