CI Evidence Convention

This guide is the contract for the CI evidence archival mechanism: the naming convention, file schemas, and bundle format that any release-publishing repo — or any future CI gate — conforms to. It is the authoritative reference for the ci-evidence-<gate> artifact, the evidence.json fragment, the evidence-producing gate set, and the manifest.json bundle index.

For the motivation, the determinism analysis, and the full architecture, see epic vergil-project/.github#140. This page documents the wire format only.

Table of Contents

• Overview
• Producer side: the ci-evidence-<gate> artifact
• The evidence.json fragment schema
• The evidence-producing gate set
• Producer prerequisite: real report files
• The bundle tree
• The manifest.json schema (v1.0)
• Verification: how a downstream auditor checks the bundle
• Deployment lifecycle

Overview

CI evidence archival makes each release's gate output durable, complete, and machine-verifiable. At publish time the release harvests every CI gate's full report, bundles it into a compressed, self-describing archive, cryptographically attests the archive to the pipeline and the exact released commit, and attaches it to the GitHub Release — where it lives permanently, independent of Actions retention.

The evidence step is an enforcing gate: a release does not publish without complete, attested evidence for every required gate. Incomplete or failed evidence hard-fails the release before anything is attached (see the deployment lifecycle).

The mechanism has two sides coupled only by a naming convention — no direct calls — which is what keeps the harvester language-agnostic and the whole mechanism fleet-wide:

• Producer side (CI gates in vergil-actions). Each gate reusable workflow uploads its full report(s) as a workflow-run artifact named ci-evidence-<gate>, alongside a small evidence.json fragment.
• Consumer side (publish-time harvest in vergil-tooling). A single language-agnostic command, vrg-ci-evidence, downloads every ci-evidence-* artifact, validates completeness, assembles the bundle tree plus a manifest.json index with per-file sha256, and produces v{version}-ci-evidence.tar.gz.

The only cross-repo coupling is the ci-evidence-* artifact name and the evidence.json shape. Everything else is derived. Adding a future gate requires no harvester change: it emits the convention and is picked up automatically.

Scope boundary

The evidence step lives inside cd-release, so it applies to — and only to — repos that publish releases through cd-release. Repos that do not cut package releases (vergil-project/.github, docs-only repos) never invoke it. "Fleet-wide" here means "every release-publishing repo," not literally every repository.

Producer side: the ci-evidence-<gate> artifact

Each evidence-producing gate uploads exactly one workflow-run artifact:

• Artifact name: ci-evidence-<gate> — for example ci-evidence-security, ci-evidence-test, ci-evidence-audit, ci-evidence-quality.
• Contents: the gate's full report files (SARIF, coverage XML, JUnit XML, audit/license JSON, SBOM, …), plus an evidence.json fragment at the artifact root describing what the gate ran and found.

One uniform evidence source — always artifacts

Every gate — including the security scanners — uploads its report(s) as a ci-evidence-<gate> artifact unconditionally, decoupled from any code-scanning upload. Under GitHub Advanced Security, CodeQL still drops its SARIF as an artifact alongside its code-scanning upload; Trivy and Semgrep do the same regardless of GHAS availability.

The harvester therefore reads artifacts only and never touches the code-scanning API — one code path, no GHAS branching, one fewer external dependency that can fail. Severity metrics come from each gate's evidence.json fragment (computed from its own SARIF at gate time), never from a server API.

The evidence.json fragment schema

The evidence.json fragment sits at the root of each ci-evidence-<gate> artifact. It is the gate's self-description:

{
  "gate": "security",
  "tools": [{ "name": "codeql", "version": "..." }],
  "metrics": { "findings_by_severity": { "critical": 0, "high": 0 } },
  "files": ["python.sarif", "trivy-results.sarif", "semgrep-results.sarif"]
}

Field	Type	Meaning
gate	string	The gate name — matches the <gate> in the artifact name.
tools	array of objects	Each tool that ran, with name and version.
metrics	object	Gate-specific summary (e.g. findings_by_severity, coverage_pct, tests).
files	array of strings	The report file names present in the artifact.

Current limitations (shipped state). The tools and metrics values above show the intended shape, not what every bundle carries today. Verified against the v2.1.195 bundle:

• The security gate's tools array currently ships empty ([]) — the scanners run and emit their SARIF, but the fragment does not yet enumerate them. (The test, audit, and quality gates do populate tools.)
• metrics currently ships empty ({}) for every gate; the findings_by_severity / coverage_pct / tests summaries are the target shape, not yet populated.

These are producer-side gaps, not schema changes: the fields exist and the consumer reads them; they populate as producers are updated.

If the fragment is absent, the harvester still bundles the raw files and records the gate's conclusion from the check-runs API — but for a required gate an absent fragment or an absent artifact counts against completeness (see the deployment lifecycle).

Completeness requires the report payload, not merely the artifact or the evidence.json envelope. A required gate whose artifact carries only an evidence.json and no report file fails completeness — the empty-payload guard (#2812) rejects an envelope with no attested report behind it. An empty report is not evidence, so it does not satisfy the gate.

The evidence-producing gate set

The set of gates that MUST emit evidence is not a hand-maintained list. It is derived from the same source of truth that drives branch protection: lib/github_config.py:desired_ci_gates_ruleset() computes a repo's required status checks from its VergilConfig (language, [ci] versions, GHAS availability). The evidence layer consumes that same computation, so the gates that are enforced to merge and the gates that are required to have evidence are provably the same set, with no drift.

This is the load-bearing invariant: management of the required gates and collection of their auditing evidence come from common configuration code.

Classification by check-name prefix

Required status checks are grouped into evidence gates by their check-name prefix:

Check name / prefix	Evidence gate	Evidence-producing?
security / …, plus GHAS Trivy / Semgrep OSS / CodeQL	security	Yes
test / …	test	Yes
audit / …	audit	Yes
quality / … (lint, typecheck)	quality	Yes
version / …	—	No (non-blocking)
docs	—	No (low-signal)

The guiding principle: any gate that can block the build is evidence worth keeping. Quality (lint/typecheck) sits alongside security, test, and audit as first-class evidence. version/ is a sanity check on version state, not substantive evidence, and docs is low-signal; the absence of either does not fail the release.

Per-repo correctness for free

Because the set is derived, a repo with a different real profile — no GHAS (CodeQL not required), a non-Python stack, or a gate legitimately disabled — demands evidence for exactly the gates it actually gates on. It cannot spuriously fail for a gate it never ran. Adding a future required gate automatically pulls it into the evidence set via the shared config; the harvester never changes.

Producer prerequisite: real report files

The evidence layer only has value if the gates emit machine-readable report files with real data. A bundle whose test / audit / quality entries carry only an evidence.json envelope and no report is an empty report — and there is no point publishing empty reports.

Producers must therefore emit real report files at the workspace-root paths the producer composite globs for:

The test, audit, and quality report files are emitted per Python version (<ver> ∈ 3.12 / 3.13 / 3.14); the security SARIF are not per-version:

Gate	Required report file(s)
test	coverage-<ver>.xml, junit-<ver>.xml
audit	pip-audit-<ver>.json, licenses-<ver>.json
quality	quality-ruff-<ver>.json, quality-mypy-<ver>.xml
security	python.sarif, trivy-results.sarif, semgrep-results.sarif

Producers emit these files today: the check registry runs pytest --cov … --cov-report=xml --junitxml=…, pip-audit / license checks write --output JSON, and ruff / mypy write their machine-readable reports — each per Python version. The empty-payload guard enforces the prerequisite at harvest time: no bundle is attached until its reports carry real data, so a gate that emitted only an evidence.json envelope hard-fails rather than shipping an empty report. (An earlier registry ran the checks pass/fail only — pytest with no --cov-report=xml / --junitxml, pip-audit with no --output — and persisted no report; that gap is closed.)

The bundle tree

The harvester assembles a single compressed archive, v{version}-ci-evidence.tar.gz, designed to be consumed by a machine auditor — self-describing and verifiable:

v{version}-ci-evidence.tar.gz
└─ evidence/
   ├─ manifest.json          # top-level machine index (see below)
   ├─ checks.json            # raw check-runs snapshot (name, conclusion, timing, log URL)
   ├─ gates/
   │   ├─ security/          # python.sarif, trivy-results.sarif, semgrep-results.sarif, evidence.json
   │   ├─ test/              # coverage-<ver>.xml, junit-<ver>.xml (per Python version), evidence.json
   │   ├─ audit/             # pip-audit-<ver>.json, licenses-<ver>.json (per Python version), evidence.json
   │   ├─ quality/           # quality-ruff-<ver>.json, quality-mypy-<ver>.xml (per Python version), evidence.json
   │   └─ sbom/              # sbom.cdx.json
   └─ README.md              # human orientation for the archive

• manifest.json is the curated, per-gate machine index (schema below).
• checks.json is the raw check-run snapshot — distinct from the manifest's curated view — capturing every check's name, conclusion, timing, and log URL.
• gates/sbom/ is populated by copying the SBOM already built during publish (also a standalone Release asset) into the bundle at harvest time, so the archive stays fully self-contained.
• README.md is a fixed human orientation for the archive.

A standalone copy of manifest.json is also attached as a small, separate Release asset, so a tool can read the summary without downloading the full tarball.

The manifest.json schema (v1.0)

manifest.json is the top-level machine index. It records the release identity, the provenance anchor, and a per-gate summary with per-file sha256:

{
  "schema_version": "1.0",
  "repo": "vergil-project/vergil-tooling",
  "release": {
    "version": "2.1.129",
    "tag": "v2.1.129",
    "released_commit": "<main merge SHA>"
  },
  "provenance": {
    "release_pr": 2281,
    "validated_head_sha": "<PR head SHA>",
    "ci_run_urls": ["https://github.com/.../actions/runs/123"]
  },
  "generated_at": "<ISO-8601, injected by CD>",
  "gates": [
    {
      "name": "security",
      "conclusion": "success",
      "tools": [{ "name": "codeql", "version": "..." }],
      "metrics": { "findings_by_severity": { "critical": 0, "high": 0 } },
      "files": [{ "path": "gates/security/python.sarif", "sha256": "..." }]
    },
    {
      "name": "test",
      "conclusion": "success",
      "metrics": { "coverage_pct": 100, "tests": 1423 },
      "files": [{ "path": "gates/test/coverage-3.14.xml", "sha256": "..." }]
    }
  ],
  "missing_gates": []
}

Field	Meaning
schema_version	Manifest schema version. This document specifies "1.0".
repo	owner/name of the release-publishing repo.
release	The released version, git tag, and released_commit (the main merge SHA).
provenance	The evidence anchor: the release_pr, its validated_head_sha, and the ci_run_urls the evidence was harvested from.
generated_at	ISO-8601 timestamp, injected by the CD environment.
gates	Per-gate record: name, conclusion, tools, metrics, and files (each with path and sha256).
missing_gates	Names of required gates that produced no evidence — recorded as data, never silently dropped.

Current limitations (shipped state). As with the evidence.json fragment, the metrics and (for security) tools values shown are the intended shape. In the v2.1.195 bundle the manifest ships metrics: {} for every gate and an empty tools: [] for the security gate. The fields are present and read by auditors; they populate as producers are updated.

Two trust properties are load-bearing:

1. Every file carries a sha256. An auditor can prove the archive is intact and unmodified.
2. missing_gates is explicit. A required gate that produced no evidence is recorded as data, never silently dropped.

Why the provenance anchor is the release PR, not the released commit

develop → main promotion uses a merge commit, and the security/audit/test gates run on pull requests, not on pushes to main. The final main merge-commit SHA therefore has no gate check-runs anchored to it. The gates do run on the release PR, whose validated tree is identical to what lands on main. Evidence is thus anchored to provenance.validated_head_sha (the release PR head) and harvested at publish time while the PR's runs and artifacts are still fresh. released_commit records what shipped; validated_head_sha records what was validated — they name the same tree.

Verification: how a downstream auditor checks the bundle

An auditor who pulls a release months later verifies the bundle in two steps:

1. Verify the attestation (chain of custody). After the bundle is assembled, the pipeline produces a build-provenance attestation over the bundle's digest (actions/attest-build-provenance), binding the archive to this repository's workflow and the released commit. Verify it with:

gh attestation verify v2.1.129-ci-evidence.tar.gz \
  --repo vergil-project/vergil-tooling

A successful verification turns "here is a tarball" into "here is a tarball cryptographically proven to be the genuine output of this pipeline for this commit." This is the chain-of-custody core of the mechanism.

2. Verify every file's integrity. Unpack the archive and confirm each file's sha256 matches its manifest.json entry:

tar -xzf v2.1.129-ci-evidence.tar.gz
cd evidence
# For each gate file listed in manifest.json:
sha256sum gates/security/python.sarif
# compare against manifest.json → gates[].files[].sha256

Every file listed in the manifest carries a sha256, so an automated auditor can confirm the archive is intact, then read gates[].conclusion and gates[].metrics to confirm what ran, with what result, against which commit — and that missing_gates is empty.

Deployment lifecycle

The evidence step is a hard, enforcing gate today. evidence-enforce defaults to true (as of vergil-actions v2.1.27), so evidence-first ordering is in force: the step harvests, validates completeness, bundles, attests, and attaches — and substantive incompleteness or any failure is terminal. Incomplete or failed evidence hard-fails the release before publish; nothing is published without complete, attested evidence for every required gate.

The gate reached enforcing mode through a completed warning → enforcing lifecycle, now behind us:

• Warning mode (historical, initial rollout). The step ran the full path — harvest → bundle → attest → attach — but on any failure it emitted a loud warning and let the release proceed with whatever evidence it had gathered. It was timeout-bounded and never aborted a release. This was always a temporary deployment state of a hard gate — not a permanent soft gate.
• Enforcing mode (current state). After baking ~4 weeks in warning mode and proving reliable, the gate was promoted by a single, global, human-gated flag flip (evidence-enforce → true). That promotion is complete: the gate now enforces on every release.

A permanent soft/report-only gate was never on the table and remains rejected. For the full lifecycle history, the bake-in window, and the human-gated flip, see epic vergil-project/.github#140 (§9.2, §14.1).

This safety property is load-bearing on the all-hard-gates model: bundling everything onto a public Release asset is safe precisely because every gate is a hard gate, so a bundle by definition reflects a passing scan with no unremediated findings to expose. For why every check that matters is a hard, asserting gate — and there are no report-only/warning gates — see the all-hard-gates principle in the vergil-actions documentation.