Testing and evidence

A passing check answers a bounded question. A local provider stub can establish request construction and uncertainty handling; it cannot establish a real fax delivery. Keep those layers explicit when choosing tests and reporting results.

Start with the affected contract

Avoid assertions that merely repeat the implementation. Exercise a failure that would violate the contract: a cross-workspace identifier, a changed review hash, a stale membership, a duplicate callback or a timeout after a provider may have accepted work.

Run the repository checks

From the repository root after local setup:

npm run typecheck
npm run lint
npm test
npm run docs:check
npm run build

npm test runs the Vitest selection and Node security suites. Integration tests use local Workers/D1 fixtures and injected transports. Their production-mode guards exercise production rules against synthetic records; they do not contact the deployed application merely because a fixture selects production mode.

Run the focused suite while developing, then the required repository checks for the final change. If another agent or process is changing the same files, agree on a stable source snapshot before treating a combined run as final evidence.

Keep the two browser environments distinct

npx playwright install chromium webkit
npm run test:e2e
npm run build:preview
npm run test:preview

Application E2E starts the real local service, migrations and seeded fixtures. Preview tests exercise an independent public design model with no business bindings. Both are useful, but preview success cannot establish tenant isolation, approval acceptance or provider behavior.

Use the configured desktop Chromium, mobile Chromium and iPhone WebKit projects for the relevant change. A Chromium fallback is not evidence for Safari. Preserve business assertions when adapting a stale label or selector, and retain failure traces privately rather than publishing customer-like fixtures by accident.

Full-page layout screenshots use fullPage: true and scale: "css" to preserve the complete document at one image pixel per CSS pixel, avoiding the browser’s 32,767-pixel bitmap limit under Retina emulation. Device viewport, pixel ratio, touch behavior and all layout assertions remain unchanged.

The local secure-distribution suite bounds recovery of an explicit SQLite lock to its read-only charge-inspection query. It keeps a fixed attempt limit and total deadline; other failures remain failures. Approval actions, billing writes and business assertions are never retried by that helper. The suite’s repository README owns the exact fixture limits. This does not permit retrying an uncertain provider outcome.

Optional local network isolation

On supported macOS hosts, npm run test:offline runs a fixed installed selection under an OS network boundary. The runner first proves loopback access and an actual denied outbound connection. It refuses unsupported hosts instead of quietly falling back to online execution.

This selection never installs dependencies or downloads engines. The optional --with-pdf case requires an already installed browser. The report describes synthetic local evidence and captures the tested source before and after the run; a changing source set cannot retain a passing result. This is a boundary for trusted local test code, not a general-purpose hostile-code sandbox.

Qualify CI, engines and hosted behavior separately

CI must report the required jobs for the same source revision. Missing, skipped, cancelled or failed required jobs do not become a passing release through an aggregate label. Reusing a build is meaningful only when its source matches the revision under review.

Qualify the actual scanner signatures, renderer, validator and remote browser runtime separately from stubbed service responses. An installed container image is not evidence of fresh antivirus signatures. Native simulator UI checks do not establish physical-device behavior or App Store distribution.

Hosted health, OAuth discovery, authenticated host calls, provider configuration and real delivery each need their own observation. Read-only checks cannot prove an unperformed send. A build, a package archive or an integration catalog entry does not publish or activate a capability.

Public monitoring separates transport from stored evidence

The scheduled public monitor requests Accept-Encoding: identity and Cache-Control: no-transform only for fixed canonical public paths. Headers are requests, not representation proof: the collector still checks received decoded byte count and SHA-256 against the release manifest. A failed observation is not replaced by automatic retries.

Representation preservation also belongs at the origin. Successful canonical public HTML and its conditional 304 responses append no-transform to the existing response cache policy, retaining bytes, ETag and any existing encoding. The application rule leaves private/API responses and non-HTML assets under their existing policies. A monitor request header alone cannot enforce this origin response contract; stripping injected scripts before comparison would weaken the proof. See Cloudflare’s response-header requirement.

Bounded diagnostics expose only public-path size/hash/encoding or an allowed authority, OIDC or ingestion stage, HTTP status and response code. They remain outside the signed report and reveal no token, OIDC URL or upstream body. An artifact saved before publication does not prove ingestion succeeded. Runner revision, observed product revision, actual report storage and authenticated business qualification remain separate evidence.

Report ingestion also checks the signed direct-job workflow claims. The legacy form omits both job_workflow_ref and job_workflow_sha; the current form must match the exact canonical main workflow and immutable runner commit as a pair. Foreign workflow/revision, partial, empty and null claims remain denied, alongside all existing signature, repository, branch, run/attempt, freshness, body-bound audience and replay checks. This is not permission for another workflow or role.

Only after an actual 401 REPORT_REJECTED can the driver diagnose that same token/body. Decoded boolean comparisons are explicitly unverified; one invocation of the server verifier separately checks the signature. GitHub key reads retain a five-second timeout and the entire diagnostic has a six-second deadline. It requests no new token, repeats no POST and cannot turn the original rejection into success, even if its local verification accepts. Claims, keys and raw errors stay private.

Origin preservation requires application publication; claim verification requires status-service publication. Both need fresh hosted evidence. An isolated signed diagnostic or local fixture does not qualify production report storage or an authenticated customer journey.

Report a result that another engineer can assess

Record the source revision or explicit tested snapshot, environment, suite scope, result and material limitations. Keep real, sandbox and deterministic simulation evidence separate. A failed or unavailable check remains visible; do not replace it with an older passing report.

Use private, untracked artifact destinations for diagnostic logs, screenshots and traces. Review sanitized summaries before any public publication. Never collect actual documents, recipient addresses, credentials or signed URLs merely to prove that instrumentation works.