Attest: Cloud Release and AI Authoring Plan
| Field | Agreed value |
|---|---|
| Status | Agreed baseline — locked for implementation planning |
| Agreed | 2026-10-07 |
| Validated refinements approved | 2026-10-07 |
| Product owner and implementation repository | Vyuh / CDX |
| First customer | Caliber |
| Initial products | eLog and SSO |
| Next product | DocsIQ |
“Locked” records the agreed scope and architecture. It does not mean the implementation is complete, production deployment is authorized, or release readiness has been verified. Changes to this baseline should be recorded explicitly in this document. This is a delivery plan, not a standing agent instruction file or a replacement for the canonical product world models.
1. Product and release scope
Attest is a product owned and implemented in CDX. This is its single active cloud-release and AI-authoring plan, using Caliber as the first delivery. It replaces the older CDX product plans and the copy previously kept in flutter_elog. The Attest world model remains the normative contract reference; the plan does not replace it.
| Repository | Ownership |
|---|---|
| CDX | Attest application, CLI, shared contracts, authoring/AI, runners, orchestration, Cloudflare deployment, storage, evidence, reports and platform documentation |
| flutter_elog | Caliber adapter and launchers, product registries, manifests, datasets/fixtures, seed policy, eLog/SSO/DocsIQ test assets, branding, integration tests and consumer qualification records |
Caliber names in examples and acceptance packs do not become platform defaults. Generic platform behavior must remain testable using CDX's independent Field Notes sample. Consumer repositories do not own forks of the Attest UI, engines or platform roadmap.
Preserve working local CLI/TUI and authoring behavior during the cloud work. Initialization remains non-destructive and does not silently provision clouds, reset application data or install automation. Registry-only distribution must be verified separately before claiming it works; the initial Caliber hosted pilot does not depend on public self-service onboarding. Developer commands and package usage are maintained in the CLI guide and editor guide.
Deliver one hosted Attest workspace for Caliber, connected to a dedicated QA environment. Caliber testers use a browser to select a product and test plan, execute supported suites, investigate failures, and download reports and evidence. Vyuh manages infrastructure, credentials, product integrations, and versioned test packs.
Build the deployment locally first. Use a dedicated self-hosted Supabase container stack for Attest, separate product-fixture data, and independent API, worker and frontend containers. A managed Supabase project and Cloudflare credentials are not prerequisites for this stage. Prove sign-in, a real queued API run, private evidence/report retrieval and persistence after container recreation before provisioning a hosted environment. Keep the proof's coverage explicit; a health-check smoke run does not establish Patrol or product qualification. See the Caliber local proof guide.
The first release tests already deployed candidate application builds. It does not provision a fresh application stack for every run. Start with one active run against the shared QA environment; queue additional requests to avoid conflicting changes to shared test data.
Ship reviewed AI drafting with hosted Attest. Add the bounded trial-and-repair agent after cloud execution is reliable. Adding DocsIQ must reuse the same platform and execution contracts.
The primary experience is scoped verification of a candidate release: what was selected, what was actually verified, what failed or remains unknown, and which evidence supports the result. A passing smoke run does not establish whole-release readiness. Deterministic qualification gates decide the outcome for the declared scope; Caliber's release authorization remains a separate human process unless explicitly integrated later.
Use existing build/run contracts for a verification snapshot containing the candidate, required coverage, qualification policy and exact supporting run references. Do not introduce a separate release-management subsystem for V1. Validate this workflow with Caliber testers and quality reviewers before treating regulated release confidence as final product positioning. AI generation and traceability alone are not unique capabilities.
2. Agreed architecture
Vyuh release pipeline
├── Deploy versioned candidate build
│ └── Caliber QA environment
│ SSO + eLog + supporting services + test database
│ ↑
│ API / UI interactions
│ │
└── Publish matching test pack
│
Caliber browser │
↓ │
Cloudflare Worker │
├── Flutter static assets │
└── Attest Dart API container
↓ │
Supabase control plane │
Durable jobs and runs │
↑ │
Attest runner container┘
↓
Private durable evidence storage| Module | Initial deployment | Responsibility |
|---|---|---|
| Dashboard | Cloudflare Worker static assets | Product selection, authoring, execution controls, results |
| Cloud API | Dart in a Cloudflare Container, routed through Worker | Authentication, authorization, run requests, artifact access |
| Execution worker | Separate Cloudflare Container | Execute approved suites and publish evidence |
| Container lifecycle | Worker and Durable Object | Start runners, monitor activity, stop idle instances |
| Durable control plane | Dedicated Supabase instance; self-hosted locally first | Projects, jobs, attempts, run records and publication metadata |
| Artifact archive | Existing private Supabase storage integration | Recordings, screenshots, logs, manifests and reports |
| Application under test | Existing Caliber deployment infrastructure | Candidate application and supporting services |
| AI authoring | Server-side Attest authoring module | Generate and validate drafts; later coordinate bounded trial runs |
Keep the existing Vyuh durable execution system as the job authority. Cloudflare manages compute lifecycle; do not introduce a second job orchestration system without a demonstrated need.
Use separate API and runner roles. Package pinned tools, dependencies and test assets into runner images. Build the Flutter dashboard in CI. The final choice between a small managed runner pool and fresh containers per job remains an implementation decision; the initial environment concurrency limit is one.
Container filesystems are temporary workspaces. Persist run records, configuration, authored drafts, approved test versions and evidence outside them. Attest's control-plane database must be separate from the application's test database.
Cloudflare reference: Container architecture.
Execution dimensions and optional browser evaluation
Preserve the existing separation of concepts:
| Dimension | Examples |
|---|---|
| Test surface | API, action, UI, performance |
| Runner | Dart test, Patrol, k6; a browser adapter only after explicit acceptance |
| Execution location | Local process, durable worker, CI worker |
| Browser backend | Container Chromium; potentially Cloudflare Browser Run |
Patrol remains the current implementation baseline. The engine rejects new direct Playwright runs; archived provider enum values do not enable support. A bounded comparison of deployed-browser execution with the instrumented Patrol path is permitted. Reintroducing Playwright requires an explicit architecture decision, corresponding contract/documentation changes and evidence qualification. This plan does not select or require a replacement.
Compare the same representative Caliber journeys on correctness, Flutter semantics/selectors, deployed-artifact fidelity, actor isolation, evidence, recovery and operational effort. Keep one proven UI path for V1. Do not build multiple providers merely to demonstrate extensibility.
Cloudflare Browser Run is an optional backend experiment, not a V1 dependency. Acceptance requires:
- Correct rendering and interaction with the actual candidate frontend, including Flutter semantics and required downloads/dialogs.
- Scoped authentication, separate actor contexts, session expiry handling and access to all SSO, API and asset origins without bypassing application guards.
- Checkpoint screenshots and any required pixel recording, timestamps and actor provenance mapped to canonical Attest artifacts.
- Recovery after session interruption or platform rollout, with partial evidence retained and no false completion.
- Measured duration/cost, bounded session usage, explicit cleanup and account limits sufficient for the serialized QA workload.
- Evidence export into Attest's private archive before provider retention ends.
As checked on 2026-10-07, Browser Run's built-in rrweb replay excludes canvas and WebGL content, so it cannot replace Flutter recording evidence. Its recordings expire after 30 days. Outbound Worker routing has HTTP-only and invocation-lifetime constraints; it is not proof that Caliber's private HTTPS or SSO flows work. Recheck provider capabilities when implementing. See recording limitations, outbound routing, session limits, pricing and Flutter web accessibility.
3. Candidate builds and test execution
- Deploy a versioned SSO/eLog candidate into the dedicated QA environment.
- Prepare known accounts, permissions and test fixtures.
- Publish the compatible versioned test pack.
- Verify readiness before enabling execution.
- Run API tests against the candidate's deployed endpoints.
- Prove the existing Patrol path in a cloud runner before exposing UI suites, or explicitly accept and qualify an alternative through the decision above.
For each UI suite, establish whether it drives the deployed frontend or uses an instrumented Flutter application connected to the QA APIs. If instrumentation is required, build from the same commit and configuration lineage as the candidate and record that distinction in evidence. Do not claim verification of the exact deployed frontend artifact when a different artifact was tested.
Each qualification run snapshots application/service build identities, test-pack and test-implementation versions, intent/requirement versions, product-configuration and grounding-bundle versions, qualification-policy version, dataset/seed version, resolved environment, execution provider, browser version, runner image/toolchain identity, attempts, results and evidence. Use immutable digests where available, not mutable labels alone. For generated fixtures, retain the generation inputs and resulting identity; record an explicit not-applicable data contract when no dataset is needed.
Capture the identities of the deployed eLog/SSO and supporting services involved in the verification. A local checkout commit or package version does not prove which deployment answered requests. Check deployment identity before and after execution and block a coherent qualification claim if it changes during the run or cannot be established.
Retries preserve prior attempts. A new rerun is linked to the original run rather than overwriting it. Record the exact run set and selection policy used for the displayed result; a passing rerun must not hide an earlier failure or combine incompatible builds, datasets or environments.
Each supported suite declares prerequisites, data mutations, cleanup behavior, concurrency constraints and retry safety. Infrastructure retries must not blindly repeat application mutations. Publication retries must not rerun tests.
AI trial runs use the same QA-environment lock as normal runs. Coordinate deployments, resets and fixture preparation with that lock as well. A single active runner alone does not prevent an external deployment from changing its target mid-run.
4. Caliber end-user journey
The normal navigation is Product → Candidate release → Verification → Result → Evidence. Internal plan, suite, dataset, run and attempt identities remain unchanged and accessible through detail views.
- Sign into Attest and select eLog or SSO; DocsIQ appears after onboarding.
- Confirm the candidate build, selected QA environment and readiness.
- Choose Smoke or a supported Qualification scope; see included checks and known coverage gaps before starting.
- Start verification. Show business-level progress first, with logs and execution details available through drill-down.
- Inspect the deterministic outcome, blockers, evidence and suggested diagnosis.
- Review or rerun eligible checks and download the report/evidence package.
Infrastructure configuration belongs in admin/advanced interfaces, but the selected environment, candidate and scope must remain visible. A smoke action must be labelled as smoke verification, not unrestricted release approval.
Results and qualification decisions
Preserve the existing qualified, qualified-with-exceptions, not-qualified and blocked outcomes. Show the selected scope, passed/failed/blocked/skipped and unverified coverage, approved exceptions, evidence completeness and timestamped environment/runner health. Unknown health must not appear healthy.
Use wording such as “Qualification passed for the selected scope.” A summary must not infer whole-release readiness from passing counts. Missing mandatory coverage or evidence prevents a complete qualification claim. Keep Caliber's human release disposition separate from the machine-computed result.
Offer evidence inspection, an eligible rerun and report download. Results and reports must reference the same immutable verification snapshot.
Failure classification
Record deterministic failure stage/code separately from a diagnosis. Use typed categories for product defect, test authoring/maintenance, environment, runner/infrastructure, test data/precondition, evidence publication and unknown. Preserve primary and contributing causes when more than one applies.
An AI diagnosis carries supporting evidence references, rationale and an explicit proposed/confirmed status; it does not change test outcomes. Human confirmation records who confirmed it. Do not relabel an assertion failure as a confirmed product defect without sufficient evidence, or call an unavailable environment a passed verification.
Closing the browser must not stop execution. Reopening Attest must recover the durable run state. Vyuh configures infrastructure credentials and endpoints; testers do not supply service-role keys or manually start services.
Start with manual run submission. Automatic smoke runs after deployment are a follow-on once manual execution is reliable.
5. Feature inventory and inspection baseline
This inventory records the source inspection performed on 2026-10-07. “Implemented” means implementation was found, not that the feature passed cloud acceptance testing. Documentation describes both existing and target capabilities; target capabilities are not release claims.
| Area | Features | Inspection assessment |
|---|---|---|
| Product catalog | Products, environments, plans, suites, tags, availability reasons | Implemented; Caliber registers eLog and SSO |
| Run configuration | Scope, browser selection, datasets, readiness checks | Implemented; remote environments remain disabled |
| Environment management | Local startup, preparation, seed/reset policies, remote endpoint configuration | Local implementation exists; cloud setup unverified |
| API testing | Code suites, declarative cases, API sequence authoring | Implemented; hosted authoring integration needed |
| UI testing | Patrol journeys, declarative authoring, assertions, checkpoints | Local implementation; remote Patrol explicitly unavailable |
| Integration testing | Product action suites and workflow checks | Framework exists; readiness is suite-specific |
| Performance testing | k6, options editing, metrics, thresholds, scorecards | Implemented; cloud execution and capacity unverified |
| Execution controls | Start, progress, logs, cancellation, failed-suite reruns | Local capabilities exist; hosted parity incomplete |
| Durable execution | Queues, attempts, leases, recovery, separate publication work | Implemented with focused tests; deployed recovery unverified |
| Live observation | Flutter frames and checkpoint evidence | Local implementation; cloud delivery unverified |
| Recording and replay | WebM, screenshots, checkpoint-linked playback | Implemented; not full browser DOM/network diagnostics |
| History and analytics | Ledger, search, product/build/session projections, cross-build analytics | Local implementation; full hosted parity unverified |
| Evidence integrity | Manifest validation, checksums, inventory, publication receipts | Implemented; deployed durable storage must be exercised |
| Qualification gates | Required checkpoints, actors, evidence and outcome evaluation | Implemented; coverage depends on executable suites |
| Requirement coverage | Requirement IDs linked to plan/checkpoint results, execution and evidence IDs; mixed-context rejection | Implemented foundation; approved requirement/intent version lineage incomplete |
| Reports | PDF, Caliber OQ packages, DOCX, report configuration | Local implementation; regeneration solely from durable data needs verification |
| Hosted access | Token verification, project authorization, membership routes, private artifacts | Foundations exist; Caliber sign-in and isolation need end-to-end checks |
| Retention and recovery | Policy contracts, private storage, immutable-record protections | Full retention, legal-hold operations and restore verification incomplete |
| Developer tools | CLI, initialization, diagnostics, validation, registry/workspace tooling | Implemented |
| AI drafting | API/UI YAML generation, assumptions, validation, review and save | Local implementation; not an autonomous authoring agent |
| Cloudflare deployment | Loopback-only Worker proxy rehearsal | Production containers and lifecycle integration needed |
| DocsIQ | Shared platform reuse | Registration, adapter generalization, fixtures and suites needed |
Some Caliber suites are explicitly planned/unavailable, including standalone navigation, area/equipment journeys and certain workflow journeys. Do not advertise those entries as implemented product coverage.
The current hosted application is not feature-equivalent to the local dashboard. Hosted authoring routes and other missing controls must be mapped explicitly before exposing their UI.
The current manifest traceability gate checks checkpoint assertion/evidence linkage. It does not by itself establish complete traceability to approved, versioned requirements. Extend the existing coverage contracts for this need. This inventory is not a requirement for full hosted feature parity in V1.
6. AI authoring: existing foundation
The existing AI authoring flow is:
Describe test → Generate YAML and assumptions → Validate → Review → SaveIt has OpenAI and Gemini adapters and supports API-case and UI-journey drafts. It makes a single model request. It does not currently inspect the application, trial-run its draft, or iterate on execution failures.
The current workspace requests a provider key, and saving uses files in the product workspace. For the hosted Caliber offering, configure provider credentials server-side and persist drafts and approved versions durably. Provider/model availability and live request compatibility must be verified; mocked adapter tests do not prove them.
7. AI authoring: agreed target
The AI assistant is an authoring agent inside Attest. It uses Attest's existing validation, execution and evidence contracts rather than a separate runner.
Normal authors work with business-readable intent, prerequisites, steps and expectations. A structured editor serializes these into existing declarative contracts and uses the same validation path. YAML remains an advanced view; do not introduce a second execution language or an unconstrained natural-language compiler. Persist approved structured definitions, not only conversation text.
Stable intent and traceability
Extend the existing requirement inventory and plan/checkpoint contracts with a minimal versioned intent record: stable identity, source requirement/version, business invariants, prerequisites, coverage/evidence expectations and review provenance. Version the execution implementation separately and bind both into the published test pack and each run. Shared contracts belong in CDX; product requirements belong to the consuming product.
The retained chain is:
Requirement/version → Intent/version → Test implementation/version
→ Candidate and verification scope → Run/attempt/checkpoint
→ Evidence → Deterministic qualification resultOne invariant may require several checks and evidence types. A successful UI click does not prove persistence, audit or authorization behavior. Missing links and unimplemented required checks remain explicit gaps. Reports use this same persisted chain, not AI-generated claims of coverage.
AI may propose implementation repairs, but even selector-only changes create visible revisions. Changes to business invariants or expectations require human review; they must never be disguised as execution repair. Qualification tests remain pinned during a run. Reuse existing contracts rather than creating a parallel intent database or test-management system.
Product knowledge and ownership
Each product supplies a curated, versioned grounding bundle referencing its canonical domain rules, requirements, workflows, roles/permissions, APIs, selectors, fixtures, constraints, data needs and existing tests. Record source IDs, versions/digests and unresolved conflicts. Prefer referenced or generated projections of authoritative sources over manually duplicated rules.
CDX owns the bundle contract, validation and provenance. Caliber integrations own eLog, SSO and DocsIQ content. Extend Attest's world model only for shared authoring/traceability concepts; do not move product rules into it. For example, eLog's X-1 access layers and I-3 product-owned roles remain authoritative; SSO login alone is not permission to perform an eLog action.
Observed application behavior is discovery evidence, not a business authority. If required context is missing or conflicts, return needs clarification with the specific unanswered question. A pause/resume example cannot establish who may resume QA Hold work without the applicable approved product rule.
Authoring loop and entry points
V1 supports requirement/intent-driven drafting, validation and reviewed publication. The full engineering loop below is follow-on after cloud execution is reliable; it is not a V1 autonomous-agent requirement.
Requirement + product contracts + existing examples
↓
Draft steps and assertions
↓
Attest schema and policy validation
↓
Trial run in QA
↓
Inspect evidence and classify failure
↓
Propose repair, request context, or identify likely defect
↓
Trial revised draft again only within allowed bounds
↓
Human review → Versioned test packRequired capabilities:
- Read approved product requirements, API contracts, UI selectors, fixtures and existing tests.
- Produce constrained API-case or UI-journey definitions with explicit prerequisites, expected outcomes and assumptions.
- Validate through the same parsers and policies as manually authored tests.
- Request trial runs in the approved QA environment and retrieve their actual results and evidence.
- Propose bounded repairs while distinguishing authoring errors, missing data, environment failures and application defects.
- Retain authoring sessions, draft versions, trial-run links and review history.
- Promote reviewed definitions into immutable, versioned test-pack releases.
The agent may request tools; Attest owns permissions, execution and recorded outcomes. Trial runs remain ordinary Attest runs, clearly labelled as authoring trials. Bound each session by iteration, execution-time and model-cost budgets.
Keep three entry points possible without implementing all three initially:
| Entry point | Delivery scope | Constraint |
|---|---|---|
| Requirement → Tests | V1 | Grounded proposal, validation and human review |
| Change → Tests | Follow-on | Recommend impacted coverage with explicit uncertainty; never silently omit mandatory qualification checks |
| Explore → Tests | Long-term experiment | Approved QA scope and bounded actions; observations do not define correct behavior |
Assertions and qualification gates determine pass/fail. The model must not invent observations or weaken an expected outcome merely to make a failing application pass. A generated draft is not approved qualification coverage. Preserve requirement provenance and the reason for each proposed change.
Begin with existing declarative formats. Arbitrary generated Dart, shell execution and unrestricted autonomous publication are outside the initial scope. Treat retrieved application content as data, not as instructions granting tools or credentials.
User interaction
- Select Create test with AI, a product and an approved environment.
- Describe the intended behavior.
- Review proposed prerequisites, steps, assertions and missing information.
- Validate the draft.
- Trial-run it when that capability is enabled.
- Inspect evidence and proposed revisions.
- Have an authorized reviewer publish the approved test version.
Evaluating the authoring agent
Maintain known authoring tasks and measure whether the agent:
- Produces valid and executable definitions.
- Preserves the intended business assertion.
- Detects deliberately introduced defects rather than only passing healthy cases.
- Flags missing information instead of inventing contracts or selectors.
- Respects environment restrictions and credential boundaries.
- Stops within configured iteration, time and cost limits.
This is the second meaning of “a harness for itself”: Attest executes generated tests, and a separate evaluation set verifies the quality of the authoring loop.
8. Adding DocsIQ
DocsIQ reuses the same hosted Attest deployment, authoring engine, execution contracts, storage and reporting infrastructure.
| Integration asset | DocsIQ work |
|---|---|
| Candidate application | Deploy DocsIQ and its required supporting services |
| Product registration | Add DocsIQ to the Caliber distribution |
| Environment | Register endpoints, readiness checks and credential references |
| Test pack | Add executable suites, checkpoints and evidence requirements |
| Dataset | Add documents, policies and author/reviewer/approver accounts |
| AI context | Supply DocsIQ rules, API contracts, selectors and examples |
| Runner image | Package the new tests and required dependencies |
First remove SSO-versus-eLog assumptions in the Caliber adapter. In particular, other target IDs must not fall back to eLog endpoints, application version or suite selection. Use explicit product configuration.
Initial DocsIQ smoke coverage can cover sign-in, repository access and draft creation. Qualification then adds policy-driven submission and workflow evidence. DocsIQ rules DP-5 and DW-4 require policy readiness and the configured workflow path; checking a status field alone does not establish those behaviors.
Registration can be quick, but complete qualification depends on authored and verified suites and fixtures. Adding a product entry does not create coverage.
9. Delivery sequence and exit conditions
| Step | Deliverable | Exit condition |
|---|---|---|
| 0. Local containers | Self-hosted Supabase, separate product-fixture database, API, worker and frontend images | Local sign-in, actual queued API execution, private PDF/evidence retrieval, access isolation and completed-run persistence survive container recreation |
| 1. QA target | Versioned SSO/eLog deployment, isolated test data and accounts | Readiness and representative API tests pass remotely |
| 2. Hosted foundation | Dashboard, authenticated API, dedicated control plane | Authorized Caliber user queues and retrieves a durable run |
| 3. Container execution | Pinned runner image and Cloudflare lifecycle integration | Runner claims a job, executes and publishes retained evidence |
| 4. Cloud vertical slice | One eLog API test, one eLog UI journey and one SSO UI journey; Patrol baseline or explicitly accepted alternative | All three execute against approved QA from cloud workers with lineage and evidence; failure/restart probes remain honest |
| 5. Hosted authoring | Structured manual/AI editors, curated context, authenticated routes, durable intent/definition versions | Reviewed draft survives restart and becomes a runnable versioned test without changing its approved intent |
| 6. Pilot acceptance | Scoped result, typed failure classification, recovery, access checks and reports | Caliber tester completes the end-to-end workflow unaided and understands verified versus missing coverage |
| 7. AI trial-and-repair | Product grounding, bounded trial runs, repair proposals and review | Evaluation set demonstrates meaningful assertions and controlled behavior |
| 8. DocsIQ integration | Product configuration, fixtures and initial coverage pack | DocsIQ executes and reports through the same platform |
DocsIQ preparation may proceed alongside earlier steps, but execution claims depend on the relevant API/browser path being proven. No completion date is committed by this plan.
The three-test vertical slice is the first end-to-end proof, before broad authoring or hosted parity work. Select and record exact registered test IDs, candidate deployment and expected evidence before execution. Reuse a healthy QA deployment; do not bootstrap the local application stack inside the runner. Retain build/test-pack identities, environment, dataset contract, runner/provider provenance, results, logs/evidence and attempt history for all three tests. Include a deliberate assertion failure and runner/publication interruption to prove that a failed or incomplete verification cannot appear qualified.
An optional browser comparison can inform step 4, but it is not a second V1 implementation commitment. If the alternative does not meet evidence and fidelity criteria, retain the Patrol path with its instrumented-build scope clearly stated. Do not claim exact deployed-frontend coverage from that fallback.
| Horizon | Committed scope |
|---|---|
| V1 | Caliber eLog/SSO, one QA environment, serialized execution, API plus one proven UI path, durable runs/evidence, scoped qualification gates, failure categories, basic reports/evidence inspection, reviewed AI drafting and minimal intent/context lineage |
| Follow-on | Bounded AI trial/repair, DocsIQ pack, change-impact recommendations, automatic deployment smoke triggers and additional proven suites |
| Experimental/long-term | Exploration-based authoring, alternative browser backends, Kitesurf, optional low-risk healing, private runners and isolated per-run environments |
Useful existing checkpoint playback may remain; a rich replay redesign, advanced analytics, complex scheduling, a broad performance service and full hosted feature parity are not V1 release gates. Required evidence integrity, access control and recovery are not reduced with UI scope.
10. Release acceptance gates
Before calling the agreed hosted release ready:
- [ ] Caliber sign-in, project access and artifact authorization work end to end.
- [ ] Unauthorized identities cannot access another project's jobs or artifacts.
- [ ] The candidate build and test-pack identities are recorded accurately.
- [ ] Deployed service identities are verified; build drift during a run blocks a coherent qualification claim.
- [ ] Intent, implementation, configuration, context, policy, dataset and runner/provider identities are retained with the verification snapshot.
- [ ] API execution and the selected SSO/eLog browser journeys pass in Cloudflare.
- [ ] The three-test vertical slice retains complete provenance and evidence; induced failures and interruptions cannot produce a qualified outcome.
- [ ] Evidence names the actual execution surface and artifact tested; required pixel capture and checkpoint correlation work for the selected provider.
- [ ] Only verified suites are enabled; unsupported capabilities explain why.
- [ ] Mandatory requirement/checkpoint/evidence gaps block qualification; smoke success is not presented as whole-release approval.
- [ ] Qualified-with-exceptions and blocked outcomes remain distinct, with explicit waivers and no inferred human release authorization.
- [ ] Shared-environment locking coordinates runs, AI trials, deployments, resets and fixture preparation.
- [ ] Runs continue independently of the browser session.
- [ ] Cancellation and runner interruption produce accurate durable outcomes.
- [ ] Recovery does not silently duplicate application mutations or overwrite attempts.
- [ ] Reruns preserve prior failures and identify the exact compatible run set supporting the current result.
- [ ] Publication failure remains visible and retryable without rerunning tests.
- [ ] Reports and evidence remain retrievable after API and runner restart.
- [ ] Database and object-storage restore checks recover a complete sample run.
- [ ] The initial retention policy is explicit and does not silently expire qualification evidence; unsupported legal-hold administration is not advertised.
- [ ] Hosted drafts and approved definitions survive container replacement.
- [ ] AI provider calls work with the configured live model and server-side credentials.
- [ ] Draft validation, review and versioned promotion work end to end.
- [ ] Business-readable authoring uses the existing declarative validators; changing an invariant requires review and a visible new version.
- [ ] Missing/conflicting product knowledge returns needs-clarification rather than invented requirements, selectors or assertions.
- [ ] Monitoring distinguishes API, worker, target-environment and storage failures.
- [ ] Failure categories include unknown and preserve supporting evidence; AI diagnoses are attributed proposals and never override gate outcomes.
- [ ] Concurrency, execution duration, artifact size and AI usage are bounded.
- [ ] Relevant analysis is clean: no errors, warnings or infos.
- [ ] Required tests and deployed acceptance checks pass, with evidence retained.
The trial-and-repair release additionally requires the authoring-agent evaluation set to pass. Full feature parity, performance capacity and additional product suites must be qualified before being advertised.
Any adopted alternative browser backend must separately pass the evaluation in §2 and have its architecture/contract changes recorded. Merely enabling a provider enum, uploading a screenshot or observing a successful login is not qualification of that backend.
11. Explicitly deferred scope
- Fresh application/database provisioning for every run.
- Migration from existing Supabase persistence to D1/R2/another job system.
- Moving Caliber application hosting merely to launch Attest.
- Private-network/customer-hosted runners and expanded browser/device matrices.
- Automatic post-deployment triggers before the manual path is reliable.
- Arbitrary customer-uploaded executable test code.
- Unreviewed AI promotion or modification of approved assertions.
- General-purpose testing SaaS onboarding beyond the supported Caliber products.
- Advanced analytics, complex distributed scheduling and broad performance platform capabilities beyond selected, separately qualified suites.
- Automated change-impact selection and exploratory test discovery in V1.
- Autonomous healing. Any later low-risk smoke/regression experiment requires an explicit policy, visible revisions, retained evidence and unchanged intent.
- Kitesurf integration. Current documented WebGL and long authenticated-session limitations make it unsuitable for initial Caliber journeys; reconsider only for a bounded supporting experiment after actual compatibility is demonstrated.
See Kitesurf's current limitations.
The existing target model also includes deeper event correlation, multi-segment recording, rich timeline replay, resumable large uploads and governed archival. These remain tracked capabilities, not implied first-release promises.
12. Verification record and references
Caliber consumer setup
Caliber-specific provisioning commands, endpoint variables and worker launch instructions belong to the consumer's Attest integration guide. The shared cloud host lives in CDX at packages/attest/runtime_supabase/bin/attest_cloud_host.dart. Its deployment remains subject to this plan's acceptance gates.
Inspection evidence
During the planning inspection, these existing tests were run from the shared CDX checkout and all 23 tests passed:
dart test \
packages/attest/engine/test/attest_ai_authoring_test.dart \
packages/attest/cli/test/ai_authoring_route_test.dart \
packages/attest/engine/test/remote_harness_http_server_test.dart \
packages/attest/engine/test/durable_attest_orchestrator_test.dart \
packages/attest/engine/test/remote_harness_execution_provider_test.dartThis was focused local verification with mocked model responses, not production certification. Cloudflare deployment, live model calls, browser execution, full analysis and the complete test suite were not verified in that inspection.
Validated refinement record — 2026-10-07
The approved review refined user terminology, scoped qualification outcomes, intent/requirement lineage, failure classification and V1 scope. It preserved Cloudflare hosting, the existing durable job authority, dedicated QA, reviewed AI authoring and configuration-driven DocsIQ onboarding. Alternative browser tooling remains an experiment, not a selected replacement.
During that review, 20 coverage and qualification-gate tests passed:
dart test \
packages/attest/engine/test/coverage_test.dart \
packages/attest/engine/test/qualification_gate_evaluator_test.dartThe coverage PDF test emitted Helvetica Unicode-support warnings. These results do not establish font correctness, clean full analysis or cloud acceptance. No live provider, deployed browser or customer release-approval workflow was validated during this review.
Current external documentation was checked for browser feasibility and AI workflow comparisons. These describe capabilities, not measured Caliber results:
- BrowserStack requirement-based test generation
- BrowserStack requirement traceability
- mabl auto-heal controls
- TestRail AI generation
- Testim AI capabilities
Requirement traceability and evidence preservation support the existing product quality needs; neither automated gates nor AI output alone establishes regulatory compliance. Caliber must confirm the applicable scope and human acceptance process. Reference reviewed: EU GMP Annex 11.
Caliber consumer references (paths remain owned by flutter_elog):
- Caliber Attest overview
- Extending Attest
- Caliber distribution
- eLog target
- SSO target
- Caliber adapter
- Remote worker
Shared CDX sources, relative to this repository root:
packages/attest/README.mdpackages/attest/docs/ATTEST_WORLD_MODEL.md— normative platform contracts; particularly THS-OWN-1/5, THS-RUN-1/6, THS-CLD-7, THS-SEC-1 and THS-REP-1.packages/attest/engine/lib/src/attest_ai_authoring.dartpackages/attest/editor/lib/src/ai_authoring_workspace.dartpackages/attest/engine/lib/src/remote_harness_http_server.dartpackages/attest/runtime_supabase/bin/attest_cloud_host.dartpackages/attest/cloudflare_local/README.md— local rehearsal only.
Local container milestone — 2026-10-18
The Caliber local proof exercised Supabase sign-in, saved suite revisions, separate API/UI workers, an API integration sequence, an independent-reviewer workflow and the eLog sign-in-screen UI smoke with screenshot and WebM evidence. Saved report template versions produced PDF/DOCX/ZIP with private downloads, generating-user attribution and evidence retained after container recreation. One live Gemini API draft was reviewed, saved and passed on the Docker worker; AI-generated UI journeys were not accepted in that check.
On 2026-10-19 both phase-2 branches incorporated main. The shared post-sync suite passed 293 engine/types/storage/CLI tests (two optional k6 tests skipped) and 25 editor tests, with clean Attest analysis. These source checks do not replace a fresh image build and Docker acceptance pass for the merged revision. Existing services were left stopped and their data preserved.
Cloud deployment, employee federation, authenticated product browser journeys, environment locks, interruption recovery, backup restoration and measured multi-user capacity remain release gates. See the consumer's local trial record and rollout checklist.
Documentation ownership consolidation — 2026-10-07
The approved plan moved to docs/attest/plan.md in CDX. Older active product plans were consolidated here; their superseded milestone questions and broad farm-first sequence are not current commitments. Existing implementation and verification evidence is retained in historical delivery evidence. Historical framework migration records and handoffs live under history/ and must not be interpreted as current execution instructions or release claims. Caliber provisioning stays in the consumer integration guide.