Testing
Test maturity
Section titled “Test maturity”The scenarios below define verification intent for the draft requirements. The artifact checks validate the unified Read and Write OpenAPI document, requirement-to-operation mappings, schema fixtures, and cross-operation journey fixtures. They inspect declared conditional-response headers and body shape, revision parent identity, caller-visible Read and Write capability metadata, and static identity and version handoffs. They do not execute cursor replay, protected-history outcomes, temporal source-unavailability behaviour, authoritative mutations, or workflow processing; those remain implementation scenarios. Passing artifact checks supports contract review only and does not establish deployed runtime behaviour or implementation conformance. Implementation conformance requires separate evaluation of a deployed Registry; an executable deployed-service suite is pending. Legacy Cucumber tests under test/openAPI remain migration evidence for the previous generated CRUD interface.
Artifact validation
Section titled “Artifact validation”From the repository root, install the pinned tools and validate the shipped specification, Consultation artifacts, and Write artifacts:
uv sync --frozenuv run python tools/validate_specification.pyuv run python tools/validate_consultation.pyuv run python tools/validate_write.pyThese checks inspect source structure, local references, requirement-to-operation
links, schemas, examples, fixture exchanges, and the combined journeys in
api/examples/combined-journeys.json. The combined fixture gate covers
Create then Retrieve, conditional Retrieve then Update then Retrieve with a
stale If-Match rejection, Transition then Retrieve, asynchronous completion
then Retrieve, and governed submission and approval followed by authoritative
application and Retrieve. It compares response validators only when a fixture
declares the exact committed guard state, no intervening mutation-relevant
change, settled companion currency, and the same effective authorisation and
disclosure, purpose, and field selection. The fixture also declares a hidden
mutation-relevant source change that leaves the permitted body unchanged,
changes the opaque guard ETag, and makes the prior token receive generic 412.
These are internally coherent artifact declarations, not observations of a
runtime concurrency engine. They do not require immediate visibility from an
ordinary non-companion read or one retained revision per Write. The checks do
not execute a Registry implementation or establish conformance. The scenarios
below describe the separate implementation evidence needed.
Runtime companion-guard procedure
Section titled “Runtime companion-guard procedure”For a deployed implementation, first record the selected Update or Transition, its companion Retrieve, the declared companion-currency condition, and one fixed effective authorisation and disclosure, purpose, and field selection. Then execute these cases against non-production source fixtures:
- Retrieve the companion guard, retain its permitted body and opaque strong
ETag, make a mutation-relevant authoritative change, and submit the old ETag
in
If-Match. Even if an ordinary read remains lagged, the Write must atomically reject the stale guard with the declared generic412and make no authoritative change. - Repeat with a source condition that affects the mutation’s validation,
mapping, or committed effect but is not visible in the permitted body.
Retrieve the companion guard again under the identical effective context;
the body stays equal, the opaque ETag changes, and a Write using the prior
ETag receives generic
412without disclosing the changed condition. - Perform a successful mutation, make no intervening mutation-relevant change, wait for the declared companion currency, and retrieve under the identical effective context. Compare the success ETag only when the later companion read observes the exact committed guard state. Repeat with one context dimension changed to confirm that view-name equality alone is not treated as validator equality.
Retain the requests, responses, source observations, effective-context record, and authoritative before-and-after state. A schema-valid trace or the static fixture does not count as a runtime pass.
Applicability and required fixtures
Section titled “Applicability and required fixtures”Verification starts with an inventory of the Registry services the implementation exposes to each intended metadata audience and the operation options it declares. Every evaluation covers Registry identity, authoritative scope, specification-version metadata, and service discovery. Operation-specific verification follows the selected option keys and the draft conformance model.
Common Record representation requirements are evaluated through each implemented operation that returns Records, using that operation to obtain the fixtures. Identifier lifecycle evidence applies wherever the implementation assigns or maintains Record Identifiers.
An implementation being evaluated provides non-production fixtures according to the following applicability rules:
| Fixture or evidence | Required when |
|---|---|
| Machine-readable Registry metadata, the inventory of exposed Registry BB services, and their linked machine-readable operational contracts, including each operation’s selectable feature declarations | Every evaluation. |
| Two distinct current Records in the same Registry, with expected scoped identity, declared schema, and field meanings | An implemented operation returns Record representations. |
| Record fixtures covering each exposed lifecycle state | An implemented operation returns Record representations in those states. |
| Identifier policy and stability/non-reassignment evidence across source changes and retirement where supported | The implementation assigns or maintains Record Identifiers. |
| Source revision, lifecycle, and provenance evidence for metadata actually exposed | The operation or selected profile exposes or requires this metadata. |
| Consumer contexts and expected permitted representations | An implemented operation returns Record representations. |
| The same Record accessible to consumers with different disclosure rights | The applicable policy provides different disclosure entitlements. |
| An unknown Record Identifier and a known Record outside the selected collection where applicable | Retrieve is selected. |
| Distinct Records in different collections | A Registry exposes multiple collections. |
| One Record available through multiple collections or views | Collections overlap or expose distinct views of one Record. |
| Known, unknown, invalid, and composite selector inputs where supported; a simulated source uniqueness failure | Lookup is selected. |
| Protected and unknown Record Identifiers, selector values, parent Records, and Revision Identifiers for the same consumer context | Retrieve, Lookup, Revision List, or Revision Retrieve is selected and the applicable policy protects Record or revision existence. |
| Zero-, one-, and multi-result collections; typed search criteria and invalid inputs; duplicate sort values where applicable | List or Search is selected, using fixtures relevant to each selected operation. |
| Continuations covering completion, mismatch, expiry, and access changes; a short or empty advancing page where supported; parent Records A and B for cursor replay | List, Search, or Revision List is selected. |
| Declared resource limits, an over-limit request, and a request that reaches the declared timeout outcome | List or Search is selected. |
A current representation and its ETag; expected 200 and 304 headers; expected 200 body and bodyless 304; authorised matching, non-matching, stale, unknown, and protected conditional requests | A Retrieve operation declares conditional-read. |
| Declared selectable data fields, any documented mandatory data fields, unsupported fields, and consumers with different disclosure rights | An operation declares field-selection. |
| An unchanged multi-page result with duplicate primary sort values, plus a configured result window or limit too small for that result | A List or Search operation declares complete-traversal. |
| Source changes between Pages, access changes, and a simulated loss of retained snapshot state | A List or Search operation declares snapshot-traversal. |
| Retained source revisions spanning schema versions, retention boundaries, and known or unknown gaps; an authorised source-history failure | Revision List or Revision Retrieve is selected. |
| Source acceptance timestamps before, at, and after revisions, including a correction; proven item absence; evaluable empty Search; source inability to evaluate history | An operation declares recorded-time. |
| Source applicability dates or instants covering boundaries, overlaps, corrections, no applicable state, evaluable empty Search, and source inability to evaluate history | An operation declares effective-time. |
| Named Search fixtures for each declared equality, range, set-membership, prefix, or domain-specific semantic | Search is selected. |
Caller contexts with different authorised operations; a rich discovery entry containing visible Read, Write, and supporting workflow operations; Read entries with their required view plus readableFields, selectors, searches, known and unknown features, and purpose requirements; Write entries with their declared input and companion-read metadata where applicable; an unknown capability entry; plus a description exceeding each declared bound | Caller capability discovery is offered. |
| Trusted contexts with missing, malformed, allowed, disallowed, and changed controlled purpose values | An operation declares purpose-bound. |
| Nested components, same-target and mixed-target references, and collection-boundary fixtures | A selected view exposes subobjects, references, or embedded collections. |
| Recorded related details and a separately changed current target Record; a parent without an individual reference where supported | A selected view exposes recorded related details or references that may be absent. |
Use synthetic fixtures and record the applicability rationale for each omitted conditional fixture.
Requirement traceability
Section titled “Requirement traceability”The row anchors below are stable test targets for the functionality files. The displayed fr-core#req-* and fr-consultation#req-* labels are historical aliases from the source draft, not current requirement identifiers. Current requirements are DRAFT and unnumbered. Previous requirement text is retained in Git history.
The abbreviated references in this table preserve the former govstack-bb-digital-registries namespaces for traceability only. They are traceability labels for the current verification rows and do not assign identifiers to DRAFT requirements.
API catalog HEAD verification
Section titled “API catalog HEAD verification”For every HTTP service origin, use the published access conditions and a synthetic consumer context permitted to obtain its discovery metadata:
- Send
GET /.well-known/api-catalogand retain the successful response headers and linkset as the comparison resource. - Send
HEAD /.well-known/api-catalogat the same origin, outside any deployment routing prefix. Check the declared successful status and verify that no response body is transferred. - Check that HEAD includes the binding’s required
Linkheader with theapi-catalogrelation, and that its target locates that origin’s API catalog. Check the remaining declared response headers against the same catalog resource; HEAD does not return theservice-metaandservice-desclinkset body. Retain the request, status, headers, and body-absence evidence.
This is a manual procedure for a deployed Registry. The artifact suite separately checks the OpenAPI description and cannot establish that a deployment follows it.
Behaviour scenarios
Section titled “Behaviour scenarios”Discovery is verified through fr-core#req-7 in the traceability table: every exposed service is reachable through the linkset and the metadata document, a family label does not by itself establish that any particular operation exists, and a service without a machine-readable contract fails the requirement. Read scenarios apply to their selected capabilities and declared operation features. Disclosure differences, protected existence, exposed metadata, and resource-limit behaviour additionally require the corresponding conditions in Applicability and required fixtures.
The following cases apply to the selected read capabilities. Run them against each operation and view to which their fixtures apply.
| Case | Exercise | Expected evidence |
|---|---|---|
| Latest Retrieve | Read a known accessible Record without temporal selection, including an exposed inactive state. | Stable identifier, latest valid permitted representation under the declared source currency, and an unchanged source Record. |
| Collection scope | Retrieve a known Record through a collection that excludes it. | The same record-not-available outcome as an unknown identifier. |
| Identity across collections | Read one Record through multiple collections or views, and compare distinct Records across collections in the same Registry. | The same Record retains its identifier; distinct Records have different identifiers within the Registry. |
| Collection actions | Retrieve Records whose identifiers are lookup, search, or resemble action names, and invoke the Collection’s /lookup or /search operation. | Item retrieval under /records/{recordId} and Collection actions route independently; no Record Identifier value is reserved by these action names. |
| Disclosure | Read the same Record under different disclosure entitlements. | Each response matches the permitted fields and metadata for its consumer. |
| Protected existence | Retrieve and Look up unknown and protected fixtures as the same consumer; list and retrieve history while independently varying unknown or protected parent Records and revisions. | Each applicable operation produces the same contract-defined 404 record-not-available and security-relevant content for unknown and protected resources; history does not reveal whether the parent or revision was unavailable; independently generated trace values may differ. |
| Exact Lookup | Submit known, unknown, incomplete, incorrectly typed, and unsupported selector inputs, including composite keys where declared. | One permitted Record for a valid unique match; declared unresolved or validation errors otherwise. |
| Selector uniqueness failure | Supply source data with multiple matches for a unique selector. | The declared failure, with protected-existence policy applied and no arbitrary Record selected. |
| Empty and single-result collections | List and Search fixtures containing zero and one permitted match. | The Page shape is retained; completed results have nextCursor: null. |
| Baseline pagination progress | Continue a result that produces a short or empty advancing Page where the implementation supports that pattern. | The Page respects the effective size, its non-null cursor advances, and the traversal reaches an explicit null cursor without silently restarting. This does not establish exhaustive traversal. |
| Declared resource limits | Exercise each published limit and cause one request to exceed a limit or its processing deadline. | Work stays within the declared limits or ends with the declared timeout outcome; evidence does not infer that backend work scales with page size. |
| Cursor binding | Change Collection, criteria, Search, ordering, view, or effective page size while reusing a cursor; replay a Revision List cursor from parent Record A under parent Record B. | 400 invalid-cursor, with no silent restart and no history from either parent in the cross-parent case. |
| Continuation size | Omit pageSize, then repeat the original effective size on another continuation. | Both retain the bound size. |
| Cursor validity | Submit malformed, tampered, and expired cursors. | The invalid-cursor outcome. |
| Authorisation change | Revoke access between Pages. | Subsequent requests enforce current authorisation before releasing results. |
| Total | Compare any reported total on a later Page with the full permitted query. | An exact count at the declared evaluation point, including matches before the cursor. Live traversal uses the current Page evaluation; snapshot traversal uses its fixed source point filtered for current rights. |
| Live source changes | Insert, update, and remove source Records between Pages. | Observed results match the published currency and pagination policy. |
| Query disclosure | Compare unknown and protected-only query fixtures where match existence is protected. | Items, counts, and continuation patterns preserve the disclosure policy. |
| Unsupported input | Submit undeclared Search names, criteria, List parameters, and sort choices. | The declared validation error. |
| Conditional Retrieve | On a Retrieve declaring conditional-read, send authorised matching, non-matching, and stale validators and repeat against unknown and protected Records; assert declared headers on 200 and 304 and body presence. | Matching current representations return a bodyless 304 with ETag only after authorisation; other permitted reads return 200 with the declared headers, current ETag, and representation body, without weakening protected-existence handling. |
| Field selection | On each operation declaring field-selection, request supported selectable top-level data fields, an unsupported field, and a permitted field withheld from the caller. | recordId, data, and documented mandatory data fields remain present; selectable fields satisfy the declared view schema and disclosure policy; unsupported input fails; continuation repeats the effective selection. |
| Complete traversal | On List or Search declaring complete-traversal, traverse an unchanged collection with duplicate primary sort values and unchanged access, then repeat with a result window or limit that prevents completion. | Every matching permitted Record appears once in the declared order; every Page respects the size bound and the last cursor is null. The limiting case returns the declared failure instead of a completion signal. |
| Snapshot traversal | On List or Search declaring snapshot-traversal, change membership, order, and data between Pages, then test revoked access and unavailable snapshot state. | Source results remain fixed at the first evaluation point, current authorisation can withhold them, and an unavailable snapshot returns 400 invalid-cursor without falling back to live traversal. |
| Revision List | Page retained history with duplicate primary ordering values, schema changes, retention boundaries, documented gaps, and an authorised source-history failure. | Every item recordId matches the parent path; Revision Identifiers and recording times, where supplied, are source-backed; order, schema interpretation, current authorisation, and completeness follow the contract. Source inability returns 409 history-not-available only after protected-existence checks. |
| Revision Retrieve | Request known, unknown, removed, outside-retention, protected, and Record-mismatched revisions, then simulate authorised source-history failure. | Known permitted history has recordId matching the parent path. Unknown and protected parent or revision uses the same 404; authorised source inability can return 409 only after protection checks; no outcome substitutes another revision or current state. |
| Recorded-time selection | Select acceptance cutoffs before, at, and after source revisions and a correction; exercise proven item absence, an evaluable Search with zero permitted matches, source evaluation inability, and an omitted selector. | Selected success uses TemporalRecord or TemporalRecordPage with echoed context, including a 200 empty Page. Proven item absence is 404; source inability is 409 temporal-state-not-available; omitted selection returns latest permitted state without temporal metadata. Revision operations reject the selector; request, occurrence, and registration dates are not substituted. |
| Effective-time selection | Select supported dates or instants at applicability boundaries and where no state applies; exercise proven item absence, an evaluable Search with zero permitted matches, source evaluation inability, and an omitted selector. | Selected success uses TemporalRecord or TemporalRecordPage with echoed context, including a 200 empty Page. Proven item absence is 404; source inability is 409 temporal-state-not-available; omitted selection returns latest permitted state without temporal metadata. Revision operations reject the selector; occurrence and registration dates are used only when the source explicitly defines that meaning. |
| Temporal and traversal separation | Apply a temporal selector to live pagination, and run snapshot traversal without a temporal selector. | Temporal selection alone does not promise a snapshot; snapshot traversal alone does not expose historical states. |
| Reusable Search semantics | Exercise all and only the equality, range, set-membership, prefix, and domain-specific semantics named by each Search. | Typed matching follows the contract; undeclared operators and fields fail; no fuzzy or universal query behaviour is inferred. |
| Caller capabilities | Request a rich capability document under callers with different rights, including visible Read, Write, and supporting workflow operations. For Read entries, include the required view, readableFields, selectors, searches, known and unknown features, and purpose requirements. For Write entries, include the declared input and companion read where applicable. Include an unknown capability entry, exceed declared bounds, and attempt allowed and unavailable Record access. | One bounded no-store document exposes only caller-available advisory metadata. Read metadata identifies its view, while Write and workflow discovery does not imply Record-read authority. readableFields does not imply field-selection; clients skip unknown capabilities and ignore unknown advisory features; normative operation extensions remain closed. Over-bound descriptions fail explicitly without truncation or silent omission; every invocation still authorises and no Record access follows from discovery alone. |
| Purpose binding | Invoke a purpose-bound operation with missing, untrusted, allowed, and unallowed trusted purpose context, then replay a cursor or validator under another purpose. | Invalid purpose fails before disclosure; allowed purpose is bound to continuation and validation and retained for audit without being treated as consent, authority, or downstream-use proof. |
Relationship cases apply to views exposing the corresponding shapes. The household and birth-registration contracts provide concrete examples; their fixtures cover schema validation.
| Case | Exercise | Expected evidence |
|---|---|---|
| Nested components | Read a household with an address and memberships. | Components validate with their declared field meanings and local identifier scope. |
| Fixed-target reference | Validate an individual reference with omitted, matching, and conflicting registryId. | Omitted or matching values resolve to the schema-declared Registry; a conflicting value fails validation. |
| Mixed-target reference | Validate a reference whose field allows multiple Registries. | The reference identifies its target Registry; omission fails validation. |
| Related read binding | Follow a reference for which the contract offers a related read. | The declared target collection, operation contract, and input mapping determine the request without parsing the opaque Record Identifier. |
| Recorded parent details | Change the individual’s current name while leaving the birth registration unchanged. | The birth registration retains nameAtRegistration; a permitted parent component may omit its individual reference. |
| Complete array boundary | Exceed the response bound of a view declaring a complete memberships array. | The declared failure, with no truncated array presented as complete. |
| Related disclosure | Read a household with restricted individual or relationship fields. | References, embedded values, omissions, arrays, and completeness metadata satisfy the consumer’s disclosure policy. |
Evidence retained for audit
Section titled “Evidence retained for audit”Retain the specification version, selected capabilities and operation features, exposed services, published metadata and contracts, and a requirement traceability report with pass, fail, or not-applicable results and applicability rationales. Applicable evidence includes fixtures, requests and responses, schema-validation results, source mapping and currency, revision and temporal evidence, resource-limit and timeout results, pagination and disclosure outcomes, purpose evaluation and audit context, caller-discovery results, and identifier lifecycle evidence.
Record artifact validation separately from deployed implementation results.
Write verification
Section titled “Write verification”The Write verification matrix covers direct creation, update, named transitions, and approval-based requests under their independently selected options. Run uv run python tools/validate_write.py for the unified contract, schema, provenance, positive and negative fixtures, exchanges, and combined journey checks. These are artifact checks. Source commitment, idempotency replay, manual-review separation, concurrent updates, crash recovery, and multi-Record atomicity still require running-implementation evidence.
Unofficial preview · 3.0.0-alpha.2 · Source revision 2350fdf8.
Content snapshot 5e66af7b9724.