External Results API — v1 Specification¶

Status: Draft
Tracking: BGSTM#299 (this spec) · BGSTM#291 (parent epic)
Pydantic schemas: backend/app/schemas/external_results.py

a. Overview¶

The External Results API is the ingestion surface that external test runners use to push execution data into BGSTM. The initial target is Playwright (via the @bgstm/playwright-core reporter); Pytest and Cypress adapters follow in later milestones.

Design goals¶

Versioning model¶

All endpoints are under the /api/v1/ prefix. Within v1, BGSTM commits to backward-compatible additions only (no field removals, no semantic changes to existing fields). A v2 path will be introduced if a breaking change is unavoidable.

Reference implementation¶

The canonical reporter is being built in bgstm-playwright-frameworks#3. That repository's @bgstm/playwright-core package is the primary integration point and serves as the living reference for how a runner consumes this spec.

b. Authentication¶

Token shape¶

Every request to a state-changing endpoint must carry a runner token in the Authorization header:

Authorization: Bearer bgstm_runner_<opaque>

Where <opaque> is a cryptographically random, base-62-encoded string (minimum 32 characters). Tokens are:

• Scoped — each token declares which scopes it may exercise (see table below).
• Revocable — an admin can invalidate a token at any time without rotating other tokens.
• Attributable — every API call is logged under the token's identity for audit purposes.
• Never logged — tokens must never appear in application logs, error responses, or audit records. Log the token's id (UUID) instead.

Token issuance, revocation, and rotation are implemented in BGSTM#296.

Scopes¶

Read-only endpoints also accept a standard user JWT (Authorization: Bearer <jwt>) issued by the regular auth flow.

Error on auth failure¶

c. Endpoints¶

0. Project lifecycle¶

Projects are created through POST /api/v1/projects before sessions are submitted. Every session references a project by project_id, and session creation validates that the project exists. Project reads are available to any authenticated user, while project writes require reviewer or admin.

1. Create session¶

POST /api/v1/external-results/session

Auth scope: external_results:write

Opens a new test-run session. Call this once at the beginning of the suite.

Request body¶

{
  "runner": "@bgstm/playwright-core@0.1.0",
  "project_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "git_sha": "abc123def456",
  "git_branch": "main",
  "ci_url": "https://github.com/org/repo/actions/runs/123",
  "metadata": {
    "os": "ubuntu-22.04",
    "node": "20.11.0"
  }
}

Success response — 201 Created¶

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "started_at": "2025-01-15T12:00:00Z",
  "finished_at": null,
  "runner": "@bgstm/playwright-core@0.1.0",
  "project_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "git_sha": "abc123def456",
  "git_branch": "main",
  "ci_url": "https://github.com/org/repo/actions/runs/123",
  "metadata": { "os": "ubuntu-22.04", "node": "20.11.0" }
}

Error codes¶

session.project_not_found is enforced server-side: session creation now fails fast when the supplied project_id does not resolve to an existing project.

2. Finish / update session¶

PATCH /api/v1/external-results/session/{session_id}

Auth scope: external_results:write

Sets the terminal status of a session. Only passed, failed, or aborted are accepted.

Request body¶

{
  "status": "passed",
  "summary": {
    "total": 42,
    "passed": 40,
    "failed": 2,
    "skipped": 0
  }
}

Success response — 200 OK¶

Returns the updated SessionResponse (same shape as POST /session).

Error codes¶

3. Get session¶

GET /api/v1/external-results/session/{session_id}

Auth scope: external_results:read OR a valid user JWT.

Success response — 200 OK¶

Returns SessionResponse. Same shape as the POST /session response.

Error codes¶

4. Create case result¶

POST /api/v1/external-results/case

Auth scope: external_results:write

Records one test-case execution. Duplicate external_id within the same session_id collapses to the same row (idempotent — see §d).

Request body¶

{
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "test_case_id": null,
  "external_id": "suite > login > should redirect to dashboard",
  "title": "should redirect to dashboard",
  "outcome": "passed",
  "duration_ms": 1234,
  "error_message": null,
  "requirement_ids": [
    "c1234567-89ab-cdef-0123-456789abcdef"
  ],
  "requirement_external_ids": [
    "REQ-LOGIN-001"
  ],
  "auto_register_requirements": false
}

Success response — 201 Created¶

{
  "id": "7f000001-0000-0000-0000-000000000001",
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "test_case_id": null,
  "external_id": "suite > login > should redirect to dashboard",
  "title": "should redirect to dashboard",
  "outcome": "passed",
  "duration_ms": 1234,
  "error_message": null,
  "requirement_ids": ["c1234567-89ab-cdef-0123-456789abcdef"],
  "created_at": "2025-01-15T12:01:00Z",
  "auto_registered": false
}

auto_registered: true when BGSTM created a new test-case record from external_id (i.e. no test_case_id was supplied and the external_id had not been seen before in this project).

requirement_external_ids are resolved against requirements.external_id before links are inserted. By default, unknown external IDs are silently dropped from the response and link set, but they are recorded in the create audit entry under unresolved_requirement_external_ids for diagnostics. Setting auto_register_requirements=true opts into creating stub Requirement rows (external_id populated, draft/functional/medium defaults) so the resolved UUID is linked immediately and future re-POSTs resolve to the same requirement.

Error codes¶

5. Update case result¶

PATCH /api/v1/external-results/case/{id}

Auth scope: external_results:write

Partial update: any combination of outcome, duration_ms, and error_message.

Request body¶

{
  "outcome": "flaky",
  "duration_ms": 5000,
  "error_message": "Element not found after retry"
}

All fields are optional. Omitted fields are left unchanged.

Success response — 200 OK¶

Returns the updated CaseResultResponse.

Error codes¶

6. Get case result¶

GET /api/v1/external-results/case/{id}

Auth scope: external_results:read OR a valid user JWT.

Success response — 200 OK¶

Returns CaseResultResponse.

Error codes¶

7. Upload artifact¶

POST /api/v1/external-results/artifact

Auth scope: external_results:write

Multipart upload (multipart/form-data). The four form fields below are locked — the reporter at bgstm-playwright-frameworks pinned SHA ab5d7c1 sends exactly these names.

Request parts (multipart/form-data)¶

Note: There is no separate content_type or size_bytes form field. The content_type is read from the file part's Content-Type header and the size_bytes is counted while streaming the body.

Content-type allowlist¶

The following MIME types are accepted. Requests with any other content-type are rejected with 415 unless kind=other, which bypasses the allowlist entirely.

image/png  image/jpeg  image/gif  image/webp
video/webm  video/mp4  video/mpeg
application/zip  application/x-zip-compressed  application/octet-stream
text/plain  application/json

Size enforcement¶

The server enforces BGSTM_ARTIFACT_MAX_BYTES (default 50 MiB) by parsing the multipart body with streaming-form-data. A _SizeLimitExceeded sentinel is raised inside the part-data callback the moment the cumulative byte count of the file part exceeds the limit — the stream loop exits immediately and the server returns 413 without reading further bytes from the connection. Server aborts streaming parse mid-request when the cumulative byte count exceeds BGSTM_ARTIFACT_MAX_BYTES; bytes past the limit are not read from the connection.

Success response — 201 Created¶

{
  "id": "af000001-0000-0000-0000-000000000001",
  "case_result_id": "7f000001-0000-0000-0000-000000000001",
  "kind": "screenshot",
  "filename": "failure-state.png",
  "content_type": "image/png",
  "size_bytes": 20480,
  "url": "https://storage.example.com/artifacts/failure-state.png",
  "created_at": "2025-01-15T12:02:00Z"
}

Error codes¶

Audit log¶

Every successful upload writes an external_results.artifact.upload audit entry. The details JSON always contains these five fields (the smoke workflow at PR #314 reconstructs case-result → artifact relationships from these fields):

{
  "case_result_id": "<uuid>",
  "kind": "<artifact_kind>",
  "size_bytes": 20480,
  "filename": "failure-state.png",
  "content_type": "image/png"
}

d. Idempotency rules¶

Case results — deduplication by external_id¶

When a runner POSTs a case result with an external_id that already exists for the same session_id, BGSTM returns the existing row (200 OK) rather than creating a duplicate or returning 409. This means re-running POST /case with identical input is safe.

Traceability links — requirement_ids¶

Inserting a (test_case_id, requirement_id) link that already exists is a no-op. No error is raised. The same rule applies when the caller supplies requirement_external_ids: after each external ID is resolved (or auto-registered) to a requirement UUID, duplicate links are ignored.

Artifact uploads are not deduplicated; reporters that retry an upload (for example on transient network errors) may create duplicate artifact rows, and this is acceptable for v0.1.

e. Status transition rules¶

Session status¶

Case result outcome¶

f. Error model¶

All error responses share a single envelope:

{
  "code": "runner_token.invalid",
  "message": "The provided runner token is invalid or has been revoked.",
  "details": {
    "token_prefix": "bgstm_runner_"
  }
}

Error code registry¶

g. Storage abstraction¶

Artifact binaries are stored via a pluggable StorageBackend abstraction (backend/app/storage/).

Backend selection¶

Controlled by the BGSTM_STORAGE_BACKEND environment variable:

Configuration¶

Recommended deploy hardening¶

Configure your reverse proxy to enforce a body-size cap as first-line DoS defense, e.g. nginx client_max_body_size 60m; (slightly larger than BGSTM_ARTIFACT_MAX_BYTES to allow multipart framing overhead). The handler also enforces the cap at the application layer via streaming abort, but the proxy cap protects against attackers exhausting Python worker time.

StorageBackend ABC¶

class StorageBackend(ABC):
    def save(self, stream, *, key: str, content_type: str) -> StorageResult: ...
    def url_for(self, key: str) -> str: ...

get_storage() (in backend/app/storage/__init__.py) is a function, not a module-level singleton, so tests can swap settings without import-time side effects.

Dev-only static route¶

When BGSTM_STORAGE_BACKEND=local, main.py mounts a StaticFiles route at /artifacts pointing at BGSTM_ARTIFACTS_DIR. This route is not mounted for any other backend.

h. Observability¶

Audit log¶

Every state-changing call now writes an audit log entry recording actor identity, action, and affected resource (implemented in BGSTM#297).

Action taxonomy¶

Each audit entry records:

• actor_kind — user or runner_token.
• user_id — UUID for user actors, nullable for runner-token actors.
• actor_token_id — UUID for runner-token actors, nullable for user actors (never the raw token string).
• action — one of the values above.
• resource_type — external_session, case_result, or artifact.
• resource_id — UUID of the created/updated resource.
• project_id — UUID of the project.
• details — JSON snapshot of the mutation (before/after where applicable).

Action taxonomy is enforced on the write paths: no state-changing External Results endpoint may skip audit emission.

i. Reference implementation¶

The TypeScript reference reporter lives in bgstm-playwright-frameworks. BGSTM smoke validation is pinned to reference reporter merge commit ab5d7c1b0740cfbb5004cb6a14851c541364451e to keep contract verification deterministic.

The BGSTM workflow .github/workflows/external-results-smoke.yml runs this pinned reporter against a live BGSTM stack on relevant pull requests, every push to main, and manual dispatch. It validates persisted session/case/artifact/audit outcomes end-to-end for the External Results v1 contract.

When the reporter releases a new version, bump only the pinned ref in external-results-smoke.yml in a dedicated PR, then confirm the smoke workflow is green at that new SHA before merging. Do not track main directly from the frameworks repository.