Web/AI Interface

Understanding the AG-UI Protocol and SSE Event Streams

Introduction

The KAP Frontend (Web) and AI Backend (Server) communicate using a real-time, event-driven protocol based on Server-Sent Events (SSE). This “AG-UI” protocol ensures that agent reasoning, tool execution, and rich media results are streamed to the user with zero latency.

Communication Flow

Canonical contract: this chapter is the narrative guide. The authoritative interface specification — payload shapes, per-backend emission matrix, the four gallery contracts, ordering guarantees — is CONTRACT-CDC-001, lint-enforced against the source at docs/api_standards/cdc_event_contract.md.

The web client initiates a chat by sending a POST request to the streaming endpoint. The AI server then responds with an open SSE connection.

  • Endpoint: /chat/agui/stream
  • Method: POST
  • Headers:
    • Authorization: Bearer <JWT_TOKEN>
    • Content-Type: application/json

Request Payload (ChatRequest)

{
  "message": "List my datasets",
  "system_type": "dataadk",
  "threadId": "user-thread-001",
  "runId": "optional-uuid"
}

Standard Event Lifecycle

All runs follow a strict event sequence to manage UI state transitions.

Event Type Purpose Key Fields
RUN_STARTED Signal start of execution. runId, threadId
TEXT_MESSAGE_START Start a new assistant response. messageId, role
TEXT_MESSAGE_CONTENT Incremental tokens/deltas. delta
TEXT_MESSAGE_END End of current message unit. messageId
RUN_FINISHED Mark end of full execution. result (metadata)
RUN_ERROR Handle backend failures. error (code, message)

Tool Execution Visibility

KAP exposes tool calls to the AG-UI so users can see the “inner monologue” of the agents.

  1. TOOL_CALL_START: Name of the tool (e.g., list_datasets).
  2. TOOL_CALL_ARGS: Incremental JSON arguments.
  3. TOOL_CALL_END: Signal arguments are complete.
  4. TOOL_CALL_RESULT: The stringified output from the tool.

Custom KAP Events

KAP extends the base AG-UI protocol with CUSTOM events discriminated by customType — DATASET_LIST, IMAGE_GALLERY, MARKDOWN_REPORT (with [[component:identifier]] directives), IMAGE_ANOMALIES, THREE_D_OVERVIEW, IMAGE_ANALYSIS_RESULT (v2.5), and reserved provenance events.

Payload shapes, which backends emit which events, and the four gallery contracts the frontend must handle (gallery event / markdown component marker / inline thumbnails / [image_uuid] reference lines) are specified normatively in CONTRACT-CDC-001 §7 — not duplicated here.

IMAGE_ANALYSIS_RESULT (§7.8) is the canonical image-analysis surface: UUID-correlated per-image findings with normalized geometry, plus the optional provenance record naming what produced them. The frontend records results by (image_id, skill.id) (web/lib/ag-ui/image-analysis.ts) and the gallery image viewer merges every skill’s findings for the open image into its BoundingBoxVisualizer annotation layer — the same overlay pipeline as stored annotations, with a re-run of one skill replacing that skill’s overlay and leaving the others alone.

The pair is the key, not the image. An image-keyed store is correct only while exactly one skill exists: run a second one and the first skill’s boxes disappear with no error to notice.

Model output is drawn as model output. Findings from this event, and undecided suggestions read back from GET /api/images/{id}/suggestions, carry kind: 'suggestion' into BoundingBoxVisualizer: dashed, one colour outside the category palette, and labelled on the box. Stored annotations stay solid and coloured by category. A reviewer deciding whether to accept a box must never have to work out which kind it is, and colour alone cannot say so here because colour already means category.

The two are also durable in different ways. An IMAGE_ANALYSIS_RESULT overlay is session-scoped and vanishes on reload; an analysis requested from the viewer’s own Analyze control is recorded — attempt, provenance and suggestions — and comes back on the next visit, along with what anyone decided about it. Accepting a suggestion is what turns it into a stored annotation, and from that moment it is drawn solid like any other.

A selected set of images can be analyzed from the gallery too, and that path is foreground work: one POST /api/images/{id}/analyze per image, sequential, capped at 25, running in the browser while the tab is open. There is no batch endpoint, deliberately — each image records its own attempt, so a selection is indistinguishable downstream from that many separate clicks. Stopping stops the queue and lets the image already sent to the model finish and record; aborting its request would discard the reply without un-running the analysis. Durable runs over a collection are a different mechanism (WP-6, analysis_runs).

State Synchronization

The AI server can sync complex agent state (e.g., progress bars, current step) via:

  • STATE_SNAPSHOT: A full replacement of the client-side agent state.
  • STATE_DELTA: Incremental updates using JSON Patch (RFC 6902) format.

Frontend Consumer (useAGUIClient / useAGUIEvents)

The React frontend parses the SSE stream with a pair of hooks and dispatches events into React state — there is no global Redux store:

  • useAGUIClient (hooks/use-agui-client.ts) implements the AG-UI AgentSubscriber pattern. It holds the message list and agent state in React state, buffers TEXT_MESSAGE_CONTENT deltas for smooth text rendering, and exposes callback refs that rich CUSTOM events (image galleries, dataset lists) fire into the owning component.
  • Resilience: the JWT is freshness-checked and refreshed before each send (useJwtAuth), and messages composed while offline are queued (lib/message-queue) and drained automatically when connectivity returns.
  • useAGUIEvents (hooks/use-agui-events.ts) normalizes event-type casing, de-duplicates events by id, and hands them to the parent component.

Last Updated: 2026-07-29