RGB Surface Analyzer

Version: 1.0
Date: 2026-07-21
Status: ✅ Production-Ready
Last Updated: 2026-08-08
Author: KavAI Backend Team

NoteRelationship to the platform image-skill boundary (2026-08-08)

This page describes DataADK’s shipped analyzer path, which remains the production implementation. A platform-level successor exists but DataADK has not been ported to it (owner decision 2026-08-08 — DataADK is not modified; the port is deferred):

  • The provider-neutral contract surface is IMAGE_ANALYSIS_RESULT (CONTRACT-CDC-001 v2.4 §7.8, proposed); IMAGE_WITH_ANNOTATIONS remains this analyzer’s KavApps-compatible emission during the transition.
  • The analyzer’s prompt, vocabulary, cap, and pixel→normalized safety net are carried verbatim by kavai-image-skills (image.surface-condition@1.0.0), which acquires media through the host resolve_media capability instead of the JWT/SAS/fetch pipeline described below.
  • The surface-analysis certification capability row skips for engines that do not declare it and blocks for engines that do. The declaring set is in ai/src/kavai/systems/e2e_registry.json, pinned by test_the_engines_that_declare_the_capability — not restated here, because a prose list of adopters is wrong as soon as one more adopts.

Proposal: docs/proposals/20260808_uuid_only_image_analysis_skills.md · plan: docs/plans/20260808_image_skills_certification_execution.md.

Overview

rgb_surface_analyzer is a ADK LlmAgent wrapped as AgentTool and wired to the database_specialist agent. It is designed to perform visual surface anonlay detection on full-resolution RGB images. It receives up to multi_image_cap (default: 5) images in a single Gemini Vision call and returns structured, per-image surface-condition detections (currently limitted to stains and corrosion) with normalized bounding box coordinates.

Role and Position

  • Role: LLM agent wrapped and wired as an AgentTool (using google.adk.tools.agent_tool.AgentTool).
  • Position: It is owned and called by the database_specialist executor agent as a tool.
  • Caller: database_specialist invokes it conditionally when resolver_classifier.query_type == "surface_analysis".
  • Callee: It does not call any other agents.

Tools

  • Tools bound to it: None. rgb_surface_analyzer has no tools bound to it.

LLM Context Design

This section details every item that ends up in the actual request sent to the LLM, categorized by its lifecycle and mutability.

Context Components

  1. Instruction Text (System Instruction) [Static / Dynamic · session (via template substitution)]
    • The core instruction text defines the agent’s role, persona, input contract, output contract, coordinate system, self-check rules, confidence guidelines, error handling, few-shot examples, and critical output rules.
    • It is loaded from adk_agents_config.yaml and processed by dynamic_instruction_provider in factory.py.
    • It contains placeholders that are substituted at runtime:
      • {multi_image_cap} [Static]: Replaced with str(load_multi_image_cap()) (loaded from adk_agents_config.yaml at startup). It is inserted exactly 3 times in the instruction text.
      • {surface_analysis_vocabulary} [Static]: Replaced with the allowed condition labels (e.g., "stain", "corrosion") loaded from adk_agents_config.yaml at startup. It is inserted exactly 2 times in the instruction text.
    • There are no session-specific placeholders
  2. Input Arguments (JSON) [Dynamic · per-turn]
    • ADK serializes the calling agent’s arguments as JSON in the first text part of llm_request.contents when the agent is invoked as an AgentTool with input_schema=RGBSurfaceAnalyzerInput.
    • The schema contains:
      • image_ids: Optional[List[str]] (default None) — list of image UUIDs to analyze.
      • image_id: Optional[str] (default None) — deprecated single-image alias; the before_model callback normalizes this to image_ids = [image_id] when present.
      • dataset_slug: str (default "") — dataset slug for RLS-scoped SAS URL minting.
    • This is recomputed every turn based on the executor’s findings.
  3. Labeled Image Parts (Part.from_bytes) [Callback-mutated / Dynamic · per-turn]
    • The before_model callback (rgb_surface_analyzer_before_model) fetches the image bytes concurrently and attaches them to llm_request.contents.
    • Each image is preceded by a text label part: "Image N (id: <uuid>):" followed by the binary image part (Part.from_bytes).
    • This is recomputed every turn based on the resolved image IDs.
  4. Output Schema Formatting Instructions [ADK-injected / default]
    • ADK automatically appends formatting instructions to the request or configures the model’s response schema to enforce JSON output conforming to RGBSurfaceAnalyzerResponse.

Explicit Callouts

  • include_contents: Set to "none" in adk_agents_config.yaml. This means that conversation history is not pulled into the request. The agent only sees the current turn’s input (the JSON arguments and the attached images). It does not see previous turns’ conversation history, tool calls, or responses. This is a critical design choice to keep the context window clean and focused purely on the vision analysis task.
  • global_instruction: Not applicable, as rgb_surface_analyzer is a sub-agent (AgentTool) and not the root agent.

State I/O

Reads

  • temp:jwt_token (fallback JWT token used by before_model callback to mint Azure SAS URLs when the request-scoped contextvar is empty; written by database_specialist_before_tool)
  • temp:rgb_image_dims_<image_id> (cached image dimensions read in after_model to normalize pixel coordinates)

Writes

  • temp:rgb_image_dims_<image_id> (cached image dimensions extracted via PIL in before_model)
  • rgb_surface_analyzer_input_error (diagnostic error message written by before_model if input validation or image fetch fails completely)
  • rgb_surface_analyzer (the final structured RGBSurfaceAnalyzerResponse written automatically by ADK’s output_key mechanism)

Scope and Persistence

  • Scope: Session-scoped. rgb_surface_analyzer has no after_agent_callback registered. The rgb_surface_analyzer output key is written directly to ctx.state by ADK’s own output_key mechanism — not by a manual persist_session_state call. Persistence of the parent session (including this key) is handled by database_specialist_after_agent.
  • Temp Prefixes: Ephemeral keys starting with temp: (like temp:jwt_token and temp:rgb_image_dims_<image_id>) or __ (double underscore) are used in callbacks but are stripped before persistence to session.db to prevent leaking transient state or secrets.
  • Session State: Standard keys (rgb_surface_analyzer, rgb_surface_analyzer_input_error) are persisted to session.db and are cleared at the start of each turn by _apply_per_turn_reset in agent.py to prevent stale data bleed across turns.

Output Schema

The agent’s output is strictly constrained to the RGBSurfaceAnalyzerResponse Pydantic schema (defined in sensor_analysis_models.py).

RGBSurfaceAnalyzerResponse Schema Fields

  • RGBSurfaceAnalyzerResponse
    • results: List[PerImageResult] (one entry per analyzed image).
  • PerImageResult
    • image_id: str (UUID of the analyzed image)
    • image_analyzed: bool (True on success, False on fetch/decode failure)
    • bounding_regions: List[Detection] (detected anomalies)
    • overall_assessment: str (plain-text summary)
    • error: Optional[str] (diagnostic message on failure)
  • Detection
    • condition_type: str (must be in surface_analysis_vocabulary, e.g., "stain", "corrosion")
    • confidence: float ([0.0, 1.0])
    • x, y, width, height: float (normalized coordinates in [0.0, 1.0])
    • description: Optional[str] (max 200 chars)

Note: ADK passes the schema to Gemini’s constrained decoding API, forcing the model to output valid JSON matching the schema. To prevent crashes, the schema uses extra="ignore" (never extra="forbid") and avoids typing.Union, typing.Literal, or Dict[str, Any].

Callbacks

This section documents the ADK callbacks for rgb_surface_analyzer.

Callback Used? Behavior
before_agent ❌ N/A
before_model ✅ rgb_surface_analyzer_before_model
before_tool ❌ N/A (no tools)
after_tool ❌ N/A (no tools)
after_model ✅ rgb_surface_analyzer_after_model
after_agent ❌ N/A
on_model_error ❌ Intentionally not registered. rgb_surface_analyzer runs inside an isolated ADK AgentTool runner; error handling is delegated to the parent database_specialist pipeline.

rgb_surface_analyzer_before_model

Resolves input args, caps at _MULTI_IMAGE_CAP, reads JWT, mints Azure SAS URLs, fetches image bytes concurrently, caches image dimensions, and attaches labeled Part.from_bytes to llm_request.contents.

  1. Argument Resolution: Resolves input args (image_ids, dataset_slug) from llm_request.contents and caps the list at _MULTI_IMAGE_CAP (default: 5).
  2. JWT and SAS Minting: Reads the JWT token from the request-scoped contextvar or state["temp:jwt_token"] fallback, and concurrently mints Azure SAS URLs (TTL: 600s).
  3. Image Fetch and Dimension Caching: Fetches image bytes using httpx.AsyncClient with exponential backoff (max 2 retries, 30s timeout). Decodes image bytes using PIL to extract and cache dimensions in state["temp:rgb_image_dims_<image_id>"].
  4. Request Mutation: Attaches labeled text parts ("Image N (id: <uuid>):") and binary image parts (Part.from_bytes) to llm_request.contents. Returns None to proceed with the mutated request.

rgb_surface_analyzer_after_model

Intercepts the raw response, parses the JSON payload, checks for pixel-valued coordinates, normalizes them using cached image dimensions, clamps them to [0.0, 1.0], and re-serializes the response.

  1. Coordinate Normalization: Scans for pixel-valued coordinates (any coordinate > 1.0). If found, looks up cached image dimensions from state["temp:rgb_image_dims_<image_id>"] and normalizes them (x / width, y / height, etc.) clamping to [0.0, 1.0].
  2. Failure Handling: If dimensions are missing, marks the result as failed (image_analyzed=False, bounding_regions=[], error populated) to prevent meaningless clamped coordinates.
  3. Response Mutation: Re-serializes the JSON and returns the mutated llm_response so ADK re-runs validation. If no rewrite is needed, it returns None (pass-through).

Planner

  • Planner usage: None. rgb_surface_analyzer is a standard LlmAgent and does not use a planner.

ADK Version Verified Against

  • Verified against google_adk version 1.26.0.

Generation Config

Parameter Value Source / Notes
Model gemini-3.5-flash Pinned in adk_agents_config.yaml
Temperature Default (not set) Inherits Gemini API default
Max Tokens Default (not set) Inherits Gemini API default
Safety Settings Default (not set) Inherits Gemini API default
Stop Sequences Default (not set) Inherits Gemini API default
Retry Options 3 attempts, 1.0s initial delay, 16.0s max delay, exp base 2, retry on [408, 429, 500, 502, 503, 504] Configured at SDK level via Gemini model object in factory.py

Open Questions / Unverified Items

  • Gemini API Bug: The workaround of using Part.from_bytes instead of Part.from_uri is due to an open Gemini API bug where Part.from_uri returns 403 on Azure SAS URLs. This should be reverted to Part.from_uri once Google resolves the bug to reduce memory overhead and latency.
  • Coordinate Normalization Fallback: If Gemini fails to echo the image_id in its response, the after-model callback cannot look up the cached dimensions for that image, resulting in a hard failure for that image result. This is a known limitation of relying on the model to echo identifiers.