> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alakazam.gg/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision and interactive objects

> Turn a world's live frame into clickable objects. Read what's on screen, detect bounding boxes, and pin a choice to an object so the player clicks it to branch.

The main interactive feature of a world is clicking the objects you see.
The Vision API is how you build it: read a live frame in plain text, detect
bounding boxes for named objects, then **anchor** a graph event to one of those
objects. Once an event has an anchor, the shipped player runs a live detect loop
and renders that choice as a clickable chip pinned to the object in the
streaming video, so the player clicks the red door to take the "open the red
door" branch instead of picking it from a list.

All five routes are thin, metered wrappers over the same vision backend that
powers the editor. The pipeline is always the same shape:

* **`/v1/perceive`**: a text read of a frame (what objects are here?). No boxes.
* **`/v1/ground/objects`** / **`/v1/ground/scene`**: bounding boxes for named
  objects (where exactly is each one?).
* an **`anchor`** on an event: pins a choice to a detected object so the player
  renders it as a clickable hotspot. See [Editing the graph](/graph-editing) for
  event edits, and [Embedding](/embedding) for where that clickable player runs.
* **`/v1/seed-frame`** / **`/v1/probe`**: paint a fresh frame, or probe one into
  a starter world.

<Note>
  Every vision route is a **write**: it needs a secret key (`sk_`) with
  `worlds:write`, because each call makes real, paid vendor spend. A publishable
  key (`pk_`) only carries `worlds:read` and is rejected here. Never ship a secret
  key to a browser. Run these from your server.
</Note>

## Metered and fail-closed

Each vision call reserves quota **before** it hits the backend, and refunds on
any failure. The failure modes are strict and uniform:

* **`402`**: you're over your daily cap for that meter (`perceive`, `ground`,
  `seed_frame`, or `probe`). Reserve-before-spend, so you're never billed past
  the cap.
* **`503`**: the quota backend is unavailable. The call **fails closed** rather
  than letting unmetered spend through.
* **`413`**: the base64 frame exceeds the per-call size cap.
* **`400`**: a required field is missing (no frame, empty `labels`, …).
* **`502`**: the vision backend itself errored, and your quota is refunded.

<Note>
  A `200` with an **empty** result (no hotspots, or an empty perception) is still
  charged. The paid vendor call was made regardless of whether it found
  anything. Refunding empties would open free-abuse, so you pay for the work
  performed.
</Note>

Grounding meters **per label**: one `ground` unit per label you ask for, capped
at **6** (the backend's hard cap on returned boxes). Ask for 6 labels and you
spend 6 `ground` units. Anything past 6 is dropped before it's billed.

## Read a frame with /perceive

`POST /v1/perceive` is a **text-only** read of a frame (no bounding boxes). Use
it first, to discover what's on screen before you decide which objects to make
clickable. It returns a one-line `summary`, the `visible` objects, the
`affordances` (things the player could act on), and the `exits`.

Pass the frame as a base64-encoded JPEG in `frame_b64`. `room_name` and
`room_description` are optional hints that sharpen the read.

<CodeGroup>
  ```bash cURL theme={null}
  # base64-encode a JPEG frame, then POST it
  FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

  curl -X POST 'https://api.alakazam.gg/v1/perceive' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d "{
      \"frame_b64\": \"$FRAME_B64\",
      \"room_name\": \"the boiler room\"
    }"
  ```

  ```javascript JavaScript theme={null}
  import { readFileSync } from "node:fs";

  const frame_b64 = readFileSync("frame.jpg").toString("base64");

  const perception = await fetch("https://api.alakazam.gg/v1/perceive", {
    method: "POST",
    headers: {
      Authorization: "Bearer sk_live_…",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ frame_b64, room_name: "the boiler room" }),
  }).then((r) => r.json());

  console.log(perception.summary);
  console.log("visible:", perception.visible); // → ["a red valve", "a rusted door", …]
  ```

  ```python Python theme={null}
  import base64, requests

  frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

  perception = requests.post(
      "https://api.alakazam.gg/v1/perceive",
      headers={
          "Authorization": "Bearer sk_live_…",
          "Content-Type": "application/json",
      },
      json={"frame_b64": frame_b64, "room_name": "the boiler room"},
  ).json()

  print(perception["summary"])
  print("visible:", perception["visible"])
  ```
</CodeGroup>

The response is `{ summary, visible[], affordances[], exits[] }`, all strings and
no coordinates. Take the labels you want to make clickable and feed them to a
grounding route next.

## Detect objects with /ground/objects

`POST /v1/ground/objects` is the **open-vocabulary** detector: pass `labels[]`
and get back a hotspot box for each one it finds. This is what turns a name like
`"the red valve"` into geometry you can pin a choice to.

<CodeGroup>
  ```bash cURL theme={null}
  FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

  curl -X POST 'https://api.alakazam.gg/v1/ground/objects' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d "{
      \"frame_b64\": \"$FRAME_B64\",
      \"labels\": [\"the red valve\", \"the rusted door\"]
    }"
  ```

  ```javascript JavaScript theme={null}
  import { readFileSync } from "node:fs";

  const frame_b64 = readFileSync("frame.jpg").toString("base64");

  const { hotspots } = await fetch(
    "https://api.alakazam.gg/v1/ground/objects",
    {
      method: "POST",
      headers: {
        Authorization: "Bearer sk_live_…",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        frame_b64,
        labels: ["the red valve", "the rusted door"],
      }),
    },
  ).then((r) => r.json());

  for (const h of hotspots) {
    const [x, y, w, h_] = h.bbox; // 0-1000 normalized
    console.log(h.label, h.confidence, [x, y, w, h_]);
  }
  ```

  ```python Python theme={null}
  import base64, requests

  frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

  hotspots = requests.post(
      "https://api.alakazam.gg/v1/ground/objects",
      headers={
          "Authorization": "Bearer sk_live_…",
          "Content-Type": "application/json",
      },
      json={"frame_b64": frame_b64, "labels": ["the red valve", "the rusted door"]},
  ).json()["hotspots"]

  for h in hotspots:
      print(h["label"], h["confidence"], h["bbox"])  # bbox = [x, y, w, h]
  ```
</CodeGroup>

Each hotspot is `{ label, bbox, confidence }`. `bbox` is `[x, y, w, h]` in
`0-1000` normalized coordinates: the box is `w`/1000 of the frame wide and
`h`/1000 tall, with its top-left corner at `(x/1000, y/1000)`. Because the
coordinates are normalized, they map onto the streaming video at any resolution.

<Warning>
  Grounding is metered **per label** (cap **6**). `labels: ["a", "b", "c"]` spends
  three `ground` units. A 7th label is dropped before it's billed. `frame_jpeg_base64`
  is accepted as an alias for `frame_b64`.
</Warning>

## Pin choices with /ground/scene

`POST /v1/ground/scene` is the **closed-vocabulary** variant: instead of loose
labels, you pass the graph choices you already have as
`expected_choices: [{ id, label }]`, and each returned hotspot carries a
`choice_id` linking the box back to the choice it matched. Use this when you're
grounding a specific set of branches (rather than free-form object discovery).
The `choice_id` tells you exactly which event to anchor.

<CodeGroup>
  ```bash cURL theme={null}
  FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

  curl -X POST 'https://api.alakazam.gg/v1/ground/scene' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d "{
      \"frame_b64\": \"$FRAME_B64\",
      \"location_name\": \"the boiler room\",
      \"expected_choices\": [
        { \"id\": \"turn_valve\",  \"label\": \"the red valve\" },
        { \"id\": \"open_door\",   \"label\": \"the rusted door\" }
      ]
    }"
  ```

  ```javascript JavaScript theme={null}
  import { readFileSync } from "node:fs";

  const frame_b64 = readFileSync("frame.jpg").toString("base64");

  const { hotspots } = await fetch(
    "https://api.alakazam.gg/v1/ground/scene",
    {
      method: "POST",
      headers: {
        Authorization: "Bearer sk_live_…",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        frame_b64,
        location_name: "the boiler room",
        expected_choices: [
          { id: "turn_valve", label: "the red valve" },
          { id: "open_door", label: "the rusted door" },
        ],
      }),
    },
  ).then((r) => r.json());

  // hotspot.choice_id links each box back to the expected_choice it matched
  for (const h of hotspots) console.log(h.choice_id, "→", h.bbox);
  ```

  ```python Python theme={null}
  import base64, requests

  frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

  hotspots = requests.post(
      "https://api.alakazam.gg/v1/ground/scene",
      headers={
          "Authorization": "Bearer sk_live_…",
          "Content-Type": "application/json",
      },
      json={
          "frame_b64": frame_b64,
          "location_name": "the boiler room",
          "expected_choices": [
              {"id": "turn_valve", "label": "the red valve"},
              {"id": "open_door", "label": "the rusted door"},
          ],
      },
  ).json()["hotspots"]

  for h in hotspots:
      print(h["choice_id"], "→", h["bbox"])
  ```
</CodeGroup>

Both grounding routes share the same `{ hotspots: [...] }` response shape. Only
`choice_id` differs: it's populated on `/ground/scene` (matched to an
`expected_choice`) and `null` when a label wasn't an expected choice.

## The interactive payoff: anchors

Grounding on its own is only detection. To make an object clickable in the
player, attach an `anchor` to the event whose branch that object should
trigger:

```json theme={null}
{ "label": "the red door", "aliases": ["red door", "crimson door"] }
```

When an event carries `anchor.label`, the shipped player (`StateMachinePlayer`
and the [embed](/embedding)) runs a live VLM detect loop on the streaming
frame and renders that choice as a clickable chip pinned over the detected
object, the same `[x, y, w, h]` `0-1000` box geometry the grounding routes
return. The player clicks the object to take the branch. `aliases` give the
detector alternate names to match on when the object's on-screen appearance
drifts.

You set the anchor with an ordinary graph edit (events are addressed by their
unique `name`): either the dedicated event patch or a batch `update_event` op.
Both funnel through the same fail-closed [validation gate](/graph-editing#the-validation-gate).

<CodeGroup>
  ```bash PATCH one event theme={null}
  curl -X PATCH \
    'https://api.alakazam.gg/v1/worlds/WORLD_ID/events/Open%20the%20red%20door' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d '{
      "patch": {
        "anchor": { "label": "the red door", "aliases": ["red door", "crimson door"] }
      }
    }'
  ```

  ```bash Batch op theme={null}
  curl -X POST 'https://api.alakazam.gg/v1/worlds/WORLD_ID/ops' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d '{
      "ops": [
        {
          "op": "update_event",
          "name": "Open the red door",
          "patch": { "anchor": { "label": "the red door" } }
        }
      ]
    }'
  ```

  ```javascript JavaScript theme={null}
  await fetch(
    "https://api.alakazam.gg/v1/worlds/WORLD_ID/events/" +
      encodeURIComponent("Open the red door"),
    {
      method: "PATCH",
      headers: {
        Authorization: "Bearer sk_live_…",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        patch: {
          anchor: { label: "the red door", aliases: ["red door", "crimson door"] },
        },
      }),
    },
  );
  ```
</CodeGroup>

## End-to-end: a frame becomes a clickable object

<Steps>
  <Step title="Perceive the frame">
    `POST /v1/perceive` with the frame's `frame_b64`. Read `visible` and
    `affordances` to find the objects worth making interactive, such as `"the red
          valve"` and `"the rusted door"`.
  </Step>

  <Step title="Ground the labels you picked">
    `POST /v1/ground/objects` with those labels (or `/ground/scene` if you're
    matching them to existing choices). Confirm each object comes back with a
    plausible `bbox` and a decent `confidence` before you wire it up. A label the
    detector can't find won't render a chip.
  </Step>

  <Step title="Anchor the events">
    For each object, `PATCH /v1/worlds/{id}/events/{name}` (or a batch
    `update_event` op) to set `anchor.label` on the event that object should
    trigger. Add `aliases` for names the detector might see instead. The write is
    validated fail-closed like any graph edit.
  </Step>

  <Step title="Play it">
    Embed the world (see [Embedding](/embedding)). The player detects the anchored
    objects live and renders each anchored choice as a clickable chip pinned to its
    object. The player clicks the object to branch.
  </Step>
</Steps>

## Paint or probe a frame

Two more routes round out the pipeline, for when you don't already have a frame:

* **`POST /v1/seed-frame`** paints a fresh 16:9 frame from a text `prompt`
  (optional `pov`: `first_person` / `third_person_shoulder`; `mood`; `style`:
  `illustrated` / `cartridge`). It returns the image as `imageBase64` and a
  hosted `url` you can pin as an entrance image, plus the resolved `prompt` and
  `art_style`. Metered `seed_frame`.
* **`POST /v1/probe`** takes a frame (`frame_b64` or `frame_url`, with an
  optional `premise`) and returns a starter **TestWorld bundle**
  (`{ testworld, verified[], probe_report, validation }`): detected objects plus
  a probe report you can author from. Metered `probe`.

<CodeGroup>
  ```bash Seed a frame theme={null}
  curl -X POST 'https://api.alakazam.gg/v1/seed-frame' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d '{
      "prompt": "a flooded boiler room lit by one red emergency lamp",
      "pov": "first_person",
      "style": "illustrated"
    }'
  ```

  ```bash Probe a frame theme={null}
  curl -X POST 'https://api.alakazam.gg/v1/probe' \
    -H 'Authorization: Bearer sk_live_…' \
    -H 'Content-Type: application/json' \
    -d '{
      "frame_url": "https://example.com/boiler-room.jpg",
      "premise": "escape the flooding station"
    }'
  ```
</CodeGroup>

## Status codes

| Status | Meaning                                                                                                             |
| ------ | ------------------------------------------------------------------------------------------------------------------- |
| `200`  | OK. The body carries the perception, hotspots, painted frame, or probe bundle. Charged even if the result is empty. |
| `400`  | A required field is missing: no frame, empty `labels`, or a missing `prompt` on `/seed-frame`.                      |
| `401`  | Missing or invalid key, or a `pk_` key used where `worlds:write` is required.                                       |
| `402`  | Over your daily cap for that meter (`perceive` / `ground` / `seed_frame` / `probe`).                                |
| `413`  | The base64 frame exceeds the per-call size cap.                                                                     |
| `502`  | The vision backend errored, and your reserved quota is refunded.                                                    |
| `503`  | The quota backend is unavailable: fail-closed, nothing is spent.                                                    |