Vision and interactive objects

The main interactive feature of a world is clicking the objects you see. The Vision API is how you build it: read a live frame in plain text, detect bounding boxes for named objects, then anchor a graph event to one of those objects. Once an event has an anchor, the shipped player runs a live detect loop and renders that choice as a clickable chip pinned to the object in the streaming video, so the player clicks the red door to take the “open the red door” branch instead of picking it from a list. All five routes are thin, metered wrappers over the same vision backend that powers the editor. The pipeline is always the same shape:

/v1/perceive: a text read of a frame (what objects are here?). No boxes.
/v1/ground/objects / /v1/ground/scene: bounding boxes for named objects (where exactly is each one?).
an anchor on an event: pins a choice to a detected object so the player renders it as a clickable hotspot. See Editing the graph for event edits, and Embedding for where that clickable player runs.
/v1/seed-frame / /v1/probe: paint a fresh frame, or probe one into a starter world.

Every vision route is a write: it needs a secret key (sk_) with worlds:write, because each call makes real, paid vendor spend. A publishable key (pk_) only carries worlds:read and is rejected here. Never ship a secret key to a browser. Run these from your server.

Metered and fail-closed

Each vision call reserves quota before it hits the backend, and refunds on any failure. The failure modes are strict and uniform:

402: you’re over your daily cap for that meter (perceive, ground, seed_frame, or probe). Reserve-before-spend, so you’re never billed past the cap.
503: the quota backend is unavailable. The call fails closed rather than letting unmetered spend through.
413: the base64 frame exceeds the per-call size cap.
400: a required field is missing (no frame, empty labels, …).
502: the vision backend itself errored, and your quota is refunded.

A 200 with an empty result (no hotspots, or an empty perception) is still charged. The paid vendor call was made regardless of whether it found anything. Refunding empties would open free-abuse, so you pay for the work performed.

Grounding meters per label: one ground unit per label you ask for, capped at 6 (the backend’s hard cap on returned boxes). Ask for 6 labels and you spend 6 ground units. Anything past 6 is dropped before it’s billed.

Read a frame with /perceive

POST /v1/perceive is a text-only read of a frame (no bounding boxes). Use it first, to discover what’s on screen before you decide which objects to make clickable. It returns a one-line summary, the visible objects, the affordances (things the player could act on), and the exits. Pass the frame as a base64-encoded JPEG in frame_b64. room_name and room_description are optional hints that sharpen the read.

# base64-encode a JPEG frame, then POST it
FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

curl -X POST 'https://api.alakazam.gg/v1/perceive' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d "{
    \"frame_b64\": \"$FRAME_B64\",
    \"room_name\": \"the boiler room\"
  }"

import { readFileSync } from "node:fs";

const frame_b64 = readFileSync("frame.jpg").toString("base64");

const perception = await fetch("https://api.alakazam.gg/v1/perceive", {
  method: "POST",
  headers: {
    Authorization: "Bearer sk_live_…",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ frame_b64, room_name: "the boiler room" }),
}).then((r) => r.json());

console.log(perception.summary);
console.log("visible:", perception.visible); // → ["a red valve", "a rusted door", …]

import base64, requests

frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

perception = requests.post(
    "https://api.alakazam.gg/v1/perceive",
    headers={
        "Authorization": "Bearer sk_live_…",
        "Content-Type": "application/json",
    },
    json={"frame_b64": frame_b64, "room_name": "the boiler room"},
).json()

print(perception["summary"])
print("visible:", perception["visible"])

The response is { summary, visible[], affordances[], exits[] }, all strings and no coordinates. Take the labels you want to make clickable and feed them to a grounding route next.

Detect objects with /ground/objects

POST /v1/ground/objects is the open-vocabulary detector: pass labels[] and get back a hotspot box for each one it finds. This is what turns a name like "the red valve" into geometry you can pin a choice to.

FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

curl -X POST 'https://api.alakazam.gg/v1/ground/objects' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d "{
    \"frame_b64\": \"$FRAME_B64\",
    \"labels\": [\"the red valve\", \"the rusted door\"]
  }"

import { readFileSync } from "node:fs";

const frame_b64 = readFileSync("frame.jpg").toString("base64");

const { hotspots } = await fetch(
  "https://api.alakazam.gg/v1/ground/objects",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer sk_live_…",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      frame_b64,
      labels: ["the red valve", "the rusted door"],
    }),
  },
).then((r) => r.json());

for (const h of hotspots) {
  const [x, y, w, h_] = h.bbox; // 0-1000 normalized
  console.log(h.label, h.confidence, [x, y, w, h_]);
}

import base64, requests

frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

hotspots = requests.post(
    "https://api.alakazam.gg/v1/ground/objects",
    headers={
        "Authorization": "Bearer sk_live_…",
        "Content-Type": "application/json",
    },
    json={"frame_b64": frame_b64, "labels": ["the red valve", "the rusted door"]},
).json()["hotspots"]

for h in hotspots:
    print(h["label"], h["confidence"], h["bbox"])  # bbox = [x, y, w, h]

Each hotspot is { label, bbox, confidence }. bbox is [x, y, w, h] in 0-1000 normalized coordinates: the box is w/1000 of the frame wide and h/1000 tall, with its top-left corner at (x/1000, y/1000). Because the coordinates are normalized, they map onto the streaming video at any resolution.

Grounding is metered per label (cap 6). labels: ["a", "b", "c"] spends three ground units. A 7th label is dropped before it’s billed. frame_jpeg_base64 is accepted as an alias for frame_b64.

Pin choices with /ground/scene

POST /v1/ground/scene is the closed-vocabulary variant: instead of loose labels, you pass the graph choices you already have as expected_choices: [{ id, label }], and each returned hotspot carries a choice_id linking the box back to the choice it matched. Use this when you’re grounding a specific set of branches (rather than free-form object discovery). The choice_id tells you exactly which event to anchor.

FRAME_B64=$(base64 -i frame.jpg | tr -d '\n')

curl -X POST 'https://api.alakazam.gg/v1/ground/scene' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d "{
    \"frame_b64\": \"$FRAME_B64\",
    \"location_name\": \"the boiler room\",
    \"expected_choices\": [
      { \"id\": \"turn_valve\",  \"label\": \"the red valve\" },
      { \"id\": \"open_door\",   \"label\": \"the rusted door\" }
    ]
  }"

import { readFileSync } from "node:fs";

const frame_b64 = readFileSync("frame.jpg").toString("base64");

const { hotspots } = await fetch(
  "https://api.alakazam.gg/v1/ground/scene",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer sk_live_…",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      frame_b64,
      location_name: "the boiler room",
      expected_choices: [
        { id: "turn_valve", label: "the red valve" },
        { id: "open_door", label: "the rusted door" },
      ],
    }),
  },
).then((r) => r.json());

// hotspot.choice_id links each box back to the expected_choice it matched
for (const h of hotspots) console.log(h.choice_id, "→", h.bbox);

import base64, requests

frame_b64 = base64.b64encode(open("frame.jpg", "rb").read()).decode()

hotspots = requests.post(
    "https://api.alakazam.gg/v1/ground/scene",
    headers={
        "Authorization": "Bearer sk_live_…",
        "Content-Type": "application/json",
    },
    json={
        "frame_b64": frame_b64,
        "location_name": "the boiler room",
        "expected_choices": [
            {"id": "turn_valve", "label": "the red valve"},
            {"id": "open_door", "label": "the rusted door"},
        ],
    },
).json()["hotspots"]

for h in hotspots:
    print(h["choice_id"], "→", h["bbox"])

Both grounding routes share the same { hotspots: [...] } response shape. Only choice_id differs: it’s populated on /ground/scene (matched to an expected_choice) and null when a label wasn’t an expected choice.

The interactive payoff: anchors

Grounding on its own is only detection. To make an object clickable in the player, attach an anchor to the event whose branch that object should trigger:

{ "label": "the red door", "aliases": ["red door", "crimson door"] }

When an event carries anchor.label, the shipped player (StateMachinePlayer and the embed) runs a live VLM detect loop on the streaming frame and renders that choice as a clickable chip pinned over the detected object, the same [x, y, w, h] 0-1000 box geometry the grounding routes return. The player clicks the object to take the branch. aliases give the detector alternate names to match on when the object’s on-screen appearance drifts. You set the anchor with an ordinary graph edit (events are addressed by their unique name): either the dedicated event patch or a batch update_event op. Both funnel through the same fail-closed validation gate.

curl -X PATCH \
  'https://api.alakazam.gg/v1/worlds/WORLD_ID/events/Open%20the%20red%20door' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d '{
    "patch": {
      "anchor": { "label": "the red door", "aliases": ["red door", "crimson door"] }
    }
  }'

curl -X POST 'https://api.alakazam.gg/v1/worlds/WORLD_ID/ops' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d '{
    "ops": [
      {
        "op": "update_event",
        "name": "Open the red door",
        "patch": { "anchor": { "label": "the red door" } }
      }
    ]
  }'

await fetch(
  "https://api.alakazam.gg/v1/worlds/WORLD_ID/events/" +
    encodeURIComponent("Open the red door"),
  {
    method: "PATCH",
    headers: {
      Authorization: "Bearer sk_live_…",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      patch: {
        anchor: { label: "the red door", aliases: ["red door", "crimson door"] },
      },
    }),
  },
);

End-to-end: a frame becomes a clickable object

Perceive the frame

POST /v1/perceive with the frame’s frame_b64. Read visible and affordances to find the objects worth making interactive, such as "the red valve" and "the rusted door".

Ground the labels you picked

POST /v1/ground/objects with those labels (or /ground/scene if you’re matching them to existing choices). Confirm each object comes back with a plausible bbox and a decent confidence before you wire it up. A label the detector can’t find won’t render a chip.

Anchor the events

For each object, PATCH /v1/worlds/{id}/events/{name} (or a batch update_event op) to set anchor.label on the event that object should trigger. Add aliases for names the detector might see instead. The write is validated fail-closed like any graph edit.

Play it

Embed the world (see Embedding). The player detects the anchored objects live and renders each anchored choice as a clickable chip pinned to its object. The player clicks the object to branch.

Paint or probe a frame

Two more routes round out the pipeline, for when you don’t already have a frame:

POST /v1/seed-frame paints a fresh 16:9 frame from a text prompt (optional pov: first_person / third_person_shoulder; mood; style: illustrated / cartridge). It returns the image as imageBase64 and a hosted url you can pin as an entrance image, plus the resolved prompt and art_style. Metered seed_frame.
POST /v1/probe takes a frame (frame_b64 or frame_url, with an optional premise) and returns a starter TestWorld bundle ({ testworld, verified[], probe_report, validation }): detected objects plus a probe report you can author from. Metered probe.

curl -X POST 'https://api.alakazam.gg/v1/seed-frame' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "a flooded boiler room lit by one red emergency lamp",
    "pov": "first_person",
    "style": "illustrated"
  }'

curl -X POST 'https://api.alakazam.gg/v1/probe' \
  -H 'Authorization: Bearer sk_live_…' \
  -H 'Content-Type: application/json' \
  -d '{
    "frame_url": "https://example.com/boiler-room.jpg",
    "premise": "escape the flooding station"
  }'

Status codes

Status	Meaning
`200`	OK. The body carries the perception, hotspots, painted frame, or probe bundle. Charged even if the result is empty.
`400`	A required field is missing: no frame, empty `labels`, or a missing `prompt` on `/seed-frame`.
`401`	Missing or invalid key, or a `pk_` key used where `worlds:write` is required.
`402`	Over your daily cap for that meter (`perceive` / `ground` / `seed_frame` / `probe`).
`413`	The base64 frame exceeds the per-call size cap.
`502`	The vision backend errored, and your reserved quota is refunded.
`503`	The quota backend is unavailable: fail-closed, nothing is spent.

Get started

Agents & SDKs

Engine SDKs

Guides

Resources

Vision and interactive objects

Metered and fail-closed

Read a frame with /perceive

Detect objects with /ground/objects

Pin choices with /ground/scene

The interactive payoff: anchors

End-to-end: a frame becomes a clickable object

Paint or probe a frame

Status codes

​Metered and fail-closed

​Read a frame with /perceive

​Detect objects with /ground/objects

​Pin choices with /ground/scene

​The interactive payoff: anchors

​End-to-end: a frame becomes a clickable object

​Paint or probe a frame

​Status codes

Metered and fail-closed

Read a frame with /perceive

Detect objects with /ground/objects

Pin choices with /ground/scene

The interactive payoff: anchors

End-to-end: a frame becomes a clickable object

Paint or probe a frame

Status codes