MCP Usage (optics mcp)¶

optics mcp runs a Model Context Protocol server that exposes the optics-framework keyword engine to an LLM client (Claude Desktop, Claude Code, Cursor, …). The model starts a device/browser session, runs automation keywords as tools, and observes device state through resources — driving a live target the way optics live does, but under agent control.

The surface is deliberately small: the keyword tools plus a few resources. There are no record/save/replay tools — a capable agent already knows the steps it ran, so to turn a session into a reusable suite it authors an optics CSV project itself (the optics://project-format resource documents the exact layout) and runs it with optics execute. The MCP layer stays driver- and device-agnostic: it never shells out to adb or assumes a device type; it only makes calls to the driver server you point start_session at.

It reuses the in-process keyword machinery from the REST server (optics serve), so whatever driver and element sources optics already supports work here too. The driver is chosen at runtime via the start_session tool — nothing is hard-coded.

1. Prerequisites¶

• Python 3.12+ and optics-framework installed.
• A driver target that optics can reach — e.g. a local Appium server with a connected device/emulator, or a remote Appium hub. You provide its URL and capabilities to start_session.
• An MCP-capable client (Claude Desktop/Code, Cursor, or the fastmcp Python client for scripting).
• (Optional) extras for richer element location: text detection (googlevision) and image detection (templatematch) require their own credentials/config, exactly as in a normal optics config.yaml.

2. Install¶

The MCP server depends on fastmcp, shipped as an optional extra:

pip install 'optics-framework[mcp]'
# from source:
poetry install --extras mcp

If the extra is missing, optics mcp exits with a clear message telling you to install it — the rest of the CLI is unaffected.

3. Run the server¶

# stdio transport (default) — for local clients that spawn the process
optics mcp

# HTTP transport — for networked / multi-client use
optics mcp --transport http --host 127.0.0.1 --port 8090

Docker¶

Containerized MCP runs HTTP transport bound to 0.0.0.0:8090 (stdio is for local clients that spawn the process). Images live under Docker/mcp/ and install the [mcp] extra (fastmcp).

Docker Compose (from the repo root):

# Production image (PyPI) — host port 8090
docker compose -f Docker/docker-compose.yml up --build mcp

# Development image (local .whl) — host port 8091
docker compose -f Docker/docker-compose.yml up --build mcp-dev

Standalone build/run:

docker build -f Docker/mcp/prod/Dockerfile -t optics-mcp-prod .
docker run -d -p 8090:8090 --name optics-mcp-prod optics-mcp-prod

Connect your MCP client to the container:

{
  "mcpServers": {
    "optics": { "url": "http://127.0.0.1:8090/mcp" }
  }
}

Use port 8091 for the mcp-dev compose service. When start_session targets Appium on the host, set "url": "http://host.docker.internal:4723". See Docker/deployment.md for vision-backend build args, Google Vision credential mounts, and dev-wheel builds.

4. Connect an MCP client¶

stdio — the client launches the server itself. Add to your client's MCP config (e.g. Claude Desktop claude_desktop_config.json, or Claude Code .mcp.json):

{
  "mcpServers": {
    "optics": {
      "command": "optics",
      "args": ["mcp"]
    }
  }
}

If optics isn't on the client's PATH, use the absolute path (e.g. /path/to/.venv/bin/optics) or "command": "python", "args": ["-m", "optics_framework.helper.cli", "mcp"].

HTTP — start the server yourself (optics mcp --transport http) and point the client at the URL:

{
  "mcpServers": {
    "optics": { "url": "http://127.0.0.1:8090/mcp" }
  }
}

5. The expected journey¶

1. start_session — open a session against your driver. Returns { "session_id", "driver_id" }. The target app is launched automatically. Capture the session_id; every other tool and resource needs it. elements_sources defaults per driver (appium → appium_find_element/appium_page_source/appium_screenshot; selenium and playwright analogous), so start_session with just a driver works. The device is identified through capabilities (e.g. deviceName/udid) — the MCP needs no adb of its own.
2. Observe — call screenshot, read optics://session/{session_id}/source, or read optics://session/{session_id}/elements for the compact list of actionable elements (labels, act, bounds) plus read-only text. These work the same across drivers via the session's own element sources.
3. Act — call keyword tools (press_element, enter_text, swipe, assert_presence, …) with the session_id. Target elements by locator (xpath=/text=/an id/an image) with press_element; when there is no stable locator, detect_and_press taps by the visible text/label, and otherwise use the exact bounds from get_interactive_elements. Do not tap raw pixel coordinates guessed off a screenshot; those misfire.
4. Build a reusable suite — see below.
5. terminate_session — release the driver when done.

Build a reusable suite (author it — there's no save tool)¶

The MCP has no recording/save/replay tools on purpose: you already know the keywords you called and their args, so you can author an optics project directly.

1. Read the optics://project-format resource — it documents the CSV layout (test_cases/, modules/, test_data/elements.csv, config.yaml), how modules reference ${variables}, and the escaping rules.
2. Write those files (the keyword display names come from the optics://keywords catalog). Parameterize by putting run-specific values in elements.csv and referencing them as ${name}, so "set alarm to 22:00" becomes a reusable ${time}. A whole cell must be exactly ${name} — the runner substitutes whole values only, not ${name} inside a larger string.
3. For loops, branching, computed values or API calls in a suite, author the control-flow keywords (Run Loop, Condition, Evaluate, Invoke Api, …) into the modules — they run under optics execute. Live, you drive that logic yourself; they are in the optics://keywords catalog but are not one-shot MCP tools because they need the runner's module/data context.
4. Run it with the CLI: optics execute <folder> (results land in execution_output/). To replay interactively instead, just call the keyword tools again in sequence — the agent is the runner.

start_session arguments¶

Example — local Appium + Android emulator:

{
  "driver": "appium",
  "url": "http://127.0.0.1:4723",
  "capabilities": {
    "platformName": "Android",
    "appium:automationName": "UiAutomator2",
    "appium:deviceName": "emulator-5554",
    "appium:appPackage": "com.android.settings",
    "appium:appActivity": ".Settings"
  },
  "elements_sources": ["appium_find_element", "appium_page_source", "appium_screenshot"]
}

Omit appPackage/appActivity to attach to whatever is already on screen. For a remote/managed hub, set url to the hub and include any hub-specific capabilities (auth token, device id) just as you would in config.yaml.

AI self-heal¶

Self-heal has two layers. Whoever starts optics mcp (or optics serve) can set a server-level default once via environment variables, so a client that's already integrated inherits it with no start_session changes:

Because a single serve/mcp process is multi-tenant, each of these is also a per-session override on start_session: ai_self_heal, llm_provider, and llm_model. A per-session value always wins over the matching env-var default, so a caller can opt in/out and pick its own model without being locked to the operator's choice; anything left unset falls back to the env default (and then to gemini). LLM credentials (e.g. GOOGLE_API_KEY/GEMINI_API_KEY for Gemini) always come from the provider's own environment variables — never through start_session — so choosing a provider/model per session exposes no secrets.

When self-heal recovers a keyword, the call still succeeds, and the result carries the recovery so a client can learn how it was fixed:

{
  "result": null,
  "healed": true,
  "heal_summary": "AI self-heal recovered 'press_element' after 2 steps: scroll down; press_element Login",
  "suggested_steps": [
    { "keyword": "scroll", "params": ["down"] },
    { "keyword": "press_element", "params": ["Login"] }
  ]
}

suggested_steps is the clean, replayable recovery sequence — only the steps that actually worked, curated to the minimal set that reproduces the goal. A platform can persist these to replace the failing step, so the next run passes without needing self-heal. optics serve's POST /v1/sessions/{session_id}/action returns the same shape under data. Un-healed calls return the bare result, unchanged.

Keyword parameters are strings¶

Every keyword tool takes session_id plus that keyword's parameters, and all parameters are typed as strings — pass "2", not 2. Element arguments accept the same locators optics uses elsewhere: xpath=…, text=…, css=…, an id, or an image template name.

6. Tools reference¶

start_session, terminate_session, and screenshot are purpose-built; every other tool is an optics keyword auto-exposed from ActionKeyword / AppManagement / Verifier. Representative set:

• Session/app: start_session, terminate_session, launch_app, launch_other_app, close_and_terminate_app, get_app_version, get_driver_session_id.
• Interact: press_element, press_keycode, enter_text, enter_number, clear_element_text, select_dropdown_option, detect_and_press. Raw coordinate taps (press_by_coordinates, press_by_percentage) are not exposed as tools because a pixel eyed off a screenshot misfires; set OPTICS_MCP_ALLOW_COORDINATE_TAPS=1 before starting the server to re-enable them for targets with no queryable elements (e.g. a canvas).
• Gestures/scroll: swipe, swipe_by_percentage, swipe_from_element, swipe_until_element_appears, scroll, scroll_from_element, scroll_until_element_appears.
• Observe/verify: screenshot (rendered image), get_text, get_interactive_elements (accepts filter_config, e.g. "buttons", and compact for the token-lean list), is_element, assert_presence, assert_equality, validate_element, validate_screen.
• Misc: sleep, execute_script.

The full machine-readable catalog (every keyword, its params and docs) is the optics://keywords resource.

screenshot returns a rendered image/png your client can display inline — prefer it over the screenshot resource when you want to see the screen.

Besides the reflected keywords, only three tools are purpose-built: start_session, terminate_session, and screenshot. There are intentionally no recording, suite-CRUD, replay, device-discovery, or doctor tools — an agent composes those from the keyword tools plus the optics://project-format knowledge (see §5). start_session supplies the one non-obvious convenience: a per-driver elements_sources default so the call just works.

7. Resources reference¶

get_interactive_elements is available both as a resource (compact by default) and as a tool (so the model can pass filter_config, or compact=false for the full untrimmed element list).

The screenshot resource delivers raw PNG bytes with a generic application/octet-stream mime (a limitation of templated MCP resources). For an image your client renders as a picture, use the screenshot tool.

8. Element sources decide what works¶

The keywords you can use depend on which elements_sources (and detection sources) you enable in start_session — same rules as a normal optics project. When you omit elements_sources, start_session enables the driver's canonical trio automatically (appium → appium_find_element/appium_page_source/ appium_screenshot); pass the argument only to narrow or extend that. An explicit empty list is passed through and disables element sources entirely (the session then fails to start) — omit the argument for the defaults:

If you enable only appium_find_element + appium_screenshot and then call get_interactive_elements, optics raises E0202: No interactive elements retrieved using available strategies — that's expected; enable appium_page_source (or a vision source) for that path.

9. Troubleshooting¶

• First start_session is slow against a remote hub (~30–60 s to allocate and launch). Give your client a generous timeout (the fastmcp Python client takes Client(url, timeout=180)).
• No module named 'fastmcp' / "mcp extra required" — install optics-framework[mcp].
• Sessions aren't shared with optics serve / optics live. Each is a separate process with its own in-memory session store. Always start_session in this server before using a keyword tool; you cannot attach to a session created elsewhere.
• Device busy / already allocated — if your hub reports the device as busy, free it through your device-orchestration API, then retry start_session.
• get_interactive_elements / source errors — usually a missing element source; see §8.
• Errors surface as MCP tool errors. An optics failure (element not found, bad config, driver error) comes back as a ToolError carrying the optics error code/message, so the model can read and react to it.
• start_session failed with "Element source configuration must be set". You passed an explicit empty elements_sources, or the driver name doesn't match any installed element-source modules (e.g. a typo — check the name against §8). Omit elements_sources to get the driver defaults, or name the sources per §8.
• Where do suites live? Wherever you write them — the MCP has no workspace of its own. Author the CSV project (see optics://project-format) at a path you choose and run it with optics execute.
• Setup errors (device unreachable, driver not running). The MCP only talks to the driver server at the url you pass; make sure that server is up and can see the device (for Appium+Android that means its host has the Android SDK — the MCP itself needs none). optics doctor on that host diagnoses the toolchain.

10. How it works (pointer)¶

optics mcp is a thin in-process wrapper over common/expose_api.py. It reflects the API keyword classes into typed tools and routes execution through the same execute_keyword path the REST server uses; read-only observers become resources. The only hand-written additions are the start_session/terminate_session/ screenshot lifecycle tools, the per-driver elements_sources default, and the optics://project-format knowledge resource — no recording/suite/device tooling. See optics_framework/helper/mcp_server.py and the "MCP server journey" section of CLAUDE.md for the internals.