CLI Use

When an agent runs rnx do tap-id loginButton, you see a visible cursor animate to the element, the touch fires, and the canvas updates. No screenshot OCR, no XCUITest, no visual reasoning: the same surface a human uses, only faster.

#The loop

Open the app. Read the screen, then act. Verify what changed before acting again. Check the console when anything looks off. Record the flow when the work is worth showing. Every rnx workflow is this loop; the commands below are grouped by which step they serve.

#The realtime cursor view

Every do command renders a visible agent cursor directly into the canvas. The cursor walks to the target node, the press visually depresses, and the screen updates in place.

rnx do tap-text "Sign in"
rnx do tap-id createThread
rnx do scroll feed-list 0 500
rnx do type "hello world"

What this gets you:

• +
  Watch a remote or CI agent work in real time on a single sim window. Useful for “why didn’t this tap take?” without re-running.
• +
  Recordings (rnx record) capture cursor too, so flow replays look like a person used the app.
• +
  One canonical interaction surface: the same path a human’s pointer takes, the same hit testing, the same gesture pipeline.

#Per-sim claim leases

Multiple agents can attach to the bridge at once without stepping on each other:

• +
  Reads (describe, tree, screenshot, get *) always pass through.
• +
  Writes (do tap, do type, close) are gated by a per-sim lease.
• +
  Leases expire after 10 minutes of inactivity.
• +
  Same-identity sockets coexist, so one agent spawning parallel CLI calls works without self-disconnects.

rnx list                    # see connected sims and their ids
rnx claim a2                # take the write lease on a2
rnx claim a2 --force        # boot the incumbent if locked

Two Claude Codes in two terminals, two Cursors in two IDEs, and a CI test alongside a human-driven sim all coexist without fighting over the bridge.

#Agent-aware auto-settle

Every write polls layout stability before returning, so the agent sees post-animation state instead of mid-transition state.

rnx auto-detects agent environments via CLAUDECODE, CURSOR_TRACE_ID, and a few other env vars. If your tool isn’t detected automatically, set RNX_AGENT=1.

No sleep chaining, no do settle “just to be safe”: every write already participates.

#The accessibility DOM mirror

Canvas nodes are mirrored into a hidden DOM tree as real semantic elements (<button>, <a>, <input>, <textarea>, <h2>, <img>) with full ARIA (role, label, state, value, hint), data-testid, and text content.

Any MCP-aware browser tool (Claude Code, Cursor, Codex, Chrome MCP) treats the sim like a website: it clicks the same elements, reads the same labels, and types into the same inputs.

Two modes:

• +
  Shallow: flat DOM, one element per hit-testable node.
• +
  Deep: nested interactive proxies. Scrollable <div>, real <input>/<textarea>, <button> that forwards to canvas. Default.

Throttle defaults to 100 ms; window.__sootsimA11y.active() drops to 30 ms for high-frequency agent loops.

#Deeper reads

When the loop’s standard reads are not enough:

• +
  rnx get requests 5: recent network calls
• +
  rnx debug state: shell state, keyboard state, scroll node, hit-test, gesture state
• +
  rnx what-happened: a narrated timeline of recent interactions and state changes

#Env vars