AI automation

Code generation is only one part of AI-assisted development. To implement a feature reliably, an AI agent also needs to see how the application behaves and check the result as a user would. MōBrowser provides a built-in automation server and CLI for this purpose. They let an agent:

• inspect the app through a compact accessibility snapshot;
• click, hover over, and type into UI elements;
• select options and press keys;
• capture screenshots;
• evaluate JavaScript in the renderer;
• inspect console messages and network requests;
• work with multiple application windows;
• answer native dialogs;
• run complete automated interaction scenarios.

The agent drives your actual MōBrowser application. It doesn’t start a separate browser and doesn’t require Playwright or another browser automation framework.

How MōBrowser guides AI agents 

Projects created with create-mobrowser-app include an AGENTS.md file with MōBrowser-specific instructions. Compatible AI development tools read this file and learn how to:

• use the framework documentation installed in node_modules/@mobrowser/api/docs/ instead of guessing APIs;
• regenerate IPC and Protobuf bindings when necessary;
• choose the correct development build and launch workflow;
• inspect and interact with the app using the MōBrowser automation commands;
• verify changes with snapshots, screenshots, console messages, and network requests;
• keep or stop the app according to the configured automation mode;
• fall back to the Chrome DevTools Protocol only when the built-in commands are insufficient.

This makes the generated project ready for an AI-assisted development flow without requiring you to explain the framework or its automation interface in every prompt.

How automation works 

When automation is enabled, MōBrowser starts a control server inside the application. The server communicates with the renderer through Chromium’s DevTools protocol and exposes higher-level commands designed for AI agents.

The server listens only on the local loopback interface and writes its connection details to .mobrowser/agent.json in the project directory. Subsequent agent commands use this file to find and authenticate with the running app.

The main inspection command is snapshot:

npm run mobrowser -- agent snapshot

It returns a compact accessibility tree instead of the complete DOM. Interactive elements have references such as [ref=e12] that an agent can use in later commands:

window "Sign in"
  textbox "Email" [ref=e4]
  textbox "Password" [ref=e5]
  button "Sign in" [ref=e8]

For example, the agent can fill the form and submit it without calculating screen coordinates:

npm run mobrowser -- agent fill --ref e4 --value "user@example.com"
npm run mobrowser -- agent fill --ref e5 --value "secret"
npm run mobrowser -- agent click --ref e8
npm run mobrowser -- agent wait-for --text "Welcome"

References are the preferred way to target elements because they give the agent a concise semantic representation of the UI. CSS selectors are also supported when needed.

Choose a development workflow 

Launch the app once and reuse the same running instance for all automation commands. Choose the workflow according to the code being changed.

Renderer and UI development 

For renderer work, start the normal development server with automation enabled:

npm run dev -- --automation

The app runs in the foreground and the renderer is served by Vite. Hot Module Replacement applies renderer changes immediately, so an agent can edit the UI and verify each iteration without rebuilding or restarting the application.

This workflow is best for interactive AI-assisted development because you can watch the agent work in the visible application window.

Detached automation 

For main-process, native, or self-contained automated work, build the development app first:

npm run dev:build

Then launch the built app with automation enabled:

npm run mobrowser -- agent launch

The launch command starts the app as a detached process and returns after the automation server and the first window are ready. It doesn’t build the app. Run npm run dev:build again and relaunch after changing the main process, native module, or bundled renderer.

Stop a detached app with:

npm run mobrowser -- agent stop

Configure the automation mode 

MōBrowser supports interactive and autonomous automation. Set the project default in mobrowser.conf.json:

{
  "automation": {
    "mode": "interactive"
  }
}

• interactive is the default. The app window is visible and remains running so you can observe the agent, inspect the result, and request follow-up changes.
• autonomous keeps the app window hidden. It is intended for unattended scenarios in which the agent launches the app, performs the requested work, and stops the app when finished, including after a failed scenario.

You can override the configured visibility for an individual launch:

npm run mobrowser -- agent launch --show-window=false

An invisible app continues rendering normally, so snapshots, screenshots, JavaScript, and UI interactions remain available.

Inspect and interact with the app 

Every command prints a JSON result to standard output and exits with a non-zero status if it fails. Run the following command for the complete, version-specific command reference:

npm run mobrowser -- agent --help

Inspect the interface 

Use an accessibility snapshot for semantic inspection and a screenshot for visual verification:

npm run mobrowser -- agent snapshot
npm run mobrowser -- agent screenshot --out screenshot.png
npm run mobrowser -- agent screenshot --full-page --out full-page.png

Screenshots are useful for checking layout and appearance, while snapshots are usually more efficient for identifying elements and planning actions.

Interact with elements 

Actions can target a reference from the latest snapshot or a CSS selector:

npm run mobrowser -- agent click --ref e12
npm run mobrowser -- agent hover --selector ".account-menu"
npm run mobrowser -- agent fill --ref e5 --value "New value"
npm run mobrowser -- agent type --ref e5 --text "More text"
npm run mobrowser -- agent press-key --ref e5 --key "Control+A"
npm run mobrowser -- agent select-option --ref e9 --value "Canada"

fill sets an input value, while type focuses the element and sends text input. Use press-key for keys and combinations such as Enter, Tab, or Control+A.

Wait for application state 

Wait for a selector, visible text, or a fixed delay before continuing a scenario:

npm run mobrowser -- agent wait-for --selector ".results"
npm run mobrowser -- agent wait-for --text "Saved" --timeout 10000
npm run mobrowser -- agent wait-for --timeout 1000

After an action, take another snapshot or screenshot to verify the resulting state instead of assuming the action succeeded.

Diagnose renderer behavior 

An agent can evaluate JavaScript and inspect information collected from the active renderer:

npm run mobrowser -- agent eval "location.href"
npm run mobrowser -- agent console
npm run mobrowser -- agent network

These commands help the agent connect visible behavior to runtime errors, application state, and failed requests while debugging a feature.

The commands inspect and drive the renderer. Native screenshots and intercepted dialogs are also available, but the automation interface doesn’t inspect or control the main process directly.

Work with multiple windows 

List the application’s windows and select the one that subsequent commands should target:

npm run mobrowser -- agent list-windows
npm run mobrowser -- agent select-window --index 1
npm run mobrowser -- agent select-window --title "Settings"
npm run mobrowser -- agent select-window --url "/settings"

Automate native dialogs 

Native dialogs are application-modal and would otherwise block an unattended automation scenario. MōBrowser intercepts them while automation is enabled.

Queue an answer before performing the action that opens the dialog:

# Choose a message-dialog button.
npm run mobrowser -- agent answer-dialog --button "OK"

# Supply a native message-dialog text-field value.
npm run mobrowser -- agent answer-dialog --text "Automated value"

# Return a file from an open or save dialog.
npm run mobrowser -- agent answer-dialog --path "/tmp/example.txt"

# Cancel the next open or save dialog.
npm run mobrowser -- agent answer-dialog --cancel

Queued answers are consumed in order. After the action, inspect the dialogs that appeared and how they were answered:

npm run mobrowser -- agent dialogs

If no answer was queued, MōBrowser uses a safe default so the scenario doesn’t remain blocked. JavaScript dialogs such as alert, confirm, and beforeunload are also resolved automatically when the app window is hidden.

Example automated scenario 

An AI agent testing a sign-in flow can perform the following loop:

1. Build and launch the app using the workflow appropriate for the changed code.
2. Run agent snapshot and find the references for the email field, password field, and submit button.
3. Fill the fields and click the button.
4. Wait for the expected page text or element.
5. Take another snapshot and screenshot to verify the result.
6. Inspect agent console and agent network if the expected result doesn’t appear.
7. Modify the source, rebuild or rely on hot reload as appropriate, and repeat.
8. Leave the app running in interactive mode, or stop it in autonomous mode.

This loop allows an agent to validate a requested feature against the produced application instead of stopping after the source code compiles.

Use Chrome DevTools Protocol as a fallback 

The built-in agent commands cover common inspection, interaction, and testing tasks and should be the first choice. For advanced operations they don’t expose, such as tracing, coverage, emulation, or precise input timing, use a CDP-compatible tool with the application’s own Chrome DevTools Protocol endpoint.

When using npm run dev or npm run mobrowser -- run, the endpoint is available at http://localhost:9222 by default. You can select another port with --remote-debugging-port:

npm run dev -- --automation --remote-debugging-port=9333

To enable the endpoint for a detached agent launch, pass the option to the application:

npm run mobrowser -- agent launch -- --remote-debugging-port=9222

Connect to the running MōBrowser app rather than launching a separate browser.

Development-only security 

The automation server is disabled by default and is enabled only for development launches that explicitly request automation. Production builds and packaged apps don’t start it.

While active, the server is restricted to the local machine and requires a random token stored in the endpoint file. Protect access to the project and this file: automation commands such as eval can execute arbitrary JavaScript in the renderer and should be treated as development tooling with access to the running app.