Pilot: A Browser Automation Tool Built Entirely with Claude Code

Pilot is a browser automation tool built entirely by Claude Code, directed and tested by a non-developer. It enables Claude to control Chrome by reading the accessibility tree—the same structure screen readers use—and assigning numbers to clickable elements for precise navigation.

How It Works

The tool converts page elements into compact text data instead of screenshots. Each clickable element gets a number, allowing Claude to issue commands like click 5 rather than guessing screen positions. This approach handles popups and works across multiple tabs.

Setup and Usage

• Install the Chrome extension and start the server
• Type /pilot in Claude Code
• Ask Claude to perform tasks like "go to YouTube and search for cooking tutorials"

Multiple actions can be batched in a single call, and the tool processes page data as compact text rather than screenshots.

Development Insights

The creator learned several key lessons while building with AI:

• Clearly describing what you want is the hardest part
• Testing remains the human's responsibility—Claude writes code but you must verify it works
• Development required many iterations, not a one-shot process

The tool is free, MIT licensed, and works on macOS, Linux, and Windows. The GitHub repository is available for those interested in the implementation or wanting to contribute.