Implementing AI Checks with Continue for Source-Controlled PR Reviews

Continue is a tool that allows developers to enforce AI checks on pull requests, using markdown files stored in the .continue/checks/ directory of the repository. These checks provide feedback as GitHub status checks, turning green when the code is acceptable, and red with a fix suggestion when issues are detected.

Each check is defined by naming and describing potential issues to look for. For example, you can create a check named Metrics Integrity, described as detecting changes that could harm metric accuracy, such as session counts and event tracking errors. The checks are capable of running functions beyond simple diff analysis: they can read/write files, execute bash commands, and interact with a browser.

An example check for metrics integrity may include looking for patterns like narrow 'find or create' queries causing data duplication, or event tracking within loops leading to multiple, incorrect calls. Files with key patterns like posthog.capture or trackEvent could be tagged as critical for this review.

The checks run automatically on every PR, capturing potential data issues that traditional testing might miss, such as subtle changes affecting dashboards without immediately obvious errors.

To set up your own checks, you can leverage your preferred AI coding agent to explore your codebase using the gh CLI, and craft custom checks tailored to your code standards.