Field Report: AI Research Partner Fails Peer Review, Prompting Methodology Codification

A geologist/geophysicist who uses Claude Opus for complex, multi-file, multi-week projects reported a failure in AI-assisted research analysis. The user asked Claude to critically evaluate an offshore wind industry-funded study reporting high bird avoidance rates at wind turbines. Claude produced a confident six-point analysis with real citations and fluent delivery.

When the user verified the sources, four points fell apart. The citations were real but couldn't carry the weight assigned to them - contextual literature was dressed up as direct rebuttal. The study still had limitations: small sample, onshore-only results, no peer review. The avoidance rates were likely real for the conditions tested, but the question remained whether they hold for nocturnal migrants at lit offshore turbines.

The user had to rebuild the evidence from scratch to produce an evaluation that actually holds up. They then codified the methodology so future evaluations start on solid ground from the first draft. The user is still actively using Claude for research analysis, noting these systems make it sustainable.

The user provided two resources: a blog post detailing the experience and a GitHub repository containing the codified methodology. The GitHub repository includes a system prompt for research projects that establishes operational discipline for AI-assisted analysis.