Researchers Use Claude Projects for Academic Scoping Review: Strengths and Limitations

Claude as Research Assistant in Academic Review

Researchers conducted a peer-reviewed scoping review published in Artificial Intelligence in Education (Emerald, open access) using Claude Projects to analyze 39 qualitative interview studies from 20 countries on how students experience generative AI in higher education.

What Worked Well

• Cross-referencing themes between papers from structured spreadsheet data
• Augmenting human memory across a large dataset
• Suggesting analytical categories the researchers had not considered
• Acting as a "critical peer" for iterative thematic analysis

What Didn't Work Well

• Early CSV analysis was inaccurate and incomplete
• Prone to hallucination when outputs were not rigorously checked against the source spreadsheet
• Could be "lazy," not fully carrying out requests
• Sycophantic responses required explicit prompting for critique
• The learning curve meant it was not actually more efficient overall (productivity paradox)

Implementation Details

The researchers did not upload full papers due to copyright and ethical considerations. Instead, they uploaded their own structured notes into Claude Projects. Performance improved significantly when .xls support was added and again with Sonnet 3.7.

The researchers concluded that Claude was useful as a research assistant but required the same oversight you would give to a competent but unreliable colleague. Every output had to be verified against the original data. They plan to use it again, but only because they now understand its specific failure modes.

The paper is available open access under CC BY 4.0 license at https://doi.org/10.1108/AIIE-06-2025-0151.