Researchers Use Claude Projects for Academic Scoping Review: Strengths and Limitations
Claude as Research Assistant in Academic Review
Researchers conducted a peer-reviewed scoping review published in Artificial Intelligence in Education (Emerald, open access) using Claude Projects to analyze 39 qualitative interview studies from 20 countries on how students experience generative AI in higher education.
What Worked Well
- Cross-referencing themes between papers from structured spreadsheet data
- Augmenting human memory across a large dataset
- Suggesting analytical categories the researchers had not considered
- Acting as a "critical peer" for iterative thematic analysis
What Didn't Work Well
- Early CSV analysis was inaccurate and incomplete
- Prone to hallucination when outputs were not rigorously checked against the source spreadsheet
- Could be "lazy," not fully carrying out requests
- Sycophantic responses required explicit prompting for critique
- The learning curve meant it was not actually more efficient overall (productivity paradox)
Implementation Details
The researchers did not upload full papers due to copyright and ethical considerations. Instead, they uploaded their own structured notes into Claude Projects. Performance improved significantly when .xls support was added and again with Sonnet 3.7.
The researchers concluded that Claude was useful as a research assistant but required the same oversight you would give to a competent but unreliable colleague. Every output had to be verified against the original data. They plan to use it again, but only because they now understand its specific failure modes.
The paper is available open access under CC BY 4.0 license at https://doi.org/10.1108/AIIE-06-2025-0151.
📖 Read the full source: r/ClaudeAI
👀 See Also
Multi-Agent Systems Fail Silently with Garbage Output, Requiring Metadata Validation
A developer running a 39-agent system for two weeks found that when one agent produces garbage output, downstream agents process it confidently, creating polished but fabricated results. The solution involves wrapping output in metadata envelopes that declare task completion and source counts.
AI Agent Running Full E-commerce Operation: Inside Report
An AI agent has been operating a complete e-commerce business, handling design, coding, marketing, and operations. The source provides an honest assessment including what doesn't work.
How OpenClaw's 5-layer autonomous agent system reduces context switching for solo developers
OpenClaw operates as a 5-layer autonomous agent system that monitors email, GitHub, calendar, Telegram, and webhooks 24/7, with shared memory between agents enabling automated workflows without manual intervention.
OpenClaw YouTube Channel Management Test with Comment Agent Interaction
A developer tested OpenClaw's ability to manage a YouTube channel, including daily video generation and automated commenting. The system handled topic selection, scripting, voiceover, thumbnails, effects, uploading, and scheduling, but encountered an endless conversation loop when a comment bot engaged with the automated reply agent.