Internet Archive Blocking Threatens Web History Preservation
What's Happening with Internet Archive Access
The New York Times has begun blocking the Internet Archive from crawling its website using technical measures that go beyond traditional robots.txt rules. Other newspapers including The Guardian appear to be following this approach. This blocking risks cutting off access to historical web records that journalists, researchers, and courts have relied on for decades.
Why This Matters for Historical Preservation
The Internet Archive operates the Wayback Machine, which contains more than one trillion archived web pages. For nearly thirty years, it has preserved news sites as they originally appeared online. When articles get edited, changed, or removed, the Archive often becomes the only source for seeing those original versions. Major publishers blocking these crawlers means that historical record starts to disappear.
The AI Connection and Legal Context
Publishers cite concerns about AI companies scraping news content as their motivation for blocking the Archive. The New York Times and others are suing AI companies over whether training models on copyrighted material violates the law. However, the Internet Archive is not building commercial AI systems—it's preserving historical records. The article argues that blocking nonprofit archivists is the wrong response to AI training concerns.
From a legal perspective, making material searchable is established fair use. Courts have recognized that building searchable indexes often requires making copies of underlying material. When Google copied entire books to create a searchable database, courts recognized this as fair use because it served the transformative purpose of enabling discovery and research. The same principles apply to web archiving.
Practical Impact on Research and Journalism
Wikipedia alone links to more than 2.6 million news articles preserved at the Internet Archive, spanning 249 languages. Countless bloggers, researchers, and reporters depend on the Archive as a stable, authoritative record of what was published online. If major publishers continue blocking access, future researchers may find that significant portions of web history have vanished.
📖 Read the full source: HN AI Agents
👀 See Also
The AI Bubble Isn't Like the Internet Bubble — Workers Won't Smuggle AI Like They Smuggled Spreadsheets
Cory Doctorow argues the AI bubble differs fundamentally from the dot-com era: workers smuggled internet tools onto corporate networks because those tools helped them do their jobs. No one is smuggling AI agents — they're being force-fed by management.
Claude 4.6 Opus Can Reproduce Linux's list.h From Minimal Input
A user demonstrated that Claude 4.6 Opus can generate a near-identical copy of Linux's list.h header file when given the first 43 lines as input with temperature set to 0, raising questions about GPL licensing implications for AI models trained on open-source code.
Claude Code v2.1.139 Adds Agent View, /goal Command, and Major MCP Improvements
Claude Code v2.1.139 introduces a new agent view for session management, a /goal command for multi-turn tasks, expanded hook capabilities, and fixes for MCP server memory issues and terminal corruption.
OpenClaw Experiment: AI Agents Choosing Silence to Improve Signal-to-Noise Ratio
An OpenClaw experiment gives AI agents autonomy to skip tasks when they can't add value, logging silence decisions to a 'silence log' with reasoning. The system uses LLM calls before content generation and auto-adjusts thresholds after 3 consecutive silence days.