Aaron Swartz prosecution contrasted with Meta's scraping impunity
A blog post argues that Aaron Swartz was harshly prosecuted for scraping academic articles while Meta scrapes data at scale with little legal consequence.
Legal & Policy
Amazonbot ignores robots.txt and falls for honeypot trap, user reports
A Hacker News user reports that Amazonbot aggressively scraped their personal website despite robots.txt directives and was caught accessing a hidden honeypot with fake HTTP endpoints.
Anthropic confirms watermarking concerns in statement
Anthropic issued a statement acknowledging all concerns about watermarking.
Cloudflare introduces Accountable AI crawler designation to let sites block training without losing search visibility
Cloudflare launches new controls and an Accountable designation, backed by Apple, Google, and Microsoft, that allow site owners to disallow AI training while remaining discoverable in search.
Cloudflare launches Pay Per Use beta to bill AI companies for publisher content
Cloudflare has introduced a beta service that tracks when AI companies use publishers' content and handles billing, payouts, and reporting.
Eric Goldman hosts Reddit AMA on web scraping law
Legal scholar Eric Goldman conducted a Reddit AMA covering the legal landscape of web scraping.
Hacker News post questions whether OpenAI used math professors' work without permission
A Hacker News post titled 'Did OpenAI steal math professors work?' raises allegations of unauthorized use of academic material.
Leading AI scientists warn of impending intelligence explosion and loss of human control
OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, and other AI researchers publish a paper warning that AI is approaching the point where it can improve itself without human involvement.
Legal experts warn Meta's new terms still grant broad scraping permissions despite language changes
Legal experts caution that Meta's updated terms of service continue to permit extensive web scraping, despite cosmetic language revisions.
LinkedIn wins court order to block mass scraping of user data
A court has granted LinkedIn an order preventing mass scraping of its user data.
Meta's updated commercial terms may permit scraping of news sites, legal experts warn
Legal experts say Meta's revised commercial terms could allow the company to scrape news sites for data.
Microsoft and OpenAI lose bid to keep internal documents calling web scraping 'theft' secret
A court has ruled that Microsoft and OpenAI must disclose internal communications in which executives reportedly described AI training data scraping as 'the largest theft of labor in human history.'
Microsoft director calls AI scraping 'the largest theft of labor in human history' in NYT lawsuit filings
Legal briefs in the New York Times lawsuit against OpenAI reveal a Microsoft director describing AI web scraping as the largest theft of labor in human history.
Microsoft exec called AI scraping 'the largest theft of labor in human history,' unredacted filings reveal
Newly unredacted court filings show a Microsoft executive described AI-powered web scraping as the largest theft of labor in human history.
OpenAI's M. Williams suggests human extinction likely in coming years
A post on Hacker News reports that M. Williams from OpenAI stated that human extinction in the next few years seems likely.
ScrapingBee publishes guide on real-time PII anonymization for GDPR-compliant web scraping
ScrapingBee released a blog post detailing how to use middleware for real-time anonymization of personally identifiable information during web scraping to meet GDPR requirements.
SerpApi countersues Reddit over API access restrictions
SerpApi has filed counterclaims against Reddit in response to Reddit's lawsuit, alleging broken promises of an open internet and unfair API pricing.
SerpApi moves to dismiss Google's amended complaint in scraping lawsuit
SerpApi has filed a motion to dismiss Google's amended lawsuit after the court previously dismissed the original complaint in July.
Web scraper declares victory after court win: 'Google and Reddit do not own the Internet'
A web scraper has won a court case against Google and Reddit, asserting that the platforms do not have exclusive ownership over publicly accessible web data.
Zyte analysis finds 75% of top sites use robots.txt, but few name specific crawlers
Zyte published a study showing that three-quarters of the world's top websites publish a robots.txt file, yet most do not name individual crawlers.
Zyte argues EU AI scraping guidelines rely on outdated robots.txt standard
Zyte publishes a blog post arguing that Europe's new generative AI scraping guidelines, which lean on the robots.txt protocol, will harm users and entrench monopolies.
Zyte blog post examines how AI and web scraping turn scattered personal data into security risks
Domagoj Marić explores the intersection of AI, web scraping, and OSINT to show how fragmented personal data is assembled into profiles, scams, and security threats at Extract Summit.
Zyte research argues web scraping faces pricing barriers, not outright blocking
Zyte's State of Web Access report, discussed in an interview with the researcher, finds that new economic barriers are making web scraping more difficult rather than technical blocks.
Zyte study finds 40% of top sites block AI crawlers via robots.txt
A Zyte analysis of the world's leading websites reveals that nearly four in ten now use robots.txt to restrict AI bots.