In a recent analysis conducted by data journalist Ben Welsh, it was found that a significant percentage of news websites are actively blocking AI web crawlers, specifically Applebot-Extended, OpenAI, and Google-Extended. This trend reflects a growing divide among news publishers regarding their approach to these AI agents.
Welsh’s analysis revealed that approximately 25% of the news websites surveyed are blocking Applebot-Extended, while a higher percentage of websites, around 53%, are blocking OpenAI’s bot. Additionally, Google-Extended, introduced last September, is being blocked by nearly 43% of the surveyed sites. This data indicates that while Applebot-Extended may still be flying under the radar, the number of websites blocking it is gradually increasing.
Welsh points out that news publishers are divided on whether to block AI web crawlers, with some websites choosing to do so while others allow them access. The decision-making process behind this trend is complex, with factors such as licensing deals playing a role. For instance, some publishers may be receiving compensation in exchange for granting AI bots access to their sites.
Several major publishers, including The New York Times and Condé Nast, have entered into strategic partnerships with AI companies such as OpenAI and Perplexity. These partnerships involve collaboration on data sharing and web crawling, indicating a deliberate business strategy on the part of publishers to leverage AI technology for their benefit.
One of the challenges faced by news outlets is the constant introduction of new AI agents and the need to update their block lists accordingly. This process can be time-consuming and complex, leading some publishers to explore automated solutions like the one offered by Dark Visitors, which updates robots.txt files automatically.
Notable examples include Condé Nast, which recently unblocked OpenAI’s web crawlers following a partnership announcement. Similarly, Buzzfeed maintains a strict policy of blocking AI web crawlers unless a paid partnership agreement is in place. Media executives are now directly involved in decision-making regarding which bots to block, underscoring the importance of AI technology in the digital publishing landscape.
The trend of news websites blocking AI web crawlers reflects a broader shift in the media industry towards strategic partnerships and data-sharing agreements with AI companies. While the decision to block or allow access to AI bots remains a contentious issue among publishers, it is clear that AI technology is reshaping the digital publishing landscape in significant ways.
Leave a Reply