- Thread Author
- #1
Almost all major publishers in the US and UK have blocked their news from being scraped by AI. OpenAI Chairman Bret Taylor said that AI is a "bubble" in terms of investment.
Almost eight out of ten of the largest news sites in the UK and US are now blocking AI training bots. Details of the research were published in Pressgazette.
Publishers in London were told this week that it is "never too late" to start blocking bots, as new scraping processes are frequently needed to keep AI-powered chatbot responses up-to-date.
79% of approximately 100 leading news sites in the UK and US are blocking at least one of the AI training crawlers such as OpenAI's GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended, and Google-Extended.
Meanwhile, 71% of users do not allow AI bots to collect data or perform live searches by crawling their sites. These bots include: ChatGPT-User, Claude-Web, Perplexity-User, and OAI-SearchBot.
Digital public relations platform Buzzstream's recent analysis, which examined a combined and de-duplicated list of the top 50 news sites in the UK and US, was shared with Press Gazette.
Among the top 50 publishers included in the analysis, those that blocked all AI bots were: BBC (both .co.uk and .com domains), The New York Times, Daily Mail, The Telegraph, Sky News, AP News, New York Post, Newsweek, NBC News, Wall Street Journal, Metro, Business Insider, ABC News, Buzzfeed, Huffpost, and The Hill. This means that 34% of the top 50 publishers blocked all bots.
Approximately 14% of the top 50 publishers allowed access to all 11 analyzed AI crawlers. These publishers were listed as: Fox News, The Independent, GB News, Substack, The Standard, Drudge Report, and Politico.
AI BUBBLE WARNING
Bret Taylor, co-founder of AI startup Sierra, said yesterday that AI is "probably" a bubble, and that this bubble has caused both "smart money" and "dumb money" to fund competitors at every layer of the technology stack.
Taylor, who is also the chairman of OpenAI's board, said that the free market will ultimately determine where the value lies and which AI players have the best products. He described himself as an AI optimist.
"When everyone knows that AI is going to have a huge impact on the economy across a wide range of industries and workflows, money is abundant," Taylor told CNBC at the World Economic Forum in Davos, Switzerland. Taylor also said, "I think you'll see a correction and consolidation in the next few years, but I don't think you can get innovation without that kind of complex competition.
