Internet infrastructure firm Cloudflare will now default to blocking AI scraper bots from accessing original content on client websites without permission or compensation.
AI has changed the internet. Before, when a user typed a query into a search engine, they were presented with a series of links. Clicking on one would direct them to a website, and this exchange generated traffic and ad revenue for content creators.
Nowadays, AI chatbots such as ChatGPT are being used instead of traditional search. These chatbots are trained on data scraped by AI bots that vacuum up information from the internet.
AI scraper bots or crawlers often take this data without permission, with content creators seeing almost no traffic and therefore almost no value.
Cloudfare, which powers more than 20% of the internet (15 million domains), is now putting a stop to this by changing the default to block AI crawlers unless they pay creators for their content.
Cloudfare’s new tech means its clients can now choose...