The internet is changing with AI and Cloudflare is fighting it
© unsplash
Here’s what that means for websites and AI companies
Cloudflare, one of the most widely used services on the internet, is making a big change: it will now block AI bots from reading websites by default. This move could shake up how artificial intelligence companies gather data from the web.
Cloudflare is used by about 1 in 5 websites, which means this change could cut off a large part of the internet from AI companies that collect data to train their models.
Why Cloudflare is doing this
AI bots—also called AI crawlers—scan websites to collect information. That data is often used to train AI tools like chatbots or content generators. Until now, these bots have been able to gather content freely from many websites.
But site owners and publishers have raised concerns about AI tools using their work without permission or payment. Cloudflare wants to change that by giving more power back to content creators.
Here’s what’s changing
Blocking AI bots by default:
All new websites that join Cloudflare will automatically block AI bots unless the owner chooses otherwise.
More control for website owners:
Site owners can now choose which AI bots they want to allow. For example, they can say yes to bots that are used for search engines, but no to those that copy content for training AI.
A new ‘Pay Per Crawl’ option:
Cloudflare is also launching a system where AI companies must pay to access content. This new model—called Pay Per Crawl—lets publishers set their own prices. AI companies can then decide if they want to pay to access that content.
What this means for the internet
Cloudflare’s move could create a new way for content creators to earn money when AI companies use their work. It also helps protect original content from being taken without permission.
Many major media and tech companies have already signed on, including Time, BuzzFeed, The Atlantic, and Ziff Davis. These companies hope this change will lead to fairer rules and better compensation in the age of AI.
Why it matters
Until now, stopping AI bots from scraping websites wasn’t easy. Some bots ignored standard rules, and many content systems didn’t have proper tools to block them. Cloudflare’s update makes it easier for publishers to say: "Yes, you can use my content" — or — "No, you can't."
As Cloudflare CEO Matthew Prince put it:
“AI is going to change the web’s business model. We want to protect the people who create original content while still allowing AI companies to innovate—with permission and fairness.”