Cloudflare goes to war with Microsoft, Google, and OpenAI's bots, with blanket free tools to block all crawlers

What you need to know

• Cloudflare is a global cloud provider that provides security and DDoS protection to millions of websites, protecting roughly 20% of all global internet traffic. 
• Yesterday, Cloudflare announced a new free tool that it will begin offering to all customers, specifically designed to blot out AI crawlers. 
• AI bots used by the likes of Google, Microsoft, OpenAI, and many others, steal copyrighted information from websites like ours in order to train its premium AI services. 
• Last week, Microsoft's AI chief said that all publicly-facing internet content was "freeware" to be stolen for its AI aspirations. 

Last week, Microsoft's AI chief said that public content on the open web was "freeware," giving the trillion dollar corporation free license to steal any and all content you've published on the web to power its premium products. The backlash to the gaffe was significant, and has served as a claxon of sorts for web content providers to reconsider their relationship with companies like Microsoft, who seek to profit from content creators' hard work while giving literally nothing back in return. Cloudflare may have also just handed those same creators a much important defensive weapon in the fight back. 

Cloudflare is a global internet service and hosting company, powering roughly 20% of all web traffic. Offering things like DDoS protection from attacks and bot verification checks on websites, Cloudflare has been instrumental in improving the general quality of the world wide web, using its massive server infrastructure as a vast security layer for companies of all shapes and sizes. 

Yesterday, the firm announced a new feature it will begin rolling out to all users, even those on its free tier, designed to combat generative AI. 

Declare "AIndependence," Cloudflare says on its blog. Its new system will allow users to opt-in to block AI bots and crawlers from accessing websites, effectively preventing Microsoft, Google, OpenAI, and others from stealing web content for free. 

After surveying its users, Cloudflare shared data that over 80% of its customers wanted the ability to shut Microsoft out from stealing their content. "We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly." Cloudflare continued, "To help, we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier." 

Generative AI training content is becoming lucrative and valuable to companies like Google and Microsoft. Google reportedly paid upwards of $60 million dollars for access to all of reddit's content for training its models, which also hilarious resulted in sarcasm and trolling appearing in Google search results. 

Can Microsoft discover a healthy balance?

I've written before how it should be in Google and Microsoft's interests to create a healthy, symbiotic relationship between human creators and its generative AI efforts. Generative AI undoubtedly has some role to play in the future of tech, but I feel like companies are still struggling with what exactly that looks like for customers.   Right now, generative AI seems best used for the most basic writing tasks, such as producing formal emails or summarising long form text. Even then, it has its issues doing even basic things when you dig into it. I've found that it often simply erodes productivity rather than enhances it, given that you have to double check everything AI does to avoid their "hallucinations." 

AI is incredibly expensive to run, too. AI queries are battering Google's emission reduction efforts, and I doubt Microsoft is faring much better here, either. Even when you disregard the climatological impacts, the business model doesn't exactly seem to work today either. Microsoft gives away Copilot for free, and I'm not sure why I'd ever pay for it.

RELATED: Why Microsoft won't be the company that mainstreams AI with consumers

The low-hanging-fruit feature that Google and Microsoft have opted for very quickly is search summaries. We produce thousands of guides here at Windows Central, and Microsoft Copilot will now simply take the content from the article and reproduce it, depriving us of traffic, and thus income. That's bad for us, but it's also bad for Microsoft and Google. If human content creators can no longer effectively monetize and earn a living, increasingly large swaths of the internet will become AI-generated. Much like JPEG compression, the quality of the content will decrease as AI starts learning from other AI, rather than human creators. Since ultimately, AI doesn't "understand" the content its reproducing, and can only infer context by doing comparisons against human content. This phenomenon is called model collapse, and it's a real concern among serious AI scientists. But right now, all Google, Microsoft, and others are thinking about is getting ahead. 

For this kind of tech to actually proliferate, it still needs human intervention. The alarm Microsoft's AI chief generated with his irresponsible "freeware" comments has contributed to the on-going backlash. And now, companies like Cloudflare are stepping up to help fight back. It won't be long until others follow, and Microsoft may have to actually reckon with its cavalier attitude towards its industrial-scale content theft.