Cloudflare's new AI tool traps web-crawling bots with AI-generated decoys
What's the story
Cloudflare, a leading global internet infrastructure company, has launched a new tool, AI Labyrinth.
The innovative tool is aimed at fighting web-crawling bots that scrape data from websites for AI training purposes without taking permission.
As the company's blog post notes, this free opt-in tool uses "AI-generated decoy pages" to divert crawlers and waste their resources.
Here's how it works.
Innovative strategy
Fresh approach to combat web-scraping bots
Traditionally, websites have depended on a system called robots.txt to control scrapers' access.
The text file either allows or denies permission for scraping, but many AI companies, including Anthropic and Perplexity AI, have been accused of ignoring it.
Cloudflare processes over 50 billion web crawler requests every day and has tools to detect and block malicious ones.
But that often prompts attackers to switch tactics in an ongoing "arms race."
Decoy pages
Unique countermeasure against web scrapers
AI Labyrinth, on the other hand, makes bots deal with irrelevant data. It serves as "a next-generation honeypot," luring AI crawlers to keep following links to fake pages.
This way, Cloudflare can easily flag malicious bots for its bad actor list, and detect new bot patterns and signatures it may not have flagged otherwise.
The company says these links shouldn't be visible to human visitors.
Accessibility
A tool for website administrators
Website admins can opt into using AI Labyrinth by heading over to the Bot Management section of their site's Cloudflare dashboard settings and turning it on.
The company also announced future plans for the tool, noting that this "is only the first iteration of using generative AI to thwart bots."
Cloudflare plans to create "whole networks of linked URLs," which bots will find hard to detect as fake.