Cloudflare’s Bot Preference Sync Can Write Your Robots.txt — But There’s a Catch
Managing robots.txt can seem like a simple SEO task, but the file plays an important role in communicating how websites want compliant crawlers to access their content. Cloudflare’s Bot Preference Sync aims to make this process easier by automatically translating AI bot preferences from its dashboard into robots.txt rules.
The idea is useful, particularly for website owners who struggle to keep their published crawler instructions aligned with their actual Cloudflare security rules. However, the approach also raises an important question: Can three broad AI crawler categories accurately represent the policy a website owner actually wants?
What Is Cloudflare Bot Preference Sync?
Cloudflare's Bot Preference Sync automatically generates robots.txt directives based on AI bot policies configured in the Cloudflare dashboard.
The generated rules are placed at the beginning of the existing robots.txt file between Cloudflare markers:
# BEGIN Cloudflare Bot Preference Sync
and
# END Cloudflare Bot Preference Sync
Existing robots.txt content remains underneath those generated directives. Cloudflare announced that the feature would be available from the free tier, with the feature enabled by default for new customers under the announced configuration.
The goal is straightforward: keep a website's published crawler preferences closer to the rules actually being enforced at the network level.
Why Keeping Robots.txt and Edge Rules Aligned Matters
A website can have two different layers of crawler control.
The first is robots.txt, which communicates which crawlers should or should not access particular content.
The second is server or edge-level enforcement, such as Cloudflare rules that can actually return an HTTP response such as 403 Forbidden.
If these two layers disagree, the website may effectively be telling crawlers one thing while enforcing another.
Cloudflare's announcement argues that some crawlers may treat this contradiction as a reason to disregard the site's stated preferences or attempt to bypass enforcement. However, Cloudflare does not publicly identify which crawlers behave this way or how frequently this happens.
This distinction is important because robots.txt is fundamentally a communication mechanism. It is not a security barrier.
Bot Preference Sync Uses Three Main Categories
Cloudflare's system generates robots.txt rules from three AI bot policy categories:
Search
Agent
Training
According to the supplied source, each category provides three choices:
Block on all pages
Block on pages with advertisements
Allow
The crawler classifications are determined by Cloudflare's tracked bot list. Individual crawlers cannot be excluded from the synchronised policy, meaning the system does not provide complete per-crawler control.
This is where the system can become restrictive for publishers with more detailed AI crawler policies.
Why Per-Crawler Control Can Matter
Not every website owner wants the same policy for every AI crawler.
For example, a publisher might:
Allow one AI company's crawler
Block another company's crawler
Permit search-related crawling
Restrict model training
Allow some services because they send referral traffic
Block others because they provide little measurable value
These decisions can be made individually according to a publisher's commercial strategy.
The source gives an example of a policy that allows OpenAI's GPTBot, Anthropic's crawler and PerplexityBot while blocking Bytespider and Meta's meta-externalagent. Such a policy cannot be represented using only Cloudflare's three broad settings.
For websites with simple policies, Cloudflare's system can therefore save time. For websites with detailed crawler-by-crawler rules, manually maintaining robots.txt may still be necessary.
Cloudflare's Conditions for AI Crawlers
Cloudflare has also described conditions that AI crawlers need to satisfy to avoid being classified as opaque when a website chooses to disallow training.
The published conditions include:
The crawler must respect a site's no-training preference.
The company must provide a way for publishers to opt out of AI summaries.
The company must provide URL-level visibility into content used for training, along with relevant search and training metrics.
The company must publicly demonstrate that opting out of training does not negatively affect traditional search results.
Cloudflare does not name individual companies in these conditions.
The source notes that some of the requirements appear particularly relevant to Google and Microsoft because of how their search and AI products operate.
Google and AI Search Controls
Google's crawler documentation states that Google-Extended controls whether content crawled by Google can be used to train Gemini models or ground Gemini responses. Google also states that Google-Extended does not affect inclusion in Google Search or serve as a Google Search ranking signal.
However, controlling Google's AI-generated search features is more complicated.
Google's documentation describes controls including:
nosnippet
data-nosnippet
max-snippet
noindex
These controls affect how content can appear in Google Search. According to the supplied source, Google's documentation does not provide a separate switch specifically allowing a publisher to remain eligible for ordinary search snippets while opting out of AI Overviews.
Microsoft has offered a different approach. Its documentation states that content using NOARCHIVE can be excluded from Bing Chat answers while remaining available in search results.
Cloudflare's Default Settings Deserve Attention
Cloudflare subsequently changed the starting configuration for new domains.
According to the supplied source, from September 15 the default approach for new domains was:
Search: Allowed
Training: Blocked on pages displaying advertisements
Agent: Blocked on pages displaying advertisements
The onboarding question about whether a website monetises pages with advertisements can therefore influence the AI training position that Cloudflare publishes through robots.txt.
This makes the default configuration something website owners should understand rather than simply accept without checking.
Robots.txt Is Not a Security System
One of the most important points to understand is that robots.txt does not technically prevent a crawler from accessing a website.
It is a request to compliant crawlers.
A crawler that deliberately ignores robots.txt can still request URLs unless another layer of enforcement blocks it. The supplied source describes examples of unwanted automated traffic that ignored the intended boundaries of a robots.txt file.
This creates a useful distinction:
Robots.txt communicates policy.
Cloudflare's edge controls enforce policy.
Using both together can provide a stronger approach than relying on either one alone.
Should You Use Cloudflare Bot Preference Sync?
The answer depends on how detailed your crawler policy needs to be.
If your policy is relatively simple—for example, allowing AI crawlers or broadly blocking training crawlers—Bot Preference Sync can reduce manual maintenance.
However, if your AI crawler strategy is based on individual companies or individual bots, Cloudflare's three-category system may not accurately represent your policy.
In that situation, Cloudflare's documented option is to disable synchronization and manage robots.txt manually.
A Simple Check for Website Owners
Before relying on Bot Preference Sync, review these three areas:
1. Check Your Current Robots.txt
Open your existing robots.txt and look for rules relating to AI and search crawlers.
Older rules may no longer reflect your current business strategy.
2. Review Cloudflare AI Bot Settings
Compare your robots.txt with the settings under Cloudflare's AI bot policy controls.
If the two don't match, decide which policy represents your current preference.
3. Check Whether Broad Categories Are Enough
Ask yourself whether your policy is:
“Allow or block AI crawlers broadly”
or
“Allow this company but block that company.”
The first approach can fit Cloudflare's simplified categories. The second may require manually maintaining robots.txt.
Final Thoughts
Cloudflare's Bot Preference Sync addresses a real website-management problem: the risk of robots.txt becoming outdated while crawler rules are being managed somewhere else.
Automatically keeping the two aligned can be useful, especially for website owners who don't regularly review their crawler policies.
But automation also means accepting the policy structure provided by the platform.
If your AI crawler strategy is simple, Cloudflare's approach may save time. If your policy depends on individual crawlers, companies, search products or commercial relationships, three broad categories may not be enough.
The most important step is simple: check what Cloudflare is writing into your robots.txt. A policy file you haven't reviewed may communicate decisions that don't accurately represent your current website strategy.
Comments