Does Cloudflare block AI crawlers by default?

Cloudflare's default posture is protective, but it is aimed at traffic that looks abusive: scrapers hammering a site, vulnerability scanners, and bots spoofing user agents. Verified AI crawlers that identify themselves honestly are treated differently from anonymous junk traffic.

The confusion comes from how the dashboard presents bot controls. Several options sit close together, and a few of them are worded in ways that sound broader than they are. That is where unintended blocking creeps in.

• Bot Fight Mode is a free toggle that challenges automated traffic. It can catch AI crawlers along with everything else, because it is not selective about intent.
• AI crawler controls let a site owner allow, block or charge specific AI bots. These are opt-in decisions, not baseline behavior.
• Custom WAF rules and firewall rules can match on user agent or ASN, and a rule written for one problem can sweep up crawlers nobody meant to block.
• Managed rules and challenge actions may interrupt a crawler that does not run JavaScript or solve a challenge. AI crawlers generally do neither.

The practical takeaway: assume nothing about the defaults. Check the settings that are actually on for your zone. A GEO audit is the quickest way to see what is blocking citation, because it checks crawler permissions and response behavior from the outside, the way an AI engine sees your site.

Checking is mostly a matter of looking in three places: the file that declares your rules, the platform settings that enforce them, and the actual response your server returns. All three need to agree.

1. Read robots.txt. Look for user-agent blocks naming OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended or a wildcard. Disallow lines aimed at everything will catch AI crawlers too.
2. Audit Cloudflare settings. Review AI crawler controls, Bot Fight Mode, and any firewall or WAF rules that match on user agent or known bot lists.
3. Test a real request. Fetch a page with an AI crawler user agent and confirm you get content back, not a challenge page or an error code.
4. Check logs. See whether those user agents appear at all. Silence usually means something upstream is stopping them.

That last row matters. Access is a prerequisite, not a result. If crawlers reach the page and the page still is not quoted, the problem has moved from technical access to content architecture.

Not every bot that says "AI" is doing the same job. Search and citation crawlers retrieve a page because a person asked a question and the engine needs current information to answer it. Training crawlers collect material for future models and never send a visitor to your site.

Cloudflare's controls reflect that split by letting owners make separate decisions. You can welcome retrieval bots while declining training crawlers, or make a different call. Some publishers have pushed for compensation for AI crawling, and the platform's tools increasingly support that kind of choice.

• OAI-SearchBot handles OpenAI's search retrieval. Allowing it supports citation in ChatGPT search results.
• PerplexityBot retrieves pages for Perplexity answers and cites what it reads.
• ClaudeBot covers Anthropic's retrieval behavior.
• Google-Extended governs Gemini and Vertex grounding. It does not control classic Google AI Overviews, which follow ordinary Googlebot access.
• GPTBot and CCBot are primarily training crawlers, a separate decision from citation access.

Blocking a training bot does not block citation, and allowing one does not guarantee it. Treat each crawler as its own decision, documented so nobody reverses it by accident later.

Unintended blocking almost never comes from one dramatic switch. It accumulates. A site owner turns on a security feature to stop scraping, adds a rule during an attack, or copies a robots.txt snippet from a template, and the effect on AI crawlers goes unnoticed because nothing breaks visibly.

• Bot Fight Mode. The free toggle challenges automated traffic without distinguishing intent. Crawlers that do not run JavaScript or solve a challenge get stopped cold.
• WAF rules on user agent. A rule built to stop one abusive bot can match a family of user agents and catch legitimate AI crawlers in the same pattern.
• ASN and IP-based rules. Rules that block whole hosting ranges or networks can sweep up crawler infrastructure nobody meant to touch.
• Copied robots.txt. A wildcard Disallow added during a spam spike blocks everything, including the bots you want.
• Challenge pages on all paths. A blanket challenge on HTML pages reads as protection but returns a challenge instead of content to every crawler that arrives.

These settings tend to arrive quietly and stay. Nobody documents them, and the person who added them may have moved on. The fix is less about clever configuration than about inventory: know which controls are active on your zone and what each one actually matches.

A GEO audit approaches this from the outside. It requests pages the way an AI engine would and records what comes back, which surfaces challenges and error responses that dashboard settings alone will not reveal.

This is the most common dead end in GEO. The crawler test passes, logs show the bots arriving, and the citations still do not appear. That usually means the technical layer is fine and the content layer is doing the work of losing.

AI engines retrieve a page, read it, and decide whether a passage answers the question directly enough to quote. Pages organized around marketing language rather than questions give the model little to lift.

• No direct answer. If the answer to the headline question appears in paragraph six, after the brand story, a model has to guess. A short answer capsule near the top removes the guesswork.
• Decorative headings. Labels like "Our Approach" do not match how people phrase questions in a chat assistant. Question-phrased headings do.
• Content that has aged. AI engines weight recently published and recently updated material heavily. Pages untouched for a quarter or more tend to slip out of citations.
• Thin coverage. A page that never states a clear, self-contained fact gives a model nothing quotable.

Notice what is not on that list: anything about Cloudflare. When access is solved, the remaining work is content architecture and upkeep. Vision's in-house content engine, Iris, exists for that second job, publishing answer-first pages on a steady cadence and refreshing them so they stay current. The audit is still the place to start, because it tells you which layer is actually failing.

The decision is a business one dressed as a technical one, so make it deliberately and write it down. Most sites want to be quoted in AI answers. That points toward allowing retrieval crawlers and thinking carefully about training crawlers, which do not send visitors but do shape future models.

1. List the retrieval crawlers that matter to you. OAI-SearchBot, PerplexityBot and ClaudeBot cover the assistants most likely to answer questions about your category. Google-Extended affects Gemini and Vertex grounding but not classic AI Overviews, which follow ordinary Googlebot access.
2. Decide on training crawlers separately. GPTBot and CCBot are primarily training crawlers. Blocking one does not block citation, and allowing one does not guarantee it. Either choice is defensible if it is intentional.
3. Keep robots.txt and Cloudflare aligned. A rule in one place and permission in the other produces unpredictable results and confusing diagnostics.
4. Document the choice. Note which bots are allowed, why, and where the controls live, so the next person to touch the dashboard understands the intent.

Then revisit it. Cloudflare's AI controls are evolving, crawler names change, and settings added during an incident are easy to forget. A periodic review keeps the posture deliberate rather than accidental.

Where a business has no existing content to structure, the work shifts to writing pages that are worth quoting in the first place, which is part of what a GEO engagement covers when there is nothing to audit yet.