How to Allow PerplexityBot in robots.txt (and Why It Matters for AI Citations)

crawler rules are grouped by user-agent. Once a bot finds a group that names it, it uses that group and ignores every other one, including the wildcard group. That behavior is the most common source of confusion: your site-wide rules may block nothing, yet PerplexityBot can still be limited by its own group.

The minimal allow rule is a group with PerplexityBot as the user-agent and an Allow line for the root:

• Allow everything: User-agent: PerplexityBot, then Allow: /. This lets the crawler fetch any URL.
• Allow by omission: User-agent: PerplexityBot, then no Disallow lines. An empty group places no restrictions on that bot.
• Allow with exceptions: keep Allow: / and add a Disallow line for specific paths, such as cart or account pages, you never want indexed.
• Case matters: crawler user-agent names are matched case-sensitively, so write PerplexityBot exactly, not perplexitybot or Perplexitybot.

A Disallow: / line under User-agent: PerplexityBot blocks your entire site from that crawler, even if other groups allow it. Remove it if it is there. The same syntax pattern applies to other AI search crawlers, and a single misconfigured group only affects the bot it names.

Many sites block everything with a blanket User-agent: * group, usually intended for scrapers or admin paths, and assume it is harmless. For AI crawlers it is not. Until a specific bot is named, the wildcard group governs it, so a wildcard Disallow: / shuts the door on every crawler that has not been given its own entry.

The precedence rules, in practical order:

1. The crawler looks for a group whose User-agent line matches its name exactly.
2. If none exists, it falls back to the wildcard group.
3. Within the applicable group, the longest matching path wins between an Allow and a Disallow for the same URL.
4. Groups for other named bots have no effect on PerplexityBot at all.

Naming PerplexityBot explicitly is what frees it from the wildcard group. That is the fix in almost every case where a site owner believes the crawler is allowed but the site never appears in answers.

A rule that looks correct in the editor can still fail in practice. robots.txt must be served at the root of the exact domain and subdomain being crawled, so a rule on example.com does not cover www.example.com or a blog subdomain. If either hostname returns a different file, or a soft 404 page instead of text, the crawler may treat access as unknown and back off.

A short verification sequence catches most problems:

• Open yourdomain.com/robots.txt in a browser and confirm it returns plain text, not an HTML error page.
• Read the PerplexityBot group line by line and check nothing there starts with Disallow: /.
• Run the URL you want cited through a robots.txt testing tool, selecting PerplexityBot as the user-agent.
• Check server logs or analytics for PerplexityBot hits after a few days; repeated 403 or 429 responses point to a firewall or rate limit rather than the file.
• Confirm your CDN or security layer is not challenging the bot before robots.txt is even served.

Success at this stage only proves access. Being quoted also requires answer-first page structure and question-phrased headings, which is why a GEO audit is the quickest way to see what is blocking citation once the crawler is let in. Vision Creative Studio runs that audit against the crawler permissions, schema and content formatting that AI answer engines reward.

Robots.txt user-agents map to different jobs. PerplexityBot fetches pages so Perplexity can answer a question with current information and cite sources. GPTBot collects data for OpenAI model training, and CCBot collects for the Common Crawl archive used in many training pipelines. They are not interchangeable.

That means a file that allows GPTBot can still leave PerplexityBot blocked, and vice versa. The decision you are making per group is: do I want this bot to retrieve my pages for search and citation, for training, for both, or for neither? Many sites want search and citation access while declining training access, which is a coherent policy as long as each bot gets its own group.

The practical check is to list the search-oriented crawlers separately from the training-oriented ones and confirm each has an explicit group:

• Search and citation crawlers: PerplexityBot, OAI-SearchBot, and the Google-Extended setting that governs Gemini and Vertex grounding. These affect whether your pages can be retrieved and quoted.
• Training crawlers: GPTBot, CCBot, Bytespider. Blocking these does not remove you from AI answers, because answer engines fetch live pages at question time.

One caution: Google-Extended does not control classic Google AI Overviews, so do not conflate the two. And a wildcard block affects every bot that has no group of its own, which is why naming each bot you care about is safer than relying on the wildcard.

None of these produce an error message. The crawler reads the file, decides it is not welcome, and moves on. You find out weeks later when your pages are absent from answers.

Watch for these in order of frequency:

1. Wildcard-only blocking: a User-agent: * group with Disallow: / and no PerplexityBot group. The bot falls back to the wildcard and is fully blocked.
2. A leftover Disallow: / in the PerplexityBot group: often pasted in during a site-wide lockdown and never removed. This blocks the bot even when other groups allow it.
3. Wrong case: user-agent names are matched case-sensitively. Perplexitybot or perplexitybot is a different string.
4. Wrong hostname: robots.txt at the root only covers that exact host. A file on example.com does not cover www.example.com or a subdomain, and each must be checked.
5. HTML instead of text: a soft 404 or login page returned in place of the file leaves the crawler unsure and likely to back off.
6. Firewall or rate limiting: a 403 or 429 at the edge blocks the bot before robots.txt is even read, which no file edit can fix.

Fixing the file is necessary but not sufficient. Once access is confirmed, the page itself has to be quotable: a direct answer near the top, question-phrased headings, and visible freshness. Vision Creative Studio's GEO service addresses both sides, starting with a GEO audit that identifies what is actually blocking citation rather than guessing.

Answer engines do not cite from memory alone. For current questions they run web searches, read the top results, and compose an answer with sources. That means ranking in ordinary search is a prerequisite for being cited, and unblocking the crawler only removes one barrier.

What you control after access is confirmed:

• Whether the page ranks for the query Perplexity runs. A model splits a question into one to three searches and then runs follow-up grounding queries, often brand or entity names. A page that ranks for those exact phrases can be cited quickly.
• Whether the page answers in a quotable form. A 40-60 word direct answer at the top and under each section heading is easy for a model to lift.
• Whether the content looks current. AI engines weight recently published and recently updated pages heavily. Content refreshed within roughly a month tends to be cited more, and content untouched for a quarter or more tends to lose citations.
• Whether the headings match real questions. Question-phrased headings mirror how people ask in a chat assistant and give the model clean anchors.

Because citation behavior is volatile day to day even when a site has not changed, no one can promise a specific citation. What you can do is build the foundation these engines reward and track results on a schedule. Vision Creative Studio tracks citations across ChatGPT, Perplexity, Gemini and Grok, and Iris, Vision's in-house content engine, refreshes pages on a cadence so freshness signals do not decay.