Short answer. Perplexity declares two different agents, with different rules. PerplexityBot exists to make sites appear and be linked in Perplexity's search results, and the documentation recommends allowing it in robots.txt and accepting requests from the published addresses. Perplexity-User visits a page because a person asked a question and, says the same page, generally ignores robots.txt because the request is the user's. The settings are independent and changes can take up to 24 hours. It follows that robots.txt governs the first agent and not the second: to limit the second you need controls on the server. And there is a limit that no file solves: in August 2025 Cloudflare published an analysis of traffic attributed to Perplexity with an undeclared identity, so what you can verify is what your logs show.
The two agents, as declared
| PerplexityBot | Perplexity-User | |
|---|---|---|
| What it is for (declared) | Making sites appear and be linked in search results; it is not used to collect content for training foundation models | User-initiated actions: when a person asks a question it may visit a page to answer and link it; it is not used for crawling or for training |
| Example User-Agent (the version number may change) | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| Published IP addresses | https://www.perplexity.ai/perplexitybot.json | https://www.perplexity.ai/perplexity-user.json |
robots.txt | Allowing it is recommended in order to appear in the results | Generally ignored, because the request is the user's |
Mind who is declaring what: “it is not used for training” is a statement by the provider and from the outside there is no way to verify it. You can count it as a public commitment, not as an established fact. The same page also describes how to configure a web application firewall (WAF) to allow the two agents, and says that the addresses must be taken from the JSON files, which are the source of truth and are updated regularly.
What to write in robots.txt, and what it produces
| Choice | Directives | Expected effect (from the documentation) |
|---|---|---|
| Allow everything | User-agent: PerplexityBot + Allow: /; no rule for Perplexity-User | You can appear in the results; visits made at the user's request are not blocked |
| Forbid only the index | User-agent: PerplexityBot + Disallow: / | You are not part of the results built by that agent; a person who pastes your URL can still start a visit |
| Forbid both in robots.txt | Disallow: / for both | For the first, robots.txt is the tool the page indicates; for the second, according to the documentation, generally not: the prohibition for Perplexity-User is only declarative |
| Really forbid Perplexity-User | A WAF or server rule, combining User-Agent and IP ranges | An actual block, with the limits of identification (below) |
An uncomfortable truth about the directives: the file this site publishes for the two agents is Allow: / for both, with Crawl-delay: 5 added for PerplexityBot. Crawl-delay is not part of the robots.txt standard (RFC 9309) and does not appear on the Perplexity page I reread today: that line is a hope, not a control. I write it because it costs nothing, but I do not count it among the protections.
robots.txt according to RFC 9309, not according to habit
Before writing a rule it helps to know how it will be read. RFC 9309 says three things that matter: the most specific rule wins, that is the one with the most octets, and on a tie allow should be used; crawlers must support * (zero or more of any character) and $ (end of the path); and, in a single sentence, the rules are not a form of access authorization. The program below implements them and compares them with urllib.robotparser, the reader in Python's standard library, on a test robots.txt for a shop: Allow: / at the top, then Disallow: /cart and Disallow: /*?filter=, a group for Perplexity-User and a User-agent: * that forbids everything.
"""What robots.txt tells an agent, according to RFC 9309, compared with urllib.robotparser.
RFC 9309 (September 2022), sections 2.2.2 and 2.2.3: the rule with the most octets wins, on a tie
allow is used (SHOULD), and crawlers must support "*" (zero or more characters) and "$" (end of the path).
"""
import random
import re
import sys
import urllib.robotparser
from urllib.parse import urlsplit
def read_groups(text):
groups, agents, rules, in_rules = [], [], [], False
for line in text.splitlines():
line = line.split("#", 1)[0].strip()
if ":" not in line:
continue
field, value = (p.strip() for p in line.split(":", 1))
field = field.lower()
if field == "user-agent":
if in_rules:
groups.append((agents, rules))
agents, rules, in_rules = [], [], False
agents = [*agents, value.lower()]
elif field in ("allow", "disallow"):
in_rules = True
if value: # an empty "Disallow:" forbids nothing
rules = [*rules, (field == "allow", value)]
if agents:
groups.append((agents, rules))
return groups
def matches(pattern, path):
end = pattern.endswith("$")
body = pattern[:-1] if end else pattern
expression = ".*".join(re.escape(p) for p in body.split("*")) + ("$" if end else "")
return re.match(expression, path) is not None
def allowed(text, agent, url):
parts = urlsplit(url)
path = parts.path + (f"?{parts.query}" if parts.query else "")
groups = read_groups(text)
chosen = [g for g in groups if agent.lower() in g[0]] or [g for g in groups if "*" in g[0]]
rules = [r for _, rs in chosen for r in rs]
found = [(len(p.encode()), permit) for permit, p in rules if matches(p, path)]
if not found:
return True
return max(found)[1] # most octets; on a tie True (allow) > False
def reference(pattern, s):
"""Second calculation, written differently: recursion on the pattern."""
if pattern == "":
return True
if pattern == "$":
return s == ""
if pattern[0] == "*":
return any(reference(pattern[1:], s[i:]) for i in range(len(s) + 1))
return s != "" and s[0] == pattern[0] and reference(pattern[1:], s[1:])
if __name__ == "__main__":
text = open(sys.argv[1], encoding="utf-8").read()
pb = urllib.robotparser.RobotFileParser()
pb.parse(text.splitlines())
trials = [
("PerplexityBot", "https://shop.test/"),
("PerplexityBot", "https://shop.test/products/tent-2p"),
("PerplexityBot", "https://shop.test/cart"),
("PerplexityBot", "https://shop.test/products/tent-2p?filter=red"),
("Perplexity-User", "https://shop.test/products/tent-2p"),
("Perplexity-User", "https://shop.test/private-area/case-1"),
("OtherBot", "https://shop.test/products/tent-2p"),
]
print(f"{'agent':16}{'URL (path)':36}{'RFC 9309':>9}{'robotparser':>13}")
for agent, url in trials:
a, b = allowed(text, agent, url), pb.can_fetch(agent, url)
path = urlsplit(url).path + ("?" + urlsplit(url).query if urlsplit(url).query else "")
print(f"{agent:16}{path:36}{str(a):>9}{str(b):>13}{' <- they differ' if a != b else ''}")
print("Crawl-delay read by robotparser for PerplexityBot:", pb.crawl_delay("PerplexityBot"))
# Same "Allow: /" rule, moved to the bottom of the group: robotparser uses the first one that matches.
inverted = text.replace("Allow: /\n", "").replace("Disallow: /*?filter=\n", "Disallow: /*?filter=\nAllow: /\n")
pb2 = urllib.robotparser.RobotFileParser()
pb2.parse(inverted.splitlines())
for name, url in (("/cart", "https://shop.test/cart"),
("?filter=", "https://shop.test/products/tent-2p?filter=red")):
print(f"Allow at the bottom, {name:10} robotparser {str(pb2.can_fetch('PerplexityBot', url)):5}"
f" RFC 9309 {allowed(inverted, 'PerplexityBot', url)}")
print("Python", sys.version.split()[0])
rng = random.Random(3)
patterns = ["/*?filter=", "/cart", "/*.pdf$", "/a/*/b", "/", "/x$"]
alphabet = "/ab.?=xpdf"
pairs = 0
for _ in range(5000):
s = "/" + "".join(rng.choice(alphabet) for _ in range(rng.randint(0, 9)))
for p in patterns:
assert matches(p, s) == reference(p, s), (p, s)
pairs += 1
print(f"comparison with the recursion on {pairs} pattern/path pairs: they agree")
# Command: python3 rules.py robots_shop.txt
# Output:
# agent URL (path) RFC 9309 robotparser
# PerplexityBot / True True
# PerplexityBot /products/tent-2p True True
# PerplexityBot /cart False True <- they differ
# PerplexityBot /products/tent-2p?filter=red False True <- they differ
# Perplexity-User /products/tent-2p True True
# Perplexity-User /private-area/case-1 False False
# OtherBot /products/tent-2p False False
# Crawl-delay read by robotparser for PerplexityBot: 5
# Allow at the bottom, /cart robotparser False RFC 9309 False
# Allow at the bottom, ?filter= robotparser True RFC 9309 False
# Python 3.14.4
# comparison with the recursion on 30000 pattern/path pairs: they agree
The comparison disproves a common habit. For PerplexityBot the RFC forbids /cart (the rule has more octets than Allow: /) and forbids the URL with ?filter= (the wildcard matches), while robotparser lets both through: it takes the first rule that matches, and Allow: / is the first; when I move it to the bottom the cart is forbidden by it too, but the filter with the wildcard remains allowed, because that reader does not support * in paths. By the RFC, the Python reader is less precise than the text: if you check your rules with a library, check how it treats order and wildcards. I do not know how Perplexity's crawler treats them: its page does not say. The final comparison, with a recursion written differently over 30,000 pairs, confirms only that my implementation matches the RFC; the result holds for Python 3.14.4 and I did not try it on other versions.
The Crawl-delay: 5 line is read by robotparser (its interface returns 5), and this proves only that the reader knows it: the RFC does not define it. The other sentence to remember is the last one: a public file that lists paths is not an access control, and it even reveals which paths you do not want to show.
What is verifiable: your logs
The only independent check is to look at who knocked. The method: classify every log line by User-Agent, then check whether the IP address falls within the ranges published for that agent. A User-Agent that says “PerplexityBot” from an IP outside its ranges is an imitation, not the crawler. The program reads nginx's combined format and the two JSON files in the published shape (a creationTime key and a prefixes array of objects with ipv4Prefix; it also handles ipv6Prefix, which does not appear in today's files). I try it on a log and on two address files that I built myself with the ranges reserved for documentation (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 2001:db8::/32), not on real traffic.
| Line of the test log | User-Agent | IP | Expected |
|---|---|---|---|
| 1, 2 | PerplexityBot | two addresses in the list | consistent |
| 3 | PerplexityBot | IPv6 inside a prefix of the list | consistent |
| 4 | PerplexityBot | address outside the list | imitation |
| 5 | Perplexity-User | address in its own list | consistent |
| 6 | Perplexity-User | address from PerplexityBot's list | not consistent |
| 7, 8 | Chrome on macOS | any address | not attributable |
| 9 | Googlebot | any address | not attributable |
| 10 | truncated line, out of format | ||
The table says what I put in the test file, so the output can be compared with what it must produce. In the program, the function analyze recognizes the declared agent with the regular expression, chooses the address list of that agent and counts the outcome; a line that does not respect the format does not make the analysis fail and is counted separately.
"""Checks from the logs who claims to be Perplexity and whether the IP is among the published ones.
Expected format: nginx "combined". The JSON files have the shape published by Perplexity:
{"creationTime": ..., "prefixes": [{"ipv4Prefix": "..."}]} (ipv6Prefix is handled too).
"""
import ipaddress
import json
import re
import sys
from collections import Counter
COMBINED = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" \d{3} \S+ "[^"]*" "([^"]*)"$')
DECLARED = re.compile(r"(PerplexityBot|Perplexity-User)", re.I)
def load_networks(path):
with open(path, encoding="utf-8") as f:
data = json.load(f)
return [ipaddress.ip_network(p[k]) for p in data.get("prefixes", [])
for k in ("ipv4Prefix", "ipv6Prefix") if k in p]
def analyze(lines, bot_networks, user_networks):
outcomes, unreadable = Counter(), 0
for line in lines:
m = COMBINED.match(line.rstrip("\n"))
if not m:
unreadable += 1
continue
ip, ua = m.groups()
declared = DECLARED.search(ua)
if not declared:
outcomes["other User-Agent (not attributable)"] += 1
continue
agent = declared.group(1).lower()
networks = bot_networks if agent == "perplexitybot" else user_networks
consistent = any(ipaddress.ip_address(ip) in network for network in networks)
outcomes[f"{agent}: {'IP consistent' if consistent else 'IP NOT consistent'}"] += 1
return outcomes, unreadable
if __name__ == "__main__":
log, f_bot, f_user = sys.argv[1:4]
with open(log, encoding="utf-8", errors="replace") as f:
outcomes, unreadable = analyze(f, load_networks(f_bot), load_networks(f_user))
for key, n in sorted(outcomes.items()):
print(f"{n:>3} {key}")
print(f"{unreadable:>3} lines not in combined format")
# Command: python3 agent_log.py access.log
# Output:
# 3 other User-Agent (not attributable)
# 1 perplexity-user: IP NOT consistent
# 1 perplexity-user: IP consistent
# 1 perplexitybot: IP NOT consistent
# 3 perplexitybot: IP consistent
# 1 lines not in combined format
The counts match the table, and they match an independent check done with grep -c on the file (4 lines with PerplexityBot, 2 with Perplexity-User). The most instructive result, though, is what the program cannot say: the three “not attributable” lines are three requests that an ordinary browser, or another crawler, would have produced identically. The program answers a precise question, do the requests that declare themselves Perplexity really come from the published addresses?, and not the one that matters to whoever wants to block: how much Perplexity traffic arrives without declaring itself. For that there is no reliable method from the outside, other than probabilistic heuristics, which get it wrong. Reload the two files periodically: the documentation indicates them as the source of truth.
Cloudflare's analysis of 4 August 2025, for what it says
Cloudflare published the results of tests run on new domains, never indexed, with a robots.txt that forbade all access and WAF rules that blocked the two declared agents. Querying Perplexity about those domains, it reported having obtained detailed answers about the content. Besides the declared agent, it observed a generic Chrome-on-macOS User-Agent, from addresses not present in the published ranges, with rotation of IPs and of autonomous systems (ASNs). The reported estimates are 20-25 million requests a day for the declared agents and 3-6 million for the undeclared identity, the latter across tens of thousands of domains; Cloudflare says it recognized the second traffic with machine learning and network signals. Afterwards it removed Perplexity from its list of verified bots and added heuristics to its managed rules.
How to use it: it is a security vendor's analysis of its own tests, referring to a specific period, and in the page I read there is no reply from Perplexity (I did not look for one elsewhere). Cloudflare itself writes that the observed behavior would almost certainly have changed after the publication. It does not prove that it repeats today and does not rule it out: the professional answer is “if this matters to you, measure in your logs and prepare a block that does not depend on the bot's courtesy”.
If you decide to block: the rule on the server
Perplexity's page describes for Cloudflare's WAF a rule that combines User-Agent and address, and warns that “Bot Fight Mode” cannot be skipped with custom rules: it can therefore challenge these agents even when you have allowed them. To block imitation the logic is the same, turned around: a User-Agent that declares PerplexityBot but an address outside the list. The following is the documentation's rule, followed by an adaptation of mine.
# 1) Allow: User-Agent AND published address, for each agent (shape from the documentation)
(http.user_agent contains "PerplexityBot" and ip.src in {<PerplexityBot IP ranges>})
or (http.user_agent contains "Perplexity-User" and ip.src in {<Perplexity-User IP ranges>})
# 2) Block imitations: it says it is PerplexityBot but the address is not among the published ones
# (my adaptation, not tried on Cloudflare)
(http.user_agent contains "PerplexityBot" and not ip.src in {<PerplexityBot IP ranges>})
The scheme is to be adapted and I did not run it: the placeholders in angle brackets must be replaced with the ranges taken from the two JSON files, and the second block is an adaptation of mine that I did not try on Cloudflare. Rule 1 goes above the rules that block or challenge, because a “Skip” action skips only what comes after it; rule 2 blocks. Any rule must be reread every time the files change, and a wrong configuration can block the agent you wanted to allow.
Two decisions, declared as scenarios
A shop that does not want filters in the index
Hypothesis: an online shop with thousands of product pages and filter URLs generated from parameters wants to allow the product pages, but not the cart and the filtered pages. It writes Disallow: /cart and Disallow: /*?filter=. What changes: according to the RFC the wildcard must be supported, but I do not know whether Perplexity's crawler supports it and Python's reader, as you have seen, does not. Test: look in the logs at which paths are actually requested after the change.
A portal with public pages and restricted areas
Hypothesis: a healthcare facility or a municipality has a portal with public pages and a restricted area, and thinks that Disallow: /private-area protects it. It does not: the RFC says the rules are not an authorization and the file lists the paths to anyone. Remedy: the restricted area sits behind authentication, and you check with a request without credentials that no personal content answers 200.
What you can do now
You can tell the two declared agents apart and say what robots.txt controls for each, you know that the Python standard-library reader and RFC 9309 do not say the same thing about order and wildcards, you can check from the logs whether whoever declares themselves Perplexity comes from the published addresses, and you know that undeclared traffic will not be visible with this method. If you want a real block, you need a rule on the server. The picture of Perplexity's APIs is in article 1, and what to expect from visibility, once the crawler is allowed, in article 8 below.
Frequently asked questions
Does Perplexity respect robots.txt?
According to the documentation, PerplexityBot is the agent for which allowing access in robots.txt is recommended; Perplexity-User, which acts at a user's request, generally ignores it. The two settings are independent and changes can take up to 24 hours.
How do I block Perplexity-User?
robots.txt is not enough, because it is generally ignored. You need a server or WAF rule that combines User-Agent and the IP ranges published in the perplexity-user.json file.
How do I verify that a request is really PerplexityBot?
Compare the IP address with the ranges in the perplexitybot.json file: a User-Agent that declares itself PerplexityBot from an IP outside the ranges is an imitation.
Does Perplexity use undeclared crawlers?
Cloudflare reported so on 4 August 2025, with estimates of 3-6 million requests a day for the undeclared identity. I have not measured current behavior: for your site the only evidence is in your logs.
Sources
- Perplexity, Perplexity Crawlers - description of the two agents, example User-Agents, behavior with respect to robots.txt, WAF configuration; read in full on 13 November 2026, the page carries no date
- Perplexity, PerplexityBot IP addresses and Perplexity-User's - shape of the files, number of prefixes and
creationTime; downloaded on 13 November 2026 - Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives - test method, request estimates, Cloudflare's countermeasures (4 August 2025); read in full on 13 November 2026
- IETF, RFC 9309: Robots Exclusion Protocol - most-specific-match rule, special characters “*” and “$”, sentence on access authorization; sections 1, 2.2.2 and 2.2.3 read on 13 November 2026
Commenti
Caricamento commenti...
Sign in to comment