See whether AI and search bots get into your site and what they receive
AI and search bots are the programs, such as GPTBot, ClaudeBot, PerplexityBot and Googlebot, that read pages to train models, build answers or index them. Path Ranker requests your home page with the identity of 18 bots, from ChatGPT, Claude and Perplexity to Googlebot and Bingbot, and compares each response with what a browser gets. It shows who gets in, who is blocked, who gets a different page and what robots.txt says to each one.
How it works
Fetches the page as a browser
The home page is first read as a regular browser would read it. The size and title of that response become the reference.
Repeats it as each bot
The same page is requested 18 times, each with the user-agent of one bot, four at a time. robots.txt is read with the longest-match rule, the same one Google uses.
Compares and gives a verdict
For each bot: HTTP status, size, title and response time. The verdict is gets in, blocked, different content or error, with a curl command to repeat it.
What it checks
The 18 bots
OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended), Bingbot, Applebot, meta-externalagent, Amazonbot, CCBot, Bytespider, GrokBot and DeepSeekBot.
What each one is for
8 are training bots, 7 are search bots and 3 fetch a page on demand, when a person asks the assistant to open it. Blocking training bots shows only as information; blocking search or on-demand bots is a high-severity finding.
Blocked by the server
A 401, 403, 429 or 503 response, or a challenge page ("Just a moment", "Checking your browser", "Access denied"), counts as blocked. The fix is to allow those user-agents in your firewall or CDN.
A different page
If the bot receives less than half the HTML the browser receives, or a different title, the verdict is different content. The fix is to serve the same HTML to everyone.
What robots.txt says
For each bot, whether the site root is allowed and which line decides (for example, "disallow: /"). A robots.txt that blocks search assistants becomes a finding, with the rule as evidence.
Each response
The report has a table with bot, company, purpose, robots.txt, status, size in KB, response time in ms and verdict.
The command to check it
Each finding comes with the command to repeat it, such as curl -A "GPTBot" -o /dev/null -w "%{http_code}" followed by your URL, and curl -s to read robots.txt.
What it does not do
- The test uses the identity (user-agent) of each bot, but the request comes from the Path Ranker server, not from the IP addresses of those companies. A firewall that checks IPs may treat the real bot differently.
- It tests the home page, and robots.txt is evaluated for the site root. A folder blocked further down does not change the verdict in this table.
- It compares the HTML each bot receives, without running JavaScript.
- xAI and DeepSeek do not publish an official crawler page. For GrokBot and DeepSeekBot, the test uses the name found in public robots.txt lists, and the report marks them with an asterisk.
Which plans include it
Starter, Pro and Enterprise, in the full analysis or in the "AI (GEO and AEO)" group. The Free plan does not include the bot test: the free analysis reads only the home page for performance, technical SEO and structured data.
See pricingFrequently asked questions
- That is a company decision, and the report treats it as information, not as an error. The serious finding is blocking search and on-demand bots, because those are the ones that read the page when an assistant looks for an answer.
- Because the block came from the server, before robots.txt: a firewall, a CDN or an anti-bot rule answered with 403, 429 or a challenge page. Allow those user-agents in the bot rules of your firewall or CDN and run the analysis again.
- The bot received less than half of the browser HTML, or a page with another title. In practice, that bot is reading something other than your page.
- In a terminal, run curl -A "GPTBot" -o /dev/null -w "%{http_code}" followed by your site URL, swapping the name for any bot you want. 200 means the server let it in; 403 or 429 means it was blocked.
- No. It is 18 requests for the home page, like the ones any visitor makes, and nothing is changed in robots.txt or on the server.
Other features
Bing analysis
Your business on Bing Maps and your site in Bing search, which also feeds Copilot
Core Web Vitals
Real Chrome users and Lighthouse on mobile and desktop, side by side
Structured data
schema.org in JSON-LD, microdata and RDFa checked against the official vocabulary
E-E-A-T and trust
28 experience, expertise, authority and trust signals, each with the page it came from
SEO and AI goals
Ten adjustments that tend to bring clients, each with the number, the goal and its state
Monitoring and alerts
Scheduled analyses, regression alerts, downtime, and expiring certificates or domains
Keyword research
Search ideas with volume, difficulty, cost per click and intent, ready to track
Google Business Profile
A 0-100 score, local competitors, a geogrid and an alert when a profile detail changes
Google rankings
Where your site shows up in Google for the keywords that matter, and how that changes
Privacy and security
LGPD and GDPR, consent, headers, certificate, DNS and exposed credentials
AI readiness
GEO, AEO, per-page citability, an agent journey and AI citations
Reports
PDF, PIN-protected links, your own brand and the company report, in 3 languages
Search Console and GA4
Google clicks, indexing and sessions in the report, including visits from AI assistants
Semantic SEO
Subjects, pillar pages, entities and passages that stand on their own
Technical SEO
Crawl, status codes, canonical, robots.txt, sitemaps, hreflang and JavaScript rendering
Find out which bots get into your site
Run an analysis and see, bot by bot, what each one receives, the robots.txt rule that applies and the command to check it.
Try it free