Free resource / Updated May 28, 2026

AI Crawler User-Agent List and Robots.txt Examples

A quick reference for publishers, SaaS teams, and SEO operators deciding how AI crawlers should access public content.

Common crawler user agents

GPTBotChatGPT-UserOAI-SearchBotClaudeBotClaude-UserPerplexityBotGoogle-ExtendedGooglebotBingbotApplebotBytespiderCCBot

Allow search crawlers, block broad AI training crawlers

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Block all bots from private paths

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /account/
Disallow: /checkout/

Allow public content discovery

User-agent: *
Allow: /

Sitemap: https://toolbulletin.com/sitemap.xml

Important caveat

User-agent names and crawler behavior change. Treat this as a starting checklist, then verify behavior with server logs, crawler monitoring, and your own analytics.

Monitor what actually crawls.

ToolBulletin recommends starting with crawl visibility before changing robots.txt rules site-wide.

Compare crawl monitoring tools