Decide how AI sees your site.
Generate a robots.txt that allows, blocks, or audits 50+ AI crawlers — plus a matching llms.txt. Free, open source, runs entirely in your browser.
Covers 50+ AI crawlers incl. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider.
Quick presets
AI crawlers (82)
Options
Advanced: per-agent crawl-delay
Block some agents first, then set a crawl-delay (in seconds) for them here.
Note: only some crawlers honor Crawl-delay. Malicious bots ignore it entirely.
Your robots.txt
# robots.txt generated by OpenRobots — https://openrobots.vercel.app # Decide how AI sees your site. User-agent: * Allow: /
5 lines · 124 characters · generated locally, nothing leaves your browser
Nothing is blocked — every crawler can access your site.
This permissive file lets all crawlers in. Pick agents above or choose a preset to start blocking.
Does this save you time? ☕ Buy us a coffee
OpenRobots is free — if it saved you time, ☕ Buy us a coffee
Why OpenRobots
The AI crawl decision is new
Cloudflare now blocks AI crawlers by default on new domains. Whether you opt in or out, every site owner has to make this decision — OpenRobots makes it a two-minute one.
Incumbents want $29–$299/mo
Existing crawler-management tools put their generator behind a signup wall and a subscription. OpenRobots is free and open source — no account, no upsell.
Your rules, your server
Copy-paste a plain robots.txt into your site root. No plugin, no proxy, no vendor lock-in — just a text file you fully control.
Frequently asked questions
▸Should I block GPTBot and other AI crawlers?
It depends on your goals. Blocking keeps your content out of AI training corpora and AI answers; allowing can mean broader reach when AI assistants cite and link to you. There is no universally right answer — but there should be a deliberate answer. OpenRobots lets you allow, block, or take a middle path (like allowing assistant bots while blocking training bots) in a few clicks.
▸What's the difference between training bots, assistant bots and search bots?
Training bots (like GPTBot and ClaudeBot) collect content to train AI models. Assistant bots fetch a page live when a user directly asks about it. Search bots index the web for AI-powered search results. A popular middle path is "allow assistants, block training": your pages stay reachable for real-time answers and citations, but don't feed future model training. The Generator's presets include exactly this option.
▸Does blocking AI crawlers hurt my Google SEO?
No. Blocking Google-Extended only stops your content from being used to train Gemini and power AI answers — it does not affect Google Search indexing or ranking. Googlebot is a separate crawler, and OpenRobots never touches it unless you explicitly say so.
▸What is llms.txt?
llms.txt is an emerging standard: a markdown file at your site root that gives AI models a clean, curated overview of your site — its name, description, and links to your most important pages. It helps LLMs understand and cite your content accurately. OpenRobots includes a simple llms.txt generator alongside the robots.txt tools.
▸Do bots actually respect robots.txt?
Most large, established AI crawlers do — and our crawler list shows which ones are known to respect it and which don't. But robots.txt is a request, not a firewall: some smaller or poorly-behaved bots ignore it. For most sites, robots.txt is the right, standards-based tool; if you need hard guarantees, combine it with server-level or firewall-level rules.
▸Is my data sent anywhere?
No. OpenRobots is a static site with no backend. Everything — the crawler list, the generator, the audit, the llms.txt builder — runs entirely in your browser. Nothing you toggle or paste ever leaves your device, and there are no accounts, cookies, or tracking.
Keep OpenRobots free
OpenRobots is free, open source, and shows no ads. If it saved you time wrangling AI crawlers, buying us a coffee keeps the crawler database updated and the project moving.
Open source — star or fork it on GitHub