Robots.txt Generator

Create an SEO-friendly robots.txt file for your website, including rules for AI crawlers.

Website URL Sitemap URL isi se banega.

Preset

Allow AI crawlers (ChatGPT, Claude, Perplexity)? Allowing them improves visibility in AI Overviews and ChatGPT search.

Block SEO crawlers (Ahrefs, Semrush)? It saves crawl budget, but also cuts off competitor analysis tools.

Extra disallow paths (ek line mein ek)

Crawl delay (seconds) Keep it to 1-2 on shared hosting. Google ignores this directive.