Robots.txt Generator

Generate a clean, correct robots.txt file for your website — control which bots can crawl which pages, block scrapers, and include your sitemap.

Your Website Base URL

Sitemap URL (optional)

Allowed Paths (one per line)

Blocked Paths (one per line)

Block Aggressive Bots (select to block)

Generated robots.txt

Download

# Allow all main search engines
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/
Disallow: /?s=
Crawl-delay: 10

# Block aggressive/unwanted bots
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

Need a full Technical SEO audit?

Our team audits crawlability, indexation, and technical SEO for complete visibility.

Get Technical Audit →