Validate, test, and build robots.txt files — following Google's specification (RFC 9309).
Or paste robots.txt content directly below ↓
Groups Found
2
Sitemaps
1
📄 https://example.com/sitemap.xml
Warnings & Notes
Line 6: ⚠️ Blocking ALL crawling for "gptbot". Does NOT prevent indexing — use noindex.
Key spec notes: robots.txt manages crawler access, NOT indexing. Use noindex to prevent indexing. Path matching is case-sensitive. Longest path wins; ties go to Allow. Google ignores crawl-delay. Max 500 KiB, UTF-8, served at root.