Test and validate your robots.txt file to ensure search engines and crawlers can access your website correctly. Check rules for Google, Bing, AI bots, and more.
Robots.txt Content
Test Configuration
Test Results
✅ALLOWED
Tested URL:/
Bot:-
Reason:-
Applicable Rules
Rules from User-agent: * (applies to all bots)
Rules specific to selected bot
Robots.txt Analysis
Sitemap URL:-
Crawl-delay:-
Total Rules:-
User Agents Found:-
Line-by-Line Analysis
Test Robots.txtClear AllCopy ResultsDownload ResultsProcessing...Please enter robots.txt contentPlease select a user agentALLOWEDDISALLOWEDNOT SPECIFIEDAllowDisallowSample robots.txt loaded!All cleared!Test completed!Results copied to clipboard!Failed to copy resultsResults downloaded!No results to downloadNo results to copyNot specifiedLineNone foundAll URLsUser-AgentSitemapCrawl-delayCommentEmpty lineMatching rulesecondsNo rules found for this user agentNo rules found for User-agent: *Rules specific toNo rules found for this user agent - access allowed by defaultNo matching Disallow rule found - access allowed by defaultAccess disallowed by rule
⚙️
Test Settings
Display all Allow/Disallow rules that apply to the selected bot
Highlight rules that match the test URL
Display line numbers in the analysis
URL paths are case-sensitive (recommended for most servers)
📋
Quick Reference
User-agent: *
Applies to all bots
Disallow: /
Block entire site
Disallow: /admin/
Block specific directory
Allow: /admin/public/
Allow specific subdirectory
Sitemap: URL
Specify sitemap location
Crawl-delay: 10
Delay between requests (seconds)
🤖 Robots.txt Tips
📋 Basic Syntax
User-agent: Specifies which crawler the rules apply to
Disallow: Tells the crawler not to access specific URLs
Allow: Overrides a Disallow to permit access to specific URLs
Sitemap: Specifies the location of the XML sitemap
Crawl-delay: Sets time between crawler requests
🎯 Pattern Matching
* (wildcard): Matches any sequence of characters
$ (end anchor): Matches the end of the URL
/path/: Matches URLs starting with /path/
/*.pdf$: Matches all PDF files
/dir/*: Matches everything in a directory
⚠️ Best Practices
Always place robots.txt in the root directory
More specific rules take precedence over general ones
Allow rules override Disallow rules of the same length
Robots.txt is publicly accessible - do not use for security
Test your robots.txt before deploying to production
Use Google Search Console to validate your robots.txt