robots.txt Fundamentals
Despite being one of the oldest SEO technical standards, robots.txt is frequently misconfigured — blocking critical resources, failing to block genuine waste, or creating conflicts with noindex tags. This guide covers correct advanced configuration for large-site deployments.
robots.txt Syntax Reference
Basic Directives
User-agent: Googlebot
Disallow: /admin/
Allow: /admin/public/
Crawl-delay: 2
User-agent: *
Disallow: /private/User-agent:— Specifies which bot the following rules apply to.*matches all bots.Disallow:— Instructs the bot not to crawl matching URLs. Empty Disallow (Disallow:) means crawl everything.Allow:— Overrides a Disallow for specific paths (Google/Bing support; not universal). More specific rules take precedence.Crawl-delay:— Seconds to wait between requests. Google ignores this; set crawl rate in Search Console instead.Sitemap:— Points to your XML sitemap location. Can appear multiple times for multiple sitemaps.
Rule Precedence
When multiple rules match a URL, the most specific rule wins. For equal specificity, Allow takes precedence over Disallow (in Google’s implementation). If no rule matches, crawling is allowed by default.
Advanced Configuration for Large Sites
Blocking URL Parameters
The most common large-site robots.txt task: blocking parameter-generated duplicate content from faceted navigation, search, and session IDs:
User-agent: Googlebot
# Block all filter/sort parameters
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?brand=
Disallow: /*?price=
Disallow: /*?page=
# Block internal search
Disallow: /search/
Disallow: /s/
# Block session IDs
Disallow: /*?sessionid=
Disallow: /*?PHPSESSID=
# Block account/transactional pages
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /login/
Disallow: /register/
Disallow: /wishlist/
# Block admin areas
Disallow: /wp-admin/
Disallow: /wp-login.php
Allow: /wp-admin/admin-ajax.phpAllowing Critical Resources
A critical robots.txt mistake is blocking JavaScript and CSS files that Google needs to render and understand your pages. Google explicitly recommends allowing all CSS and JavaScript. If your assets are in directories that would be caught by a broad Disallow rule, use Allow to override:
User-agent: Googlebot
Disallow: /assets/admin/
Allow: /assets/js/
Allow: /assets/css/
Allow: /wp-content/themes/
Allow: /wp-content/plugins/Managing Multiple Subdomain Crawlers
Each subdomain needs its own robots.txt file. A rule in www.example.com/robots.txt does not apply to blog.example.com/robots.txt. For large enterprises with many subdomains, audit each subdomain’s robots.txt separately — legacy subdomains often have misconfigured or missing robots.txt files.
robots.txt for Specific CMS Platforms
WordPress
Standard WordPress blocking requirements:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xmlShopify
Shopify auto-generates robots.txt but allows customization via the robots.txt.liquid template. Default Shopify robots.txt blocks: /admin/, /cart/, /orders/, /checkouts/, /checkout/, /carts/, /account/, /collections/*sort_by*, and other internal pages. Customize to add parameter blocking specific to your theme’s faceted navigation implementation.
Magento
Magento generates significant URL parameter waste. Essential blocks: Disallow: /*?SID= (session IDs), Disallow: /*?___store= (store switcher), Disallow: /catalog/product_compare/, Disallow: /tag/, all filter parameter combinations.
Testing and Validation
Google Search Console robots.txt Tester
The official testing tool. Access: Search Console → Settings → robots.txt. Features: live fetch of your current robots.txt, URL testing against current rules, syntax validation. Test every pattern change before deploying — a missed regex can block important sections of your site.
Manual Testing
Use Google’s robots.txt testing library (available as a Python package: pip install robotexclusionrulesparser) for programmatic validation of large rule sets:
from robotexclusionrulesparser import RobotExclusionRulesParser
r = RobotExclusionRulesParser()
r.fetch('https://www.example.com/robots.txt')
print(r.is_allowed('Googlebot', 'https://www.example.com/search/?q=test'))
# False — correctly blockedScreaming Frog for Audit
In Screaming Frog, enable the “Respect robots.txt” option to see which URLs your rules would block, and disable it to audit URLs that Googlebot can’t currently crawl. The Coverage report shows blocked vs. allowed pages across your full crawl.
Common robots.txt Mistakes
| Mistake | Impact | Fix |
|---|---|---|
| Disallow: / (entire site blocked) | Complete de-indexation | Immediate fix; remove or narrow directive |
| Blocking /wp-content/ or /assets/ | Google can’t render pages; ranking drops | Allow JS/CSS directories explicitly |
| Blocking pages that have noindex (conflict) | noindex tag never read; pages may index | Use noindex OR robots.txt, not both |
| Missing Sitemap directive | Missed crawl priority signal | Add Sitemap: https://example.com/sitemap.xml |
| Blocking mobile subdomain incorrectly | Mobile pages not indexed | Audit m. subdomain robots.txt separately |
| Using Crawl-delay for Googlebot | Google ignores it; false sense of control | Set crawl rate in Search Console instead |
robots.txt and AI Crawlers
As of 2026, blocking AI training crawlers via robots.txt is increasingly relevant for publishers. Major AI crawler user agents: GPTBot (OpenAI), CCBot (Common Crawl, used for many AI training datasets), Google-Extended (Google’s AI training crawler, separate from Googlebot), anthropic-ai (Anthropic), PerplexityBot. Example block:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /Note: blocking AI crawlers does not affect standard search ranking. Googlebot (for search) and Google-Extended (for AI training) are separate crawlers — blocking Google-Extended has no impact on Google Search indexing.
Conclusion
robots.txt configuration for large sites is primarily about precision: blocking exactly the URLs that waste crawl budget without touching the resources Google needs to correctly render and index your content. Audit your current robots.txt against server logs to identify what Googlebot is crawling, target the largest waste categories with pattern-based Disallow directives, test every change in Search Console before deploying, and monitor Crawl Stats monthly to confirm improvement. The goal is a configuration where Googlebot’s available crawl capacity is directed entirely toward URLs that contribute to your search performance.
-
By Guy Sheetrit
- Jul 8, 2026
By Guy Sheetrit
