robots.txt Fundamentals
Despite being one of the oldest SEO technical standards, robots.txt is frequently misconfigured — blocking critical resources, failing to block genuine waste, or creating conflicts with noindex tags. This guide covers correct advanced configuration for large-site deployments.
robots.txt Syntax Reference
Basic Directives
User-agent: Googlebot
Disallow: /admin/
Allow: /admin/public/
Crawl-delay: 2
User-agent: *
Disallow: /private/User-agent:— Specifies which bot the rules apply to.*matches all bots.Disallow:— Instructs the bot not to crawl matching URLs.Allow:— Overrides a Disallow for specific paths.Crawl-delay:— Google ignores this; set crawl rate in Search Console instead.Sitemap:— Points to your XML sitemap location.
Advanced Configuration: Blocking URL Parameters
The most common large-site robots.txt task is blocking parameter-generated duplicate content:
User-agent: Googlebot
# Block filter/sort parameters
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?page=
# Block internal search
Disallow: /search/
# Block session IDs
Disallow: /*?sessionid=
# Block account/transactional pages
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
# Block admin areas
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpAlways Allow Critical Resources
Never block CSS and JavaScript required to render your pages. Use Allow overrides:
User-agent: Googlebot
Disallow: /assets/admin/
Allow: /assets/js/
Allow: /assets/css/
Allow: /wp-content/themes/
Allow: /wp-content/plugins/WordPress Recommended Configuration
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xmlCommon Mistakes
| Mistake | Impact | Fix |
|---|---|---|
| Disallow: / (entire site blocked) | Complete de-indexation | Remove or narrow directive immediately |
| Blocking /wp-content/ or /assets/ | Google can’t render pages | Allow JS/CSS directories explicitly |
| Blocking pages that also have noindex | noindex tag never read | Use noindex OR robots.txt, not both |
| Missing Sitemap directive | Missed crawl priority signal | Add Sitemap: https://example.com/sitemap.xml |
| Using Crawl-delay for Googlebot | Google ignores it | Set crawl rate in Search Console instead |
robots.txt and AI Crawlers (2026)
Blocking AI training crawlers is increasingly relevant for publishers:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /Note: blocking AI crawlers does not affect standard Google Search ranking. Googlebot and Google-Extended are separate crawlers.
Testing Your robots.txt
Use the Google Search Console robots.txt Tester (Settings → robots.txt) to validate every configuration change before deploying. Test every URL pattern against Googlebot specifically. A single wrong Disallow can block major sections of your site.
Conclusion
robots.txt configuration for large sites is primarily about precision: blocking exactly the URLs that waste crawl budget without touching resources Google needs to render your content. Audit current configuration against server logs, target the largest waste categories with pattern-based Disallow directives, test every change before deploying, and monitor Crawl Stats monthly to confirm improvement.
-
By Guy Sheetrit
- Jul 8, 2026
By Guy Sheetrit
