What Should I Put in My Robots.txt File?

Author: Guy Sheetrit Updated Date: March 7, 2026 Category: Advanced SEO Techniques , SEO Basics

You can make some great optimizations to your robots.txt file if you just knew how.

Let’s go over some advice on this sensitive subject. What is it? How you can benefit from keeping it up to date?

The importance of this file should never be underestimated when it comes to good SEO practice. The file essentially allows you to speak to different search engines and inform them as to what sections of the website should be indexed. This provides specific directions to search bots and can determine a large part of your SEO success if well managed. For a deeper dive, explore our guide on Voice Search SEO.

For more technical SEO insights, explore our Core Web Vitals checklist and SEO fundamentals guide.

Table of Contents

Toggle

Do I really need it?

This is a common and fair enough question. The absence of a robots.txt file is the thing to consider here. If you don’t have one it won’t actually stop your site from being indexed by search bots. However, you will lose out on a lot of potential SEO power.

I don’t have one. How can I make it?

It’s simple enough. You’ll usually find a robots.txt file hanging out in the root folder of a site. You will want to make sure that you are connected to your site via a cPanel manager or FTP client.

It’s a simple matter of opening or creating it with Notepad. There’s a range of YouTube tutorials that delve deeper into the subject.

>> Free Robot.txt Generator <<

So what should I put in it?

Let’s get down to the nitty-gritty and run through some ideas and suggestions as to the content of your file.

We’ll start with robots exclusion protocol tags or REP. When a REP tag is applied to a URL it will steer clear of particular indexer tasks. Each search engine views and interprets REP differently.

Google will wipe out a URL-only listing on their SERPs if a single resource is amended with a no-index tag. Bing is different in that it will often list these references on its SERPs.

The point here is that REP tags can be put in the meta elements of HTML content in addition to HTTP headers on any website object. The general view is that robot tags will overrule any conflicting directives that are found in meta elements.

Microformats

If you put an indexer directive as a microformat it will work for you by overruling any page settings for certain elements of HTML.

An example of this is when a page’s robots tag shows “follow”. The rel-nofollow directive of a different element or link will win.

While your robots.txt won’t have indexer directives, it is possible to set these for a group of URLs so that they apply only to the robots tag.

This will require some programming skills from you as well as a sound sense of web servers and HTTP protocol.

Pattern matching

You’ll find a similarity between both Bing and Google in that they will both honor regular expressions – two, specifically. You can use these to identify any pages or folders in your site that you want to be excluded from the ranking. These characters are the dollar and the asterisk ($ and *).

The dollar matches the end of the URL and the asterisk acts as a wild card, which will show any sequence of characters.

General information

It’s important to keep in mind that the robots.txt file on your site will be viewable by the public. This means that anyone analyzing your site will be able to see which areas the owner has blocked the search engines from viewing.

This means that you´ll have to apply other techniques if you have private information on your website that you don’t want the public to know about. You might want to use more suitable file protection methods such as passwords to prevent visitors from accessing that information.

Some rules to keep in mind

Meta robots that have the parameters “noindex, follow” are best used as a method to restrict or direct the indexing of your site via search bots and crawlers.

You’ll also want to keep in mind that if a crawler is malicious in nature it very likely won’t refer to your robots file in the first place. This means that you can’t reliably make use of the robots.txt file as any kind of security measure. This is a common pitfall as it isn’t quite shouted from the rooftops in some cases.

You can also only have one disallow: line for every URL that you are using – no extras.

It’s also good to remember that Google and Bing both accept the dollar and asterisk expression characters. This is a key factor in making the best use of pattern exclusion.

Make sure of proofreading. The robots text file is case-sensitive! Don’t get caught out and find yourself scratching your head for an hour because your finger slipped on the shift key.

Scary things can happen if you don’t maintain your robots file well.

There have been cases of well-kept sites with great backlinks and organic content, but just no SEO luck for unfathomable reasons. Many just suffered because a single disallow forward slash was included, killing their SEO by telling crawler bots not to index any of their pages at all!

Yes and no

A brief summary of what is good and what isn’t when it comes to robots.txt.

Do:

Look at the directories on your site. You’re likely to have some areas that you want to block using the txt file.
Stop indexing areas of your site with legitimate duplicate content such as a printable page version or a repeated recipe or manual.
Be sure that the search engines aren’t being blocked from indexing your main site.
Check around for specific files on your site that might be best blocked. These can include phone numbers or email addresses.

Don’t:

Use any comments in the txt file. This will cause all kinds of problems as it’s very sensitive.
Put all the details of your files in the txt file. As we’ve established, this is public and could defeat the point of masking some areas of your site!
Don’t add “/allow” to the file as it won’t have any effect

In conclusion

There’s a lot of use for robots.txt files. It’s almost always to your benefit to have and maintain one. However, you do need to keep in mind how sensitive it is in nature.

With some proper diligence and a sound understanding of its limitations, it can help you guide and direct search bots and crawlers to the specific content you want to be ranked. You can also use it to enhance privacy by hiding content from the public.

It isn’t the beginning and end of SEO and your problems, to be honest. But with careful use, it will keep you safe and on track toward that page one prominence.

For a deeper dive, explore our guide on Keyword Research.

Frequently Asked Questions

Q: What is this guide about?

This comprehensive guide provides strategies and best practices for achieving success. Following these approaches can help improve your results and competitive advantage.

Q: How long does it take to see results?

Results vary. Most strategies require 3-6 months before significant improvements. Ongoing optimization and consistency are essential for sustainable success.

Q: Do I need professional help?

While basic implementation can be done independently, professional guidance often accelerates results and helps avoid costly mistakes.

Q: What are the most important factors for success?

Key factors include thorough research, consistent execution, quality over quantity, regular performance monitoring, and adapting to industry changes.

Q: How do I measure success?

Track KPIs like traffic, conversions, revenue, and engagement rates. Regular analysis helps identify areas for improvement.

Q: What channels should I focus on?

Most businesses benefit from SEO, content marketing, social media, and paid advertising. Start where your target audience is most active.

The Evolution of Digital Marketing Strategy

Digital marketing has transformed dramatically over the past decade, evolving from simple banner advertisements to sophisticated, data-driven strategies that leverage artificial intelligence and machine learning. Understanding this evolution provides context for developing effective modern marketing strategies that resonate with today’s consumers.

Modern digital marketing requires integrated approaches combining multiple channels into cohesive customer experiences. The most successful businesses recognize that consumers interact with brands through complex journeys spanning multiple devices and platforms.

Content Marketing Best Practices

Content remains the foundation of successful digital marketing, serving as the primary mechanism for attracting organic traffic, building brand authority, and engaging target audiences. Effective content addresses specific search queries while providing genuine value to readers through comprehensive answers and actionable insights. For a deeper dive, explore our guide on Zero Search Volume Keywords.

Data-Driven Marketing Decisions

Modern marketing success depends on sophisticated analytics enabling data-driven decisions. Understanding which metrics connect to business outcomes allows continuous optimization and improved return on investment through testing and iterative improvement.

Building Brand Authority

Establishing thought leadership provides significant competitive advantages including increased brand awareness and customer trust. Effective thought leadership addresses emerging trends, challenges conventional wisdom, and provides actionable guidance. For a deeper dive, explore our guide on SEO Practices.

Maximizing Marketing ROI

Proving marketing ROI requires clear objectives, sophisticated tracking, and continuous optimization. The most successful marketing organizations treat marketing as an investment delivering measurable returns through continuous testing.

Learn More: Home

Technical SEO in 2025: The Foundation That Determines Your Ceiling

Technical SEO is the least glamorous discipline in the search marketing stack — and the most consequential. You can have the best content, the most authoritative backlinks, and the strongest brand signals in your niche, but if Googlebot can’t efficiently crawl and index your site, or if your Core Web Vitals scores are in the bottom quartile, those assets are being systematically undervalued.

The technical SEO landscape in 2025 has expanded significantly. Where technical SEO once meant XML sitemaps and robots.txt management, it now encompasses JavaScript rendering, Core Web Vitals, structured data, site architecture, and increasingly, AI-readiness signals like entity markup and knowledge graph integration.

Core Web Vitals: The Performance Metrics That Directly Impact Rankings

Google’s Core Web Vitals became an official ranking signal in 2021 and have been progressively weighted more heavily since. The three metrics and what they actually measure:

Largest Contentful Paint (LCP): How quickly does the main content of a page load? Target: under 2.5 seconds. The most common LCP killers are unoptimized hero images, render-blocking JavaScript, and slow server response times. Fix priority: compress and convert images to WebP, implement lazy loading for below-fold images, and enable browser caching.
Interaction to Next Paint (INP): How quickly does the page respond to user interactions (clicks, taps, keyboard input)? This replaced First Input Delay in March 2024. Target: under 200ms. INP problems are almost always JavaScript-related — heavy third-party scripts, main thread blocking, or inefficient event handlers.
Cumulative Layout Shift (CLS): How much does the page layout shift as it loads? Target: under 0.1. Common causes are images without defined dimensions, dynamically injected content (ads, banners, cookie notices), and web fonts loading after text is rendered.

Google’s PageSpeed Insights provides field data (real user measurements from Chrome users) that is the actual data used in rankings — not the lab data from manual tests. Optimize for field data improvement, not just lab score improvement.

Crawl Budget Optimization

Crawl budget — how many pages Googlebot crawls on your site per day — is finite and valuable. Wasting it on low-value pages means high-value pages get crawled less frequently. Crawl budget optimization is critical for sites with 10,000+ pages.

Pages that consume crawl budget without adding value:

Faceted navigation duplicates (color/size/price filters creating unique URLs)
Paginated archives beyond page 2-3
Tag and author archive pages on CMS platforms
Session ID URLs and UTM parameter variations
Staging or development URLs accidentally accessible to crawlers

Management approach: use robots.txt to block parameter-based duplication, implement canonical tags on near-duplicate pages, and configure the URL Parameter tool in Google Search Console to indicate which parameters change page content versus just tracking parameters.

JavaScript SEO: The Invisible Technical Barrier

Over 70% of websites now use JavaScript frameworks (React, Vue, Angular, Next.js) for their front-end. JavaScript SEO is the discipline of ensuring these frameworks don’t create rendering barriers for Googlebot.

Googlebot renders JavaScript, but with significant caveats: rendering happens in a second-wave queue (hours to days after initial crawl), JavaScript errors can prevent content from rendering entirely, and complex client-side routing can prevent proper canonicalization.

The safest architecture for SEO: Server-Side Rendering (SSR) or Static Site Generation (SSG) for all content that needs to rank. Dynamic content (personalization, user-specific data) can be client-side. This hybrid approach gives you the performance and SEO benefits of server rendering without sacrificing the interactivity of modern JavaScript frameworks.

By Guy Sheetrit
Sep 2, 2016