How to Create Robots.Txt in WordPress Website or Blog for SEO?

Robots file is necessary for your website or blog because it will help you to boost your SEO. With the help of Robots.txt file, you can give the robots an order which web page should crawl or which should not. So you should know about How to Create Robots.txt file in WordPress.

Before create or submit a robots.txt file, you should know about What is Robots.txt file?

What is Robots.txt File?

The robots.txt file is used to instruct the search engines about how to index or crawl your web pages present on the website. With the help of robots.txt, you can decide which web page should index or crawl for your website.

If you do not want to index or crawl any web page then you can prevent search engine form indexing and crawling.

Robots.txt file is present in the root directory of your Website or blog.

Basic Format of the robots.txt file.

User-agent: ( User-Agent Name )

Disallow: ( URL String not to crawl and index)

User-agent: ( User-Agent Name )

Allow: ( URL String to crawl and Index )

Sitemap: ( Your website Sitemap URL )

Example: This is the ideal robots.txt file for your WordPress website or blog.

User-agent: *

Allow: /wp-content/uploads

Disallow: /wp-admin/

Allow: /

Sitemap: https://www.example.com/sitemap_index.xml

Why You Should Use Robots.txt file?

  • You can give permission to search engines for crawling and indexing of any web page.
  • If you do not want to crawl or index any page then you can do with the help of robots file.
  • You can block unnecessary pages like admin, plugins, themes pages.
  • Search bots have quota then you have to need a robots.txt file.

How to Create Robots.Txt File in WordPress for SEO Purpose?

So you can add robots.txt file in WordPress with the help of two methods.

  • Create Robots.txt file with the help of Plugin.
  • You can add Robots.Txt file with the help of cPanel.

First Method:

How to Create Robots.Txt File in WordPress using Yoast SEO Plugin?

Step 1: Go to the WordPress admin dashboard and Install the Yoast SEO plugin.

Yoast-SEO

Step 2: Go to Yoast SEO and click on Tools.

Tools

Step 3: In tools, there are three tools available in Tools.

  • Import and Export
  • File Editor
  • Bulk Editor
  • Text Link Counter
Tools Yoast SEO

So go to File Editor. You can edit robots.txt file and .htaccess file in File Editor.

Step 4: Click on Create Robots.Txt File.

By default, you will see this robots.txt file in Yoast SEO.

User-agent: *

Disallow:

So delete this file and paste your robots.txt file in robots.txt section.

Step 5: Put your robots.txt in Yoast SEO. Click on Save Changes to robots.txt.

Save Changes to Robots File

So here your robots.txt file is created and added to your WordPress website or blog.

This is the first method for How to Create Robots.Txt file in WordPress using a Yoast SEO plugin.

Second Method:

How to Add or Edit Robots.Txt File in WordPress using cPanel?

Step 1: Open a Notepad and Put the robots file instructions in Notepad. Save this notepad as robots.txt in your computer folder.

robots.txt file in Notepad

Step 2: Go to your WordPress website cPanel. Click on File Manager and open public_html file.

Step 3: Upload the robots.txt file in public_html. Click on Upload. Your robots.txt file will upload by choosing Select File.

robots.txt file

Step 4: So your file will upload in WordPress and robots.txt file successfully added to your WordPress website or blog successfully. You can see robots.txt file in public_html.

Robots.Txt in cPanel

If you want to edit your robots file then select your robots.txt file and click on Edit. So you can do any changes after edit your robots.txt file.

This is the complete guide about How to Add Robots.Txt file in WordPress using cPanel.

How to Test Your Robots.Txt File?

There are many robots.txt testing tools but we recommend to use the Google Search Console tool for robots.txt. So you should test your robots.txt with the help of robots.txt tester tool.

Step 1: Open your Google Search Console by login through Gmail.

Step 2: Click on Go to Old Version. It will redirect you to the old version of the Google Search Console tool.

Go to the old version

Step 3: In Crawl, you will see robots.txt tester. Go to robots.txt tester and your robots.txt file will show in this section.

robots.txt tester

So with the help of this tool, you can test or validate your robots.txt file. It will give you errors and warnings if exists.

Step 4: If you want to test the archives then put archives in the testing tool.

Test
  • If it has blocked then it will show in red.
  • Otherwise status has shown in the green format if it has not blocked.
Test Status

So this is the complete guide about how to test your Robots.Txt file with the help of robots tester tool.

Note: If you want to see your website robots.txt file then put robots.txt behind your website URL.

https://www.example.com/robots.txt

Otherwise, You can see your robots.txt file in robots.txt tester tool which is present in the Crawl section. Click on See Live robots.txt in the robots.txt tester and you can see your robots.txt file.

So this is the complete guide about How to Create Robots.Txt in WordPress and how to test the robots.txt file through the testing tool.

Conclusion:

With the help of this guide, you can easily create robot file for your WordPress website or blog. You can easily test your website or blog robots.txt file.

If you have not created robots file for your website then you should create robots file for indexing unnecessary pages of your WordPress website or blog.

It will help you to index your desired web pages and you can get more help through robots file for SEO purposes.

If you have liked this article then share it on your social media profiles and like our Facebook Page.

Read More:

How to Create Sitemap in WordPress and Submit to Google Search Console?

6 Best Sitemap Plugins for WordPress Website or Blog in 2019

Last Modified on by VISAL TYAGI

Share this post

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on pinterest
Pinterest

15 thoughts on “How to Create Robots.Txt in WordPress Website or Blog for SEO?”

  1. Hi there it’s me, I am also visiting this website daily, this web site is truly good and the viewers are truly sharing fastidious thoughts.

    Reply
  2. Hi everyone, it’s my first go to see at this site, and post is really fruitful for me,
    keep up posting these articles or reviews.

    Reply
  3. I have read so many articles on the topic of the blogger lovers
    however this piece of writing is in fact a fastidious paragraph, keep it
    up.

    Reply
  4. Excellent way of describing, and good piece of writing to
    obtain information on the toppic of my presentation focus, which i am going to present in school.

    Reply
  5. I’m curious to find out what Ƅlog system you’rе using?
    I’m experiencing some small security issues with my latest website and Ӏ’d like to fond something more seⅽure.
    Do yοu have any solutions?

    Reply
  6. Hello to every , because I am truly eager of reading this webpage’s post to be updated on a regular basis. It includes nice information.

    Reply
  7. Valuable information. Lucky me I found your web site by accident, and I’m shocked why this accident didn’t happened earlier! I bookmarked it.https://www.espnplusactivate.com/

    Reply
  8. At this time it appears like BlogEngine is the best blogging platform out
    there right now. (from what I’ve read) Is that what you’re using on your blog?

    Reply

Leave a Comment

Receive The Latest Traffic Tips

30 Proven Ways
To
Increase Website Traffic

Discover the Secrets – 15000 Words

2 Shares
Share
Tweet
Pin2
Share
Share