Friends, do you know what Robots.txt File is? And how to create this file for the website? And how important is this file for SEO?
Friends, if you are blogging, then you must know about this important file, otherwise knowingly or unknowingly this can cause a lot of damage to your website. Some URLs of your website will be indexed in search engine. Most of them rely on this Robots.txt file.
If you make a mistake while cr
eating or updating this important file, your entire website may be removed from the search engine or the content that you do not want to display in the search engine may be indexed.So through today's post I will give you information about what is Robots.txt File in Blogging? And how important is it to create a perfect Robots.txt file? If you read this post carefully, you will understand this term of blogging better.
Robots.txt File ? What is Robots.txt File In Bengali?
Table of Contents
Robots.txt File is a .txt file located in the Root Folder of your website 's web hosting that helps search engine bots to point out that bots will index any URL of your website to show to search engines and will not index any URL TXT File of any website. You can check through https://example.com/robots.txt
The robots.txt file manages the indexing rules of the website in such a way that it can create different rules for different file paths. Each rule prevents crawlers from crawling a specific file path
It is very important to keep this file in the right place because if you keep it separate from the root directory of the website, it may be very difficult for search engine crawlers to find it which is not very accurate.
When a search engine crawlers crawl your website, the first thing they do is target the Robots.txt file in the root directory of the website.
This is why it is very important for every website to have this important file so that all search engines like Google, Bing etc. can give the right direction for crawling and only the content that you have allowed can be publicly visible in the search engine.
What are the benefits of Robots.txt File?
1. Robots.txt File Search engine crawlers are instructed to crawl which part of the website and which part not.
2. With the help of this file you can index and noindex your blog as a category in search engines
3. This helps search engines to quickly understand the indexing structure of your blog
4. With this file you can prevent all search engines from indexing the blog
5. Sometimes there are some content that you do not want to make visible in search engine but if you do not have Robots.txt file then everything is indexed quickly. In this case you can delay the indexing of the blog with this file if you want.
. You can add the URL of the Sitemap to this file to help index the blog's indexable content quickly because crawlers are the first to target the Robots.txt file.
Why Robots.txt File Is Important In SEO?
According to the information given above, now you must have understood what is Robots.txt File on the website? And what are its benefits? Let's find out now how important this type of .txt file is for website SEO? This file is a very important part of the technical SEO of any website.
If you want to make your website SEO friendly like search engine then it is very important to create a Robots.txt file so that crawlers can get the right direction to crawl all the content of your website.
Search engine crawlers have no time to crawl the content of any website if viewed manually but if you submit sitemap to Google search console then in this case crawling is very fast.
But if you do not configure the Robots.txt file for your website then all the content will be indexed in search engines and this will also index all the content which you do not want to be visible in the search results. Creating a Robots.txt File has become very important
Where should the Robots.txt File be located?
If you are a WordPress user then ideally the Robots.txt file should be located in the root directory of your website so that whenever crawlers visit your website they can easily access this file and crawl the content according to its instructions.
But if you keep this important file elsewhere from the root directory of your website, crawlers will not be able to access it and your entire website will be indexed by search engines. https://example.com/robots.txt
Now a text file will open in front of you as you can see in the picture below. This is a Robots.txt file of BengaliTech.info.
And if you see any code like the code given below, then it means that this file has been created for the website but this Robots.txt file has not been optimized according to SEO.
User-agent: *
Crawl-Delay: 10
Guide to Robots.txt File
১. User agent
This command is used to point to search engine bots / crawlers. It may contain one or more User-agents such as *, Googlebot, AdsBot-Google, Bingbot etc. (Here if you use * it means you are all crawlers. Let us understand this with the help of an example
Example 1. To block Googlebot
User-agent: Googlebot
Disallow: /
Example 2. To block Googlebot and Adsbot
User-agent: Googlebot
User-agent: AdsBot-Google
Disallow: /
Example 3. To Allow All Search Crawlers to Crawling
User-agent: *
Allow: /
২. Disallow
This command is used to prevent search engine bots from crawling. You can see the use of this command in the example above.
৩. Allow
This command is used to allow search engine bots to crawl
৪. Sitemap
Here you can specify the URL of your website's Sitemap so that crawlers can easily find all the content you want to crawl.
How to create Robots.txt File for WordPress?
Robots.txt File Do you understand this well by now? Robots.txt is a simple text file. For this reason, it is very easy to create this file for your website. You can use any text editor to create this file like Notepad, Notepad ++ etc.
Open any text editor then create a blank file called robots.txt
Then sign-in to the web server of the hosting through cPanel of your web hosting
After signing in, open the public_html folder or the root folder on your website and upload the text file you created to this folder.
And if you want, you can create Robots.txt File directly in your cPanel (Root Folder) without having to use any Text editor.
To do this you need to go to the Root Folder of your website and then click on the File option at the top left then type the robots.txt in the Display Box that will open and click on the Create New Folder option.
Just click on the OK option and you will see that a text file named robots.txt has been created
Before proceeding, you must set the correct permissions for this file, such as Owner Permissions, Group Permissions, and for Public Permissions, you must right-click on this file. After clicking, select the Change Permissions option. Number Value must be 644)
How to use Robots.txt File?
Now a Robots.txt file has been created for your website but it is still an empty file.
For example if you want to block search engine crawling for your website you just need to add some permissions in Robots.txt
If you want to hide a specific page of your website from crawlers then you can do it easily
Although robots.txt does not automatically improve the SEO of your website, it is important to use this file to control the behavior of search engine crawling.
To modify this file or do any kind of editing, usually open it after visiting your cPanel and any changes you want to make to that file will be immediately effective for your website.
1. Block all bots for websites
If you want to block the crawling of all its crawlers then add this code in Robots.txt File
User-agent: *
Disallow: /
This code actually looks like something in your Robots.txt file which you can see in the picture below
2. Block Crawler's Crawling for a specific folder
If Crawlers can't access a specific folder on your website, add the following code to the file.
User-agent: *
Disallow: /write_folder_name/
If you don't want any crawlers to be able to crawl your website's images folder for search engines, add this code to your original file as you can see in the image below.
Note: - Here / images folder is taken as an example only it is not necessary that you also block your / images folder for search engine bots it may be different in your case the name of the folder for which you want to do this work is this code Add with
3. To block certain search engine bots
If you want to block certain search engine bots like Googlebot or Bingbot on your website then you should use this code
User-agent: robot name
Disallow: /
For example, if you want to block Bingbot, add the above code in your Robots.txt file as you can see in the picture below.
4. Preventing a specific file from crawling
Robots.txt File also allows you to prevent a file from a specific folder from being crawled by search engines. For this you need to add the following code to that file.
User-agent: *
Disallow: /folder_name/file_name.extension
You need to add this code to the Robots.txt file as you can see in the picture below
5. To block an entire folder but to allow a file
There are times when you do not want to read Crawling to a folder but want to allow one of the files in that folder for search engine crawling, then you can do this very easily with the help of Robots.txt file. Will be
User-agent: *
Disallow: /folder_name/
Allow: /folder_name/file_name.extension
For example, if you want to block a folder named “images” on your website but want to access a file named “img_001.png” in that folder then you have to display this code in the original file as you can see in the picture below.
How to create Robots.txt File with RankMath SEO Pl?ugin
WordPress Plugin RankMath SEO lets you create Robots.txt File in the Admin Area of your WordPress Dashboard This is the fastest and easiest way to create this file.
To create an SEO Friendly Robots.txt File through RankMath SEO you first need to install the RankMath plugin in WordPress then click on Rank Math-> General Settings-> Edit robots.txt then you will see an interface like the one below. As you can see in the picture
Below I have given the coding of my own Robots.txt file if you want you can copy and paste it in RankMath and if you want to make it yourself then type it and click on Save Changes option below.
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /comments/feed/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /feed/
Disallow: */feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Allow: /wp-includes/js/
Allow: /wp-includes/css/
Allow: /wp-content/cache/minify/
Allow: /wp-admin/admin-ajax.php
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
User-agent: Bingbot
Allow: /
Site map: https://example.com/sitemap.xml
Site map: https://example.com/post-sitemap.xml Site map:
https://example.com/page-sitemap.xml
My last word
In today's important post I told you what is Robots.txt File? Why should you use it for your website? How to create Robots.txt File for Website? If you read this post carefully, you will understand everything about this
Then you can create a Perfect Robots.txt File for your website. If you have any problems while creating this file, please let us know by commenting in the comment box below. I will definitely try to help you.
Friends, I hope you enjoyed reading this post today. If you like it, don't forget to share this post on your social media and you must comment on how you felt after reading today's post. Thank you so much for reading today's post.