To help search engines index your blog correctly, you need to make a proper Robots txt file for WordPress. Let’s see how to create it and how to fill it.
Contents
What does Robots.txt give?
It is needed for search engines to correctly index a web resource. The content of the file “tells” the search robot which pages to show in the search and which to hide. This allows you to manage the content in the search results.
Where is Robots located?
This regular test file is located in the root directory of the site. It can be obtained from
https://site.ru/robots.txt
Can’t find this file
If the contents of the file are displayed at the specified address, but it is not on the server, then it is created virtually. The search engine doesn’t care. The main thing is that it is available.
What does it consist of
4 main directives:
- User-agent – rules for search robots.
- Disallow – Denies access.
- Allow – allows.
- Sitemap – The full URL of the XML map.
Also see: How to Remove WordPress Archives
Correct robots.txt for WordPress
There are many options. The instructions are different for each site.
Here is an example of the correct Robots, which takes into account all sections of the site. Let’s briefly analyze the directives.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-content/cache Disallow: /wp-json/ Disallow: /wp-login.php Disallow: /wp-register.php Disallow: /xmlrpc.php Disallow: /license.txt Disallow: /readme.html Disallow: /trackback/ Disallow: /comments/feed/ Disallow: /*?replytocom Disallow: */feed Disallow: */rss Disallow: /author/ Disallow: /? Disallow: /*? Disallow: /?s= Disallow: *&s= Disallow: /search Disallow: *?attachment_id= Allow: /*.css Allow: /*.js Allow: /wp-content/uploads/ Allow: /wp-content/themes/ Allow: /wp-content/plugins/ Sitemap: https://site.ru/sitemap_index.xml
The first line indicates that the resource is available to all search robots (crawlers).
The Disallow directives prohibit the search for service directories and files, cached pages, authorization and registration sections, RSS (Feed), author, search, and attachment pages.
Allow allows you to add scripts, styles, downloads, themes and plugins to the index.
The last one is the address of the XML map.
How to create robots.txt for a website
Let’s consider several methods.
Manually
This can be done, for example, in Notepad (if the local server) or through an FTP client (on the hosting).
This can also be achieved with WP plugins. Let’s analyze the best of them.
ClearfyPro
Clearfy Pro creates a virtual file. For this:
- Go to the Clearfy Pro admin menu.
- On the SEO tab, enable the Generate correct robots.txt option.
- Fill in the contents of the file.
- Save your changes.