{"id":321,"date":"2013-08-13T19:47:03","date_gmt":"2013-08-13T18:47:03","guid":{"rendered":"https:\/\/metricspot.com\/blog\/?p=321"},"modified":"2016-12-14T17:52:10","modified_gmt":"2016-12-14T16:52:10","slug":"robots-txt-file","status":"publish","type":"post","link":"https:\/\/metricspot.com\/blog\/robots-txt-file\/","title":{"rendered":"The ROBOTS.TXT File"},"content":{"rendered":"

The ROBOTS.TXT file is a text document that serves to establish crawling guidelines<\/strong> for Bots to follow when exploring your Website.<\/p>\n

Bots (also called Spiders or Crawlers) are used by search engines<\/strong> to access your Website and index the content (text, images, files…) of its pages.<\/p>\n

 <\/p>\n

WHAT IS IT FOR?<\/h3>\n

With the ROBOTS.TXT file we can discourage Bots to access a certain folder<\/strong> of our Website. We can also avoid a certain Bot from crawling our Website<\/strong> or limit its crawling frequency. Some of the reasons we might want to do this are:<\/p>\n

– Avoid duplicate content<\/strong>. This is the most important reason because, if we do this, we will rank higher in search engines, thus increasing our traffic.<\/p>\n

– Reduce server overload<\/strong> due to excess of search engine petitions that could saturate it.<\/p>\n

– Avoid indexing of certain pages<\/strong> that you want to be accessible to users but not indexed in Google due to privacy reasons.<\/p>\n

We can also add a sitemap of our Website<\/strong> or SITEMAP.XML file to indicate Bots the URLs of all the pages of our site.<\/p>\n

 <\/p>\n

WHAT IS IT NOT FOR?<\/h3>\n

As we have said before, the ROBOTS.TXT file establishes crawling guidelines and bots may not honor your rules<\/strong>, especially the so-called \u00abbad bots\u00bb, whose only purpose is to crawl your Website searching for e-mails, private data or vulnerabilities.<\/p>\n

If you have sensitive information on your Website and you don’t want bots to crawl it, you should use other security means<\/strong> to protect it.<\/p>\n

Also, with the ROBOTS.TXT file you can’t protect your Website from Hackers<\/strong> who are using \u00abbrute force\u00bb attacks.<\/p>\n

 <\/p>\n

HOW TO CREATE A ROBOTS.TXT FILE<\/h3>\n

You can use one of the following Online tools<\/a> to create the ROBOTS.TXT file, although we highly recommend you follow Google’s instructions<\/strong> in order to create it manually<\/a>. You can also read this Wikipedia article<\/strong> about Robots exclusion standards<\/a>.<\/p>\n

It has to be located in the root foder<\/strong> of your Website, the same as the FAVICON and the SITEMAP.<\/p>\n

 <\/p>\n

ROBOTS.TXT FILE EXAMPLE<\/h3>\n

This MetricSpot’s ROBOTS.TXT<\/a> file:<\/p>\n

User-agent: *
\nDisallow: \/new\/
\nDisallow: \/tos\/
\nDisallow: \/items\/
\nDisallow: \/no\/
\nDisallow: \/condiciones-de-uso\/
\nDisallow: \/blog\/cat\/
\nDisallow: \/blog\/tag\/
\nDisallow: \/blog\/wp-admin\/
\nDisallow: \/blog\/wp-includes\/
\nDisallow: \/blog\/wp-content\/plugins\/
\nDisallow: \/blog\/wp-content\/themes\/
\nDisallow: \/blog\/feed\/
\nDisallow: \/api\/www.metricspot.com
\nDisallow: \/*.js$
\nDisallow: \/*.css$
\nSitemap: https:\/\/metricspot.com\/sitemap.xml<\/div>\n

 <\/p>\n

The User-agent: *<\/strong> line indicates the following rules apply to ALL Bots.<\/p>\n

The following 5 lines block specific pages or folders using the Disallow:<\/strong> rule, followed by the URI to be blocked.<\/p>\n

For security reasons<\/strong>, we have blocked the \u00abTerms of Service\u00bb page because it includes information we don’t want to be indexed.<\/p>\n

In order to avoid duplicate content issues<\/strong> we have blocked the \/new\/<\/strong> and \/items\/<\/strong> folders, which are used by our App to create temporary content. We have also blocked the \/blog\/cat\/<\/strong> and \/blog\/tag\/<\/strong> folders used by our Blog to host categories and tags.<\/p>\n

The Disallow: \/*.js$<\/strong> and Disallow: \/*.css$<\/strong> rules block crawling of all JavaScript and CSS files to avoid server overload.<\/p>\n

Last, with the Sitemap: https:\/\/metricspot.com\/sitemap.xml<\/strong> line we show Bots where to find our Sitemap.<\/p>\n","protected":false},"excerpt":{"rendered":"The ROBOTS.TXT file is a text document that serves to establish crawling guidelines for Bots to follow when exploring your Website.","protected":false},"author":2,"featured_media":2196,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[39],"tags":[40,19,41],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/posts\/321"}],"collection":[{"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/comments?post=321"}],"version-history":[{"count":0,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/posts\/321\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/media\/2196"}],"wp:attachment":[{"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/media?parent=321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/categories?post=321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metricspot.com\/blog\/wp-json\/wp\/v2\/tags?post=321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}