Reduce Magento resources using meta tag robots

Reduce the amount of resources your Magento webshop uses by setting meta tags robots. Google and other search engines crawl your website. Every single page needs to be generated which will cost a lot of resources. Pages like Layered Navigation don’t need to be crawled.

In this post I’ll explain you how to reduce the consumption of resources for the Layered Navigation pages in Magento. This will also increase your pagerank since you tell the search engines like Google which pages are irrelevant. 
By default a searchengine will index the pages on your website and follow the links on it. The syntax for tag is:

<meta name="robots" value="index,follow" />

Add noindex,follow in your meta tag robots

All you need to do is to change the value of meta tag robots on the pages where a Layered Navigation is used.

• Open or create local.xml in your Magento theme.

  <meta name="robots" value="index,follow" />

  If you haven’t used a local.xml for your theming yet, please read and learn from Theming in Magento on the website of magento.com

• Add the following code to your existing local.xml

  <meta name="robots" value="index,follow" />

  If you’re using a blank local.xml copy and paste the following code.

  <meta name="robots" value="index,follow" />

  It will call the handles of the layered navigation and set the values of meta tag robots to noindex and follow. I don’t want the search engines to index my pages using Layered Navigation. But I do want to let it follow the links on it. Specially to the products and menuitems on the page.

• Clear your cache and refresh the page containing a Layered Navigation.
• Open the source code of that page to see that the value of meta tag robots is set to noindex,follow.

Add rel=nofollow to links in layered navigation

To prevent search engines to follow the weblinks in the block containing the items to filter the results you just have to change one file.

• Copy app/design/frontend/base/default/template/catalog/layer/filter.phtml to the same directory in your own theme.
• Open the copied file and find the following:

  <meta name="robots" value="index,follow" />

• Add after:

  <meta name="robots" value="index,follow" />

• Save & close
• Clear your cache and refresh the page containing a Layered Navigation.
• Open the source code of that page to see that every link in the block containing the filter is expanded with the new attribute rel=nofollow.

Enjoy the reduced resources

Every visit to a page on your website will use resources of your server. Even every visit of a search engine. Exploring several webstats of different webshops we came to notice that most resources are consumed by search engines.
Using the method described above is one of the many ways to reduce this consuming site traffic.

Visit “Performance en optimalisatie” on our wiki for more optimization suggestions

[widget id=”magereport_widget-5″ /]