Robots Meta tag is a direct instruction to Google whether to index your page or not. Robots tag is implemented on the page level. The Robots Meta Tag is included in the HEAD Section of web pages. While robots.txt is implemented on the site level, it will be discussed in the later posts.
In various instructions, you can use the Robots Meta Tag. But, even if you do not mention the Robots Meta Tag on a webpage, Google by default Index and Follow the links. So, it is not necessary to include the command or instruct the Google to index your pages. But, if you do not want the Google to Crawl or Index your pages then you need to implement the following.
The Valid Values of Robots Meta Tag are “Content” attributes are “INDEX”, “NOINDEX” and “FOLLOW”, “NOFOLLOW”
Find the below examples instructing the Search Engines using Robots Meta Tag:
Meta Name = “Robots” Content = “INDEX, FOLLOW”
Meta Name = “Robots” Content = “INDEX, NOFOLLOW”
Meta Name = “Robots” Content = “NOINDEX, NOFOLLOW”
Meta Name = “Robots” Content = “NOSNIPPET”
Meta Name = “Robots” Content = “NOARCHIVE”
Meta Name = “Robots” Content = “NOODP”
Meta Name = “Robots” Content = “NONE”
Understand the below interpreting the Robots Meta Tag:
Instruction to Google to include in the index.
Instruction to Google bot to follow the links of the page.
Instruction that prevents the page from getting indexed.
Instruction that prevents the Google bot from following the links of the page
Instruction that prevents the description from appearing below the page also prevents the page from caching.
Instruction that prevents the cached copy of the page available in the search results.
Instruction that blocks the open directory project description of the page to appear in the search results.
Instruction that is similar to “NOINDEX, NOFOLLOW”
Robots Meta Tag is the only Meta tag that can be used to manage the bots from accessing, caching and indexing your web pages. At a page level, this is the best way to control your web pages to appear in the search results.