.Google's John Mueller answered a question concerning why Google.com marks web pages that are forbidden from creeping by robots.txt as well as why the it is actually safe to disregard the associated Look Console reports about those crawls.Crawler Website Traffic To Concern Guideline URLs.The individual asking the inquiry recorded that crawlers were actually developing web links to non-existent query specification Links (? q= xyz) to webpages along with noindex meta tags that are actually additionally shut out in robots.txt. What triggered the concern is that Google is actually crawling the web links to those webpages, acquiring blocked out by robots.txt (without watching a noindex robots meta tag) then getting reported in Google.com Look Console as "Indexed, though blocked by robots.txt.".The person asked the observing question:." Yet below's the big question: why would Google index pages when they can't also find the content? What is actually the advantage during that?".Google.com's John Mueller validated that if they can not creep the page they can not see the noindex meta tag. He additionally helps make a fascinating acknowledgment of the site: search driver, urging to neglect the outcomes considering that the "common" customers will not see those outcomes.He wrote:." Yes, you're right: if we can not crawl the webpage, our company can not see the noindex. That pointed out, if we can't crawl the webpages, at that point there is actually not a lot for our company to index. Thus while you could find some of those pages along with a targeted website:- inquiry, the typical user will not view all of them, so I wouldn't fuss over it. Noindex is likewise fine (without robots.txt disallow), it simply suggests the URLs are going to wind up being crawled (as well as end up in the Search Console file for crawled/not listed-- neither of these statuses cause concerns to the remainder of the site). The important part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limitations in operation the Internet site: hunt advanced hunt operator for diagnostic causes. Some of those main reasons is actually considering that it is actually certainly not linked to the normal hunt index, it is actually a different thing entirely.Google.com's John Mueller commented on the internet site hunt driver in 2021:." The brief response is actually that a web site: inquiry is actually not indicated to be total, neither made use of for diagnostics objectives.A web site question is a specific sort of hunt that limits the end results to a particular internet site. It's generally just the word internet site, a bowel, and afterwards the website's domain.This concern restricts the end results to a specific web site. It is actually not indicated to be a comprehensive collection of all the pages from that website.".2. Noindex tag without using a robots.txt is fine for these kinds of situations where a crawler is actually linking to non-existent web pages that are acquiring discovered by Googlebot.3. Links with the noindex tag will produce a "crawled/not listed" item in Browse Console and also those will not possess an unfavorable effect on the rest of the site.Go through the concern as well as respond to on LinkedIn:.Why would certainly Google.com index webpages when they can not even view the web content?Included Picture by Shutterstock/Krakenimages. com.