Google’s John Mueller answered a question from an e-commerce publisher whose pages were not being indexed. Mueller suggested specific diagnostic tips then recommended the value in promotion for getting a site indexed. According to Mueller, Google sometimes needs encouragement to index an entire site.
Then the site publisher revealed an important detail about the site.
Website Inadequately Indexed
The person asking the question explained the situation of the site not being indexed. They noted how a site colon (site:) search showed only a few pages indexed.
Of interest, is that he was leaving out an important clue about the problem, which was only revealed after John had offered an extensive amount of advice.
John addressed this issue but somewhat dismissed it as an ongoing factor.
Here is the question asked (minus a super important detail):
“So there’s a website… that is taking… longer to index. And the current situation is, I have 170 valid pages in Search Console.
But when I check it through site colon (site:), it just says three, four, sometimes five pages.
So that has been happening for three to four months now.
I’m not sure what the correct way to go from now on.
It’s …an e-commerce website. And it has about less than 500 product pages.”
Google’s Mueller addressed the shortcomings of the site colon search, saying it’s not a diagnostic tool and explained why it’s not good for diagnosing site indexing issues.
Screenshot of John Mueller Offering Site Indexing Advice
Mueller:
“So, in general, I would not worry about the difference between a site/colon query shows and what Search Console shows.
In practice, the site/colon query is not meant to be used for diagnostic purposes.
So sometimes you do see quite different numbers.
And for some sites, you see much lower numbers. For other sites you see 100 times larger numbers.
And essentially, the numbers we show in the site query result are optimized for speed and to give (kind of) a sense of the website.
But they’re not optimized to be comprehensive.So that’s where the Search Console numbers would come in.”
In this next part, Mueller explained the background information about what it takes to index a smaller site (not much but something) and offered encouraging site promotion advice.
But first he suggested specific diagnostic tests.
Mueller:
“I think in general, if you’re talking about a site that is about 500 pages or so and after a period of several months only like 100 are indexed, that seems like something where probably you could do a lot better with minimal extra effort.
So that could be something where, what I would first of all check is just the technical side of things to make sure that technically everything is okay, that the website can be crawled.
There are some website crawling tools available, I think they’re even free for very small websites, where you can check your site to see if it’s crawlable or not.”
Mueller is correct, many site auditing tools offer free versions for smaller sites and that’s something any site should take advantage of.
Google Sometimes Needs a Hint to Encourage Full Indexing
Next Mueller explained why site promotion was important for a smaller site that needs to be indexed, explaining that sometimes Google needs encouragement.
Mueller:
“And if it’s well crawlable then the next thing I would consider trying to figure out is what you can do to promote your website a little bit better.
And that could be something like encouraging users to come to visit, maybe by buying ads, maybe by working together with someone else for a while just to (kind of) get the ball rolling.
It could also be if you’re an e-commerce site or especially if you’re a small local business site, maybe there are local chambers of commerce that would be interested in linking to your website to give you a little bit of extra information.
So that when our systems look at your website, they say, oh this is actually a legitimate small business. We should try to index everything.
Because especially if you’re talking about a smaller website with a couple hundred pages, that feels like something where if we have a little bit of a hint then we’ll go off and get all of that.
If you’re talking about an e-commerce site that has 500,000 pages then obviously (like) if we get all of those pages or not, that’s a totally different story.
But with 500 pages it feels like something that with a little bit of extra incentive to say that we should at least check it out, we should be able to get a significant part of those indexed.”
The Important Detail that Was Left Out
Now we get to the part where the publisher asking the question does the “oh and one more thing” bit and drops a bombshell revelation that should have been mentioned at the very beginning.
The person asking the question next added that search console is reporting several hundred errors that are related to having previously noindexed the site in January.
He removed the noindex several months ago from the product pages and left it on what he called “unwanted pages” but that now it keeps failing validation in Google Search Console.
John Mueller said that he believes the transition from noindex to getting content indexed could by the reason for current issues the publisher was having in getting product pages adequately indexed.
He also noted that the validation tool did not reflect Google’s ability to index pages but rather it’s just a way to provide information to the publisher.
Mueller explained:
“So just because the validation fails for that and the other URLs are okay, I don’t think that would be holding your website’s indexing back.
It’s more, Search Console is a little bit confused because you said you fixed the issue but you didn’t really fix the issue.”
Mueller Recommends Site Promotion to Get the Site Indexed
Mueller said that switching the noindex to index could slow the indexing down but that it’s been a couple of months since the noindex was removed. So he returned to recommending that the publisher consider actively promoting the site, including leveraging their social media followers to promote the site.
He reiterated that it doesn’t take a lot to encourage Google to index a smaller website like that.
Site Promotion and Indexing
Google crawls websites through the links that connect one website to another website. And according to Mueller it doesn’t take much in the way of links to get Google to index more of the website.
Citation
Watch Mueller answer question at the 21:14 minute mark
