John Mueller was asked if getting crawled by Googlebot Desktop can explain why they’re suffering from delayed indexing
Google’s John Mueller answered a question about whether it matters to indexing speed if Googlebot mobile or Googlebot desktop crawls a page. The person asking the question was worried about slow indexing and wanted to know if getting crawled by Googlebot desktop (instead of Googlebot mobile) was the reason.
Googlebot Desktop and Mobile
Google has a multitude of website crawlers that follow from link to link discovering new web pages to crawl and index.
If you look at your traffic logs you might see:
- Googlebot Desktop
- Googlebot Smartphone
- Googlebot Image
- Mediapartners-Google (Mobile AdSense)
- FeedFetcher-Google
- Google-Read-Aloud
- AdsBot-Google
- Googlebot-Video
Those are just a few of the Google-related Google crawlers that can visit a website.
The question John Mueller answered was primarily concerned with Googlebot Desktop and Googlebot Smartphone and what impact on indexing the desktop version might have.
This is the question:
“Our website’s posts are being indexed after one week. The site is old and mobile-friendly.
When I submit URLs in Inspection Tools, it shows it’s crawled by Google smartphone.
But in the About section, the indexing crawler is shown as Googlebot Desktop.
Does this mean the Smartphone bot is crawling the URL and the Desktop bot is indexing this URL?
Is this the reason behind the indexing delay?”
John Mueller’s answer:
“This would not be a reason for any kind of indexing issues.
I don’t know offhand why you might see this mix of desktop and mobile crawlers there.
It might just be that the Search Console is essentially saying well, by default, the Inspection Tool will just use the mobile Googlebot.
In practice, we always crawl with both of these crawlers.
It’s just a matter of how much we crawl and index with any of these crawlers.
So, usually you’ll see something like 80% is coming from mobile Googlebot and 20% maybe from desktop Googlebot.
And that kind of ratio, that is essentially normal and doesn’t mean that things are slower than anything else.”
Related: Why GoogleBot Doesn’t Crawl Enough Pages on Some Sites
Multiple Versions of Googlebot Crawl Websites
Google’s Googlebot support page states that both versions of Googlebot visit web pages.
According to Google’s official documentation:
“Your website will probably be crawled by both Googlebot Desktop and Googlebot Smartphone.
You can identify the subtype of Googlebot by looking at the user agent string in the request.”
According to Mueller and Google it’s totally normal to be crawled by both and there’s no indexing impact related to getting crawled by Googlebot Desktop.
Citation
Google’s Googlebot Page
Googlebot
Google’s Overview of Google Crawlers (User Agents)
Does Predominantly Googlebot Desktop Crawl Cause Indexing Delay?
Watch the video at the 34:28 minute mark
