Google’s John Mueller answered a question about declining crawl rates after making large changes to the websites infrastructure, in this case, moving to a new content delivery network (CDN).
Mueller explains why indexing may appear to slow down after these kinds of changes.
The person asking the question noted that they created redirects for URLs and moved to a new Content Delivery Network (CDN) and soon after began experiencing notable drops in indexing rates.
Background on Website Indexing by Google
Indexing is a reference to the process of Google’s web crawler, Googlebot, visiting a web page and downloading it to Google’s search index.
Google’s search index is where candidate pages eligible to be ranked exist. Search results come from Google’s search index.
That is the reason why indexing rate by Google is so important. Google needs to crawl and discover new pages to rank.
It is considered a major problem if something causes Google to not crawl a web page.
According to Google’s help page on crawling and indexing:
“We use software known as web crawlers to discover publicly available webpages. Crawlers look at webpages and follow links on those pages, much like you would if you were browsing content on the web.
They go from link to link and bring data about those webpages back to Google’s servers.
…We take note of key signals — from keywords to website freshness — and we keep track of it all in the Search index.”
Changing Content Delivery Network is a Major Change
A content delivery network is a network of servers positioned all around the world. The purpose is to speed up the delivery of website content by serving the content from a server that is close to the person trying to visit the web page.
So in theory Google should be able to visit the site and index it even faster, which is the reasonable expectation.
This is the question asked:
“After redirecting and changing and CDN we’re seeing a drastic drop in crawl rate.”
John Mueller Explains Severe Drops in Crawling
Google’s John Mueller answered:
“Yes! That’s every reasonable.
If you change your website’s infrastructure then we will change our crawling.
On the one hand, first, to be a little bit conservative and make sure we don’t cause problems and then later on we automatically ramp up again.
So if you change to a different CDN that’s a significant change in the infrastructure and we recognize that change and we hold off crawling for a while and then we ramp up again if we think everything is fast.”
Google is Avoiding Being Disruptive
The whole point of a “drastic drop” in indexing when major infrastructure changes are made is to make it easy on the updated site.
Sometimes a site that is rolling out a major change can have multiple changes over the course of a week or longer.
Indexing a site that might not be completely updated or ready to be indexed can result in confusion at Google if the site is in a state of constant change.
So it’s actually a good thing that Google slows down its crawling after a major infrastructure change to the website.
Citation
Watch Mueller answer question at the 29:10 minute mark
