While major companies like Wix, Duda and Cloudflare announced enthusiastic adoption of the new open source IndexNow indexing protocol that promises instant indexing, it was also confronted with questions over whether the IndexNow was necessary, worth the trouble adopting and doubt as to whether any of its benefits will be realized.
What is IndexNow?
IndexNow is an open source project that is attempting evolve how search engines discover and index content in a way that benefits both search engines and publishers.
IndexNow also promises to rapidly speed up the process of re-indexing content that has been updated and indexing content that is brand new.
Lastly, IndexNow promises to be more energy efficient because they will be crawling websites less often.
The old way of discovering web pages and indexing content is largely unchanged for decades.
Crawlers check web pages on the Internet over and over, regardless if a web page is unchanged or does not exist.
While search engines employ strategies to reduce how many pages are crawled, crawling an unchanged web page can be seen as a profligate use of electricity, like watering a lawn during a rainstorm.
IndexNow claims to solve that problem by eliminating redundant crawls and focusing just on the pages that are new or updated.
But not everyone is convinced.
Some in the search community have expressed doubts about every single benefit IndexNow claims to provide, including the claim that less pages will be crawled.
Yoast Declines to Adopt IndexNow
Joost de Valk, founder of Yoast WordPress SEO plugin tweeted a series of statements questioning the value offered by IndexNow. He stated that the Yoast plugin would not be adopting the IndexNow standard.
He tweeted why Yoast is not adopting IndexNow:
“We’ve not added support to @yoast SEO for it yet, for the simple reason that we don’t see the value yet.”
Yoast: IndexNow is Not Needed
Joost next laid out the case that IndexNow is solving a problem that already has a solution.
He claimed that XML sitemaps already accomplish what IndexNow is trying to do.
He tweeted:
“Ever since XML sitemaps were conceived, URL discovery as far as we’re concerned is a “solved problem”.
For most sites, getting search engines to crawl content is not a problem.
Yoast SEO generates XML sitemaps and it can even ping search engines when those change.”
IndexNow Is Useful for Large Websites
Someone responded to Joost and said that the Bing API for fast indexing provided value for sites they worked on like Groupon and GitHub.
She tweeted:
I have used the Bing API WordPress plugin on numerous sites and developed a feed for the API for both Groupon and GitHub and saw the value.
No need for Yoast to incorporate it when Bing makes their own plugin though.— SEOGoddess @GitHub 🤔 (@SEOGoddess) October 19, 2021
Joost Acknowledges that IndexNow Benefits Big Sites
He tweeted:
“For large sites like that I definitely see the value.
But to do this and make it a default for 12 million sites, most of which have less than 100 pages, seems overkill.”
Duda Explains Why IndexNow Provides Value
One of the companies that is supporting IndexNow is Duda, a professional site building platform that competes with WordPress and Wix.
Duda is a website building platform that allows web professionals to create a web presence that performs well out of the box.
But it’s also flexible similar to WordPress and allows customers to build custom widgets, embed the website builder within their own software (SaaS) or migrate legacy sites entirely in large scale.
I asked Russ Jeffery, Director of Strategic Integrations at Duda to explain why XML sitemaps are not enough and why IndexNow provides value:
“There is a delay between publishing, crawl, index, and finally showing in search results.
Right now search engines must re-crawl all pages of a site to find new content and not on the update schedule of an actual site update.
The difference here is that you’re telling Bing to only crawl these updated pages and do it now. A push, rather than a pull, approach is a widely accepted best practice of updating data/content between different systems efficiently.”
I pointed out to Russ how Joost insisted that XML sitemaps were good enough and that IndexNow was a better value for large sites.
Russ responded:
“I do agree with Joost that for a lot of websites, it is not the most important capability to have at the moment.
But, I think this approach to indexing is a lot better than the traditional sitemap system and the industry should move towards ‘pushing’ content to search engines for them to crawl.”
Waiting for Google to Adopt IndexNow
I turned to longtime UK search marketing expert Ammon Johns (@Ammon_Johns) for his opinion. Like Joost, he too was not enthusiastic about IndexNow and didn’t feel it offered anything substantial.
Ammon acknowledged the elephant not in the IndexNow room, which was Google.
He said:
“Unless and until Google sign up for a Bing led protocol, it’s not a huge deal.
But if Google do go along for some reason (rather than make their own), we’ll certainly all have to revisit a lot of understanding about crawl prioritization”
Promise of Less Crawling in Doubt
Ammon downplayed any benefit of less crawling, saying that they’d likely use the freed-up resources to go out and crawl lower priority web pages that would not ordinarily be crawled.
He said:
“Even if this were to free up some resource allocated to crawling more than can be crawled (the state of the web currently, and growing exponentially), I can’t see the search engines not simply crawling more of the lower priority, or re-crawling the absolute highest with more frequency.
…Microsoft don’t even actually promise they’ll grab it at once, only that it will be indexed ‘soon’ or ‘quickly’.”
He then pointed out that adopting the IndexNow protocol will eliminate a latent signal of what pages Google or Bing feel are most important.
IndexNow May Remove a Data Point
Ammon noted:
“One of the major reasons I never, EVER, use submission tools is that seeing when content gets crawled on its own is far too useful.
There’s no other way to see ‘importance’”
There are multiple things that influence how often a web page is crawled, links being one of them. How often a page or section of a website changes can also influence how often a web page is crawled.
Whether fast indexing is worth it to give up seeing which pages a search engine feels are more “important” is up to the individual and their priorities. Ammon feels it’s not worth it for him.
Will the Web Adopt IndexNow?
Joost’s initial post stated that they haven’t decided to adopt IndexNow yet because they don’t see the value in it.
Yesterday Microsoft Bing introduced IndexNow, a new standard to “instantly index” content. They do so in conjunction with Yandex.
We’ve not added support to @yoast SEO for it yet, for the simple reason that we don’t see the value yet.
🧵 1/6
— Joost de Valk (@jdevalk) October 19, 2021
One of the important features of the Yoast SEO plugin is the ability to automatically create XML sitemaps, something that Joost feels is sufficient for most websites.
Should Google adopt the IndexNow protocol, which at the moment seems remote, XML sitemaps might lose a fair bit of their importance, which might dampen the shine on what is currently an important feature of the Yoast plugin.
Should Google adopt IndexNow it raises the odds that WordPress might follow through and include it into the core, eliminating the need for Yoast to turn it into a plugin feature.
So it seems for Yoast there is relatively little to any value from IndexNow.
It’s up to the search community to decide for themselves. There is some opposition and skepticism of IndexNow.
But there are also many big companies like Cloudflare who are integrating IndexNow enthusiastically.
Joost acknowledged that IndexNow is useful for big companies but also pointed out that in his opinion there’s less value in it for the smaller sites.
Perhaps as Ammon noted, for some in the search community, particularly the smaller sites, unless Google adopts IndexNow, the decision for them is going to feel like it’s optional and not a necessity.
