With the new Bing Webmaster Tools recommendations section, we are learning more about what Bing Search likes and dislikes. One of the things Bing does not like is when a site using batch mode, or batch URL groups for IndexNow. Bing says it can cause server overload and delays in indexing.
This was spotted by Ryan Siddle who posted about it on LinkedIn.
Ryan wrote, "do not batch URL groups for IndexNow. Bing creates a Moderate error because it can overwhelm the servers." He posted this screenshot that has this "recommended action" from Bing. It says, "Batch mode is not recommended as it can cause server overload and delays in indexing."
Here is the screenshot:
Here's what this means according to Bing:
- Batch Processing vs. Streaming
- Batch Processing: URLs are submitted in groups or batches. This method can overwhelm servers and cause delays.
- Streaming: URLs are submitted individually or in smaller groups as changes occur. This allows for more immediate updates to search engines.
- Benefits of Streaming
- Immediate Updates: Ensures that changes are reflected in search results as quickly as possible.
- Reduced Server Overload: Distributes the load more evenly over time, preventing server strain.
- Enhanced Search Engine Responsiveness: Keeps search engines informed of changes in real-time, improving indexing efficiency.
- Switching to streaming mode helps manage the current load more effectively and ensures timely updates.
Ryan added, "The IndexNow documentation shows the `urlList` attribute as an array which would suggesting batching is possible. However, this is more likely it is for batching in short pulses (i.e. every few seconds) rather than every couple of hours (or once per day). The URLs pushed to Bing's IndexNow count towards the total crawl budget allocated."
Forum discussion at LinkedIn.
Update: Fabrice Canel from Bing responded to this on LinkedIn saying:
Right, on behalf of all search engines supporting IndexNow. There is really no need for daily batch submissions, please.The goal of IndexNow is to provide an easy way for website owners to INSTANTLY inform search engines about the latest content changes on their site.
For instance, there's no need to read your sitemaps daily and submit all URLs to IndexNow. We already aim to process all sitemaps at least once a day, so that's not necessary, we will discover these URLs and URLs with lastmod correctly set will be prioritized.
So please submit your URLs to IndexNow whenever content is added, updated, or deleted. You can find best practices for updated URLs here.
