Fabrice Canel posted an SEO best practices post for sites that use subscription-based and paywall content. While Google has something called flexible sampling, previously a first click free program - Microsoft Bing does not.
Bing basically wants you to detect that BingBot is trying to access this subscription-based and paywall content and then give BingBot full access to that content. In a sense, show Bing all the content and show the users only a snippet of the content. Bing is saying this is okay, this is not a form of cloaking and this is allowed.
Fabrice wrote "the first step is to allow search engines, like Bing, to see the full content that normally resides behind a paywall or a subscription. By accessing the full content, search engines will be able to index more text which will help match more customer queries." "You can identify Bingbot by referencing the IP address against the public list of Bingbot IP addresses," he added. The list of BingBot IPs were published by Microsoft recently.
Then Fabrice explained the next step is to "prevent search engines from exposing publisher subscription-based or paywall content in search engines cache pages." He aid "publishers can control whether search engines show a cached page of a document by using a special robots meta tag in the head section of the web page or, alternatively, by using a customer HTTP response header returned to the search engines crawler for each URL."
His blog post explains how to do this Robots Meta Tag and/or X-Robots-Tag.
News web sites, academic and research web sites, enabling indexing of your subscription-based and paywall content in search engines is easy. Get more customers visiting your web sites and no need to engage with search engines. https://t.co/P8VSJcfewB
— Fabrice Canel (@facan) May 2, 2022
He then explains why they don't take Google's approach:
JSON-LD can help but not a strong requirement for me. This is occasionally abused, implemented with issues and JSON-LD solution does not work for binary files as subscription based academic PDF files which is a common scenario.
— Fabrice Canel (@facan) May 3, 2022
Forum discussion at Twitter & WebmasterWorld.
