Google has announced that they have published their Google Assistant and voice search quality guidelines, similar to how Google published their search quality raters guidelines. These give specific guidelines for third parties to evaluate the quality of the search results, and in this case, the Google Assistant voice search responses.
The full and short PDF is available as a download over here.
Google wrote:
In the past we have received requests to see our evaluation guidelines from academics who are researching improvements in voice interactions, question answering and voice-guided exploration. To facilitate their evaluations, we are publishing some of the first Google Assistant guidelines. It is our hope that making these guidelines public will help the research community build and evaluate their own systems.
Here are the dimensions Google is asking these raters to cover:
- Information Satisfaction: the content of the answer should meet the information needs of the user.
- Length: when a displayed answer is too long, users can quickly scan it visually and locate the relevant information. For voice answers, that is not possible. It is much more important to ensure that we provide a helpful amount of information, hopefully not too much or too little. Some of our previous work is currently in use for identifying the most relevant fragments of answers.
- Formulation: it is much easier to understand a badly formulated written answer than an ungrammatical spoken answer, so more care has to be placed in ensuring grammatical correctness.
- Elocution: spoken answers must have proper pronunciation and prosody. Improvements in text-to-speech generation, such as WaveNet and Tacotron 2, are quickly reducing the gap with human performance.
Check out the full guidelines over here.
Forum discussion at Twitter.
