Google’s BLEURT is BERT for Evaluating Natural Language Generation Models

The new method uses BERT pretrained models to evaluate the quality of the output of NLG models.

Source: https://www.mediaupdate.co.za/media/143578/what-is-natural-language-generation

What Makes a Good NLG Sentence?

To understand the magnitude of the challenge BLEURT is trying to address, it might be helpful to first develop a criteria to evaluate the quality of a NLG sentence. When presented with a specific text, how do we judge its quality? There are many aspects that go into consideration. Fluency, clarity of the main idea, expressiveness, syntactic and semantic correctness and dozens of others. Many of those criterions are incredibly subjective but human judgement but still is possible to develop quantitative metrics that reflects its results.

Source: https://ai.googleblog.com/2020/05/evaluating-natural-language-generation.html

Enter BLEURT

The fundamental challenge of creating a quality metric for NLG system is that it should not only match human judgement but do so across all sorts of conversational domains. This challenge is even more relevant if we consider the small amount of training data available. Fortunately, Google has been at the forefront of some of the most impressive breakthroughs in language understanding and representations. BERT sparked a new wave of innovation in the NLU space with its Transformer based architecture that were able to achieve state of the art performance across different NLP tasks. BERT seems like an ideal candidate in order to leverage unsupervised representations that could mitigate the absence of large training datasets for NLG evaluation tasks.

Source: https://ai.googleblog.com/2020/05/evaluating-natural-language-generation.html
Source: https://arxiv.org/abs/2004.04696
Source: https://arxiv.org/abs/2004.04696

CEO of IntoTheBlock, Chief Scientist at Invector Labs, I write The Sequence Newsletter, Guest lecturer at Columbia University, Angel Investor, Author, Speaker.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store