Exploring NLP Part 2: A New Way to Measure the Quality of Synthetic Text

This text discusses the evaluation of synthetic text quality using various metrics. It presents an overview of reference-based metrics like BLEU and ROUGE, as well as newer ones such as BERTScore and BLEURT. The main focus is on Fréchet BERT Distance (FBD), a metric that evaluates the distance between distributions created by real text embeddings and synthetic text embeddings. The authors also introduce an enhancement to FBD called Fréchet Cosine Similarity Distance (FCSD). They replicate experiments from Xiang et al.'s paper, which first introduced FBD, and find that their new metric outperforms the original one in several measures.