Autonomous measure of creativity in large language models (LLM)

Abstract

The present work proposes a new metric to measure the creativity of Large Language Models (LLM). Unlike traditional approaches that compare the results produced by LLMs with those produced by humans, this study presents an autonomous approach that does not depend on external reference systems. The methodology consists of providing a specific input prompt to LLMs, for example, generating a short three-paragraph story describing children in a park. Then, a large number of stories are generated, and techniques such as morphological or semantic grouping are applied to determine the number of original stories produced. Given that natural language processing (NLP) comparison techniques are computationally intensive and the calculation of distances is of order $$O(n^2)$$O(n2), it’s not possible to directly count the number of original stories generated by prompt. To address this, models have been developed from a calibration set, which simulate story generation and from which the maximum number of stories that can be generated is inferred. This methodology is not specific to the LLM or the prompt used, so it can be used to compare existing systems.

Relevance Assessment

Research Gap

Notes are automatically saved as you type

Search Queries