Abstract: Breakthroughs in generative artificial intelligence have led to the emergence of phenomenon-level large language models (LLMs), such as ChatGPT, posing unprecedented challenges to traditional data utility assessment methods. In response, this study focuses on evaluating the utility of instruction-tuning data for LLMs by establishing a multi-dimensional assessment framework that integrates three key dimensions—complexity, usability, and diversity—and accordingly proposes a novel data utility evaluation function. Experiments on multiple publicly available instruction-tuning datasets demonstrate that the proposed approach provides a reasonable and effective means of measuring data quality, while the reasoning loss observed in LLMs fine-tuned on different datasets exhibits a high degree of consistency with the proposed evaluation metrics. This work is the first to directly employ reasoning loss as a measure of the quality of LLM instruction-tuning data, further introducing the three dimensions—complexity, usability, and diversity—to characterize “high-quality data”. By proposing new quantitative metrics, this study offers important theoretical and practical guidance for future improvements in the quality of instruction-tuning data for large language models and related research applications.

Key words: Data elements, Data utility, Data quality, Large language models, Prompt fine-tuning data