Gap-K%：测量Top-1预测差距用于检测预训练数据

The inherent opacity of massive pretraining corpora in Large Language Models (LLMs) poses significant privacy and copyright concerns, elevating pretraining data detection to a critical research challenge. Current state-of-the-art detection methodologies predominantly rely on token likelihoods, yet frequently overlook two crucial aspects: the divergence from the model's top-1 prediction and the localized correlation between adjacent tokens. To address these limitations, a novel pretraining data detection method, termed Gap-K%, is introduced. This approach fundamentally leverages the gap between the model's top-1 prediction and its subsequent best prediction, integrated with K% of local contextual information. Specifically, Gap-K% identifies pretraining data by analyzing the difference between the highest and second-highest probability predictions for a target token within a given K% context window. When an LLM has been trained on specific data, its prediction confidence for that data typically becomes anomalously high, leading to a significantly enlarged gap between the top-1 and top-2 prediction probabilities. By quantifying this 'prediction gap,' Gap-K% effectively captures the model's memorization traces of pretraining data. Concurrently, the incorporation of the K% local correlation concept ensures that detection is not solely based on individual token likelihoods, but also considers the local structural and semantic information of the text. This combined approach, integrating prediction gap analysis with local contextual understanding, enables Gap-K% to exhibit superior accuracy and robustness in determining whether a model has been exposed to a particular text segment, particularly when dealing with subtle memorization signals that might only manifest in minor secondary predictions.