临床数据抽象的稳定性感知提示优化

Large language models, when applied to clinical data abstraction, exhibit significant sensitivity to prompt wording. However, most existing research treats prompts as static entities and investigates uncertainty in isolation. This work argues for a joint consideration of prompt sensitivity and uncertainty. Across two distinct clinical tasks—MedAlign applicability/correctness and multiple sclerosis subtype abstraction—and utilizing multiple open-source and proprietary models, prompt sensitivity is quantified through flip rates. This sensitivity is then correlated with model calibration and the potential for performance improvement via selective prediction. Findings indicate that high prompt sensitivity is frequently associated with poorer calibration performance and a greater potential for performance gains through selective prediction. Building upon these observations, a stability-aware prompt optimization methodology is proposed. This method involves generating and evaluating multiple prompt variants to identify those that minimally impact model outputs. Such an approach not only discovers more stable prompts, thereby enhancing model robustness across different prompt formulations, but also improves calibration by more accurately identifying model uncertainty. By optimizing prompts, the models are able to produce more consistent and reliable outputs when faced with similar yet differently worded instructions. Furthermore, this method contributes to a better understanding of model limitations in specific clinical tasks and offers more robust strategies for deploying language models in clinical applications. Experimental results demonstrate the significant effectiveness of this approach in improving the accuracy and reliability of clinical data abstraction tasks, providing a novel optimization pathway for the practical application of large language models in the medical domain.