LumberChunker：长篇叙事文档分段

LumberChunker is a technique that leverages large language models (LLMs) to determine where to segment long-form narratives, creating more natural text chunks that assist Retrieval Augmented Generation (RAG) systems in retrieving the right information. Long-form narrative documents typically have an explicit structure, such as chapters or sections, but these units are often too broad for retrieval tasks. At a lower level, significant semantic shifts occur within these larger segments without any visible structural breaks. When text is split solely based on formatting cues, like paragraphs or fixed token windows, passages belonging to the same narrative unit may be separated, while unrelated content can be grouped together. This misalignment between structure and meaning results in chunks that contain incomplete or mixed information.