📄 中文摘要
语言扩散模型旨在提升自回归大型语言模型（LLMs）的采样速度和生成连贯性。引入神经流扩散模型（NFDM）用于语言生成，作为NFDM的扩展，该模型能够将连续扩散模型直接应用于离散状态空间。NFDM从数据中学习多变量前向过程，确保前向过程和生成轨迹与语言建模高度契合。模型通过构建一个与数据分布相匹配的连续流，将离散的文本数据映射到连续的潜在空间，从而规避了传统扩散模型在处理离散数据时面临的挑战。在潜在空间中，模型利用连续扩散过程进行高效的采样和生成，随后通过逆映射将潜在表示转换回离散的文本序列。这种方法显著改善了生成文本的质量和多样性，同时大幅提升了采样效率。与现有扩散模型相比，NFDM特别针对语言的离散性和结构性特点进行了优化，例如，通过在设计前向过程时融入语言的语法和语义信息，使得生成的文本不仅流畅自然，而且语义准确。此外，模型的泛化能力强，能够应用于多种语言生成任务，包括文本补全、摘要生成和机器翻译等。NFDM的框架允许更灵活地控制生成过程，例如通过调整潜在空间中的路径，可以实现对生成文本风格、长度或主题的精细化控制。实验结果表明，NFDM在多个基准数据集上均取得了优异的表现，在保持高生成质量的同时，显著超越了传统自回归模型和现有扩散模型在采样速度上的限制，为语言生成领域带来了新的突破。
📄 English Summary
Towards Latent Diffusion Suitable For Text
Language diffusion models aim to enhance sampling speed and coherence compared to autoregressive large language models (LLMs). This work introduces Neural Flow Diffusion Models (NFDM) for language generation, an extension of existing NFDM frameworks that facilitates the straightforward application of continuous diffusion models to discrete state spaces. NFDM learns a multivariate forward process directly from the data, ensuring that both the forward process and the generative trajectory are well-suited for language modeling. The model maps discrete text data into a continuous latent space by constructing a continuous flow that aligns with the data distribution, thereby circumventing the challenges faced by traditional diffusion models when handling discrete data. Within this latent space, the model leverages continuous diffusion processes for efficient sampling and generation, subsequently transforming the latent representations back into discrete text sequences via an inverse mapping. This approach significantly improves the quality and diversity of generated text while substantially boosting sampling efficiency. Compared to existing diffusion models, NFDM is specifically optimized for the discrete and structured nature of language. For instance, incorporating linguistic syntactic and semantic information into the design of the forward process ensures that the generated text is not only fluent and natural but also semantically accurate. Furthermore, the model exhibits strong generalization capabilities, applicable to various language generation tasks, including text completion, summarization, and machine translation. The NFDM framework allows for more flexible control over the generation process; for example, by adjusting paths in the latent space, fine-grained control over the style, length, or topic of the generated text can be achieved. Experimental results demonstrate superior performance of NFDM on multiple benchmark datasets, maintaining high generation quality while significantly overcoming the sampling speed limitations of traditional autoregressive models and existing diffusion models, thus presenting a novel advancement in the field of language generation.