📄 中文摘要
构建现代 AI 知识系统时，通常直接讨论提示、检索管道或模型选择。然而，在模型生成答案之前，数据必须转化为模型能够理解和高效检索的格式。这一转化通常涉及几个基础步骤：1. 标记化——将原始文本转换为模型可读的单元；2. 分块——将文档拆分为可管理的段落；3. 向量化——将文本转换为嵌入；4. 索引——存储向量以便高效的相似性搜索。这些步骤构成了基于检索的 AI 系统的基础，设计决策对系统的性能和成本有着重要影响。
为何 AI 系统变得昂贵：云端（AWS）中的标记化、分块和检索设计
📄 English Summary
Why AI Systems Become Expensive: Tokenization, Chunking, and Retrieval Design in the Cloud (AWS)
Building modern AI knowledge systems often leads to discussions about prompts, retrieval pipelines, or model selection. However, before a model generates an answer, data must be transformed into a format that models can understand and retrieve efficiently. This transformation typically involves several foundational steps: 1. Tokenization - converting raw text into model-readable units; 2. Chunking - splitting documents into manageable segments; 3. Vectorization - converting text into embeddings; 4. Indexing - storing vectors for efficient similarity search. These steps form the foundation of retrieval-based AI systems, and design decisions significantly impact system performance and costs.