在本地运行 Qwen 397B 的苹果“闪电 LLM”研究

Dan Woods' research demonstrates how to run a custom version of Qwen3.5-397B-A17B at over 5.5 tokens per second on a 48GB MacBook Pro M3 Max, despite the model occupying 209GB (120GB when quantized) on disk. Qwen3.5-397B-A17B is a Mixture-of-Experts (MoE) model, meaning each token only needs to compute against a subset of the overall model weights. These expert weights can be streamed into memory from SSD, preventing the need for all weights to be held in RAM simultaneously. Dan utilized techniques introduced by Apple in 2023 to achieve this.