可切换激活网络

Deep neural networks and large-scale generative models, such as large language models (LLMs) and large vision-action models (LVAs), demonstrate remarkable performance across various domains. However, their prohibitive computational costs hinder deployment in resource-constrained environments. Existing efficiency techniques provide only partial solutions: dropout improves regularization during training but does not affect inference, while pruning and low-rank factorization compress models post hoc into static forms with limited adaptability. SWAN (Switchable Activation Networks) introduces a framework that equips each neural unit with a deterministic, input-dependent binary gate, enabling the network to learn when to activate a unit, thereby enhancing the model's flexibility and efficiency in different environments.