Mixture-of-Experts Pattern

Using a sparse mixture-of-experts attention architecture activates only 32 B parameters at inference, enabling scaling to a trillion-parameter model cost-effectively.