Mixture of Experts
-
China’s AI Bets on Memory, Not Just Compute
Moonshot AI’s Kimi K3, a 2.8 trillion-parameter open-weight model, utilizes “mixture-of-experts” and quantization-aware training to optimize memory over compute, potentially circumventing U.S. restrictions. This approach allows it to function with reduced computational demands, though memory requirements remain substantial. Despite impressive performance in specific benchmarks, Kimi K3 still trails leading models in overall capabilities. Its release on July 27th will offer a significant opportunity to assess its impact on the open-weight model landscape.
-
AI model trained on AMD GPUs achieves milestone
Zyphra, AMD, and IBM have collaboratively developed ZAYA1, a Mixture-of-Experts foundational model, using AMD’s GPUs and platform. Trained on AMD’s Instinct MI300X accelerators within IBM Cloud, ZAYA1 demonstrates comparable or superior performance to established open-source models. Zyphra optimized ROCm for AMD GPUs, focusing on memory capacity and inter-GPU communication. This initiative highlights the viability of AMD-based solutions as a cost-effective alternative to NVIDIA for large-scale AI model training, potentially impacting GPU market dynamics and AI procurement strategies.
-
“Huawei Boosts DeepSeek’s AI Performance: 10% Reduction in Inference Latency Through Expert Optimization”
When it comes to the most talked-about models in recent times, the Mixture of Experts (MoE…
