AI model trained on AMD GPUs achieves milestone

In a significant development for the AI landscape, Zyphra, AMD, and IBM have concluded a year-long collaboration, culminating in the creation of ZAYA1 – a groundbreaking Mixture-of-Experts (MoE) foundational model. This endeavor, utilizing AMD’s GPUs and platform, successfully demonstrates the viability of AMD-based solutions for large-scale AI model training, potentially challenging NVIDIA’s dominance in the market.

The partnership’s flagship achievement, ZAYA1, is touted as the first major MoE model built entirely on AMD GPUs and networking infrastructure. This serves as a concrete example, suggesting that enterprises seeking to scale AI initiatives are no longer solely reliant on NVIDIA’s hardware and software ecosystem. The potential impact on the GPU market dynamics could be substantial.

ZAYA1’s training was conducted on AMD’s Instinct MI300X accelerators, complemented by Pensando networking, and leveraging AMD’s ROCm software platform, all deployed within IBM Cloud’s robust infrastructure. What distinguishes this system is its conventional architecture. Rather than employing experimental or esoteric hardware configurations, Zyphra opted for a setup mirroring typical enterprise clusters, critically omitting NVIDIA’s core components. This pragmatic approach suggests a readily deployable alternative for businesses.

According to Zyphra, ZAYA1’s performance metrics are on par with, and in some cases surpass, established open-source models in key areas such as reasoning, mathematical computation, and code generation. This achievement presents a compelling alternative for organizations facing supply chain constraints or escalating GPU prices, offering comparable capabilities without the associated vendor lock-in or premium cost.

How Zyphra Leveraged AMD GPUs for Cost-Effective AI Training

When formulating training budgets, most organizations prioritize memory capacity, inter-GPU communication speed, and predictable iteration times over sheer theoretical processing throughput. This is a strategic decision impacting project timelines and overall ROI.

The MI300X’s 192GB of high-bandwidth memory (HBM) per GPU offers engineers considerable flexibility, facilitating initial training runs without immediate recourse to intricate parallelism techniques. This simplification reduces the inherent fragility often associated with complex and time-consuming tuning processes.

Zyphra designed each node with eight MI300X GPUs interconnected via InfinityFabric, pairing each GPU with a dedicated Pollara network card. A separate network handles I/O operations like dataset loading and checkpointing. This deliberately straightforward architecture minimizes switching costs and simplifies network management, ultimately contributing to more consistent iteration times and streamlined workflows. This attention to detail maximizes infrastructure efficiency.

ZAYA1: A High-Performance AI Model

ZAYA1-base activates 760 million parameters out of a total 8.3 billion, a testament to the model’s efficient design, and was trained on a massive 12 trillion tokens across three distinct stages. The model architecture emphasizes compressed attention mechanisms, an advanced routing system for directing tokens to the most appropriate expert modules, and nuanced residual scaling to maintain stability in deeper layers. This sophisticated design allows for efficient scaling and performance.

The model utilizes a hybrid optimization approach incorporating both Muon and AdamW. To optimize Muon’s performance on AMD hardware, Zyphra implemented kernel fusion and reduced unnecessary memory traffic, ensuring the optimizer doesn’t bottleneck each iteration. Batch sizes were progressively increased during training, contingent on the storage pipeline’s ability to deliver tokens at a sufficient rate. This requires sophisticated storage and I/O management.

The culmination is an AI model trained entirely on AMD hardware that rivals the performance of larger models such as Qwen3-4B, Gemma3-12B, Llama-3-8B, and OLMoE. A key advantage of the MoE architecture is that only a fraction of the entire model is actively engaged during inference, thereby minimizing memory requirements and reducing associated serving costs. This could be particularly beneficial for edge deployments.

For instance, a financial institution could develop a domain-specific model for fraud detection without initially resorting to convoluted parallelism strategies. The MI300X’s abundant memory capacity provides engineers with ample room for experimentation, while ZAYA1’s compressed attention mechanism accelerates prefill times during model evaluation. This allows for agile development and faster deployment of specialized models.

Optimizing ROCm for AMD GPUs

Zyphra openly acknowledged the effort required to migrate a mature NVIDIA-based workflow to the AMD ROCm environment. Rather than simply porting existing code, the team dedicated significant effort to analyzing the behavior of AMD hardware and adapting model dimensions, GEMM patterns, and microbatch sizes to align with the MI300X’s optimal performance characteristics. This involved a granular understanding of both the hardware and software.

InfinityFabric achieves optimal performance when all eight GPUs within a node participate in collective operations, and Pollara tends to exhibit peak throughput with larger messages. Consequently, Zyphra tailored fusion buffers accordingly. Long-context training, ranging from 4k to 32k tokens, utilized ring attention for sharded sequences and tree attention during decoding to mitigate potential bottlenecks. This demonstrates a nuanced understanding of the underlying hardware architecture.

Storage considerations were equally pragmatic. Smaller models place a heavy burden on IOPS (Input/Output Operations Per Second), whereas larger models require sustained bandwidth. Zyphra addressed these challenges by bundling dataset shards to minimize scattered reads and increasing per-node page caches to accelerate checkpoint recovery, which is crucial during extended training runs where rollbacks are inevitable. This highlights the importance of optimizing I/O for AI workloads.

Ensuring Cluster Stability and Reliability

Training jobs that span weeks are inherently prone to failures. Zyphra implemented Aegis, a comprehensive monitoring service that analyzes logs and system metrics, identifies anomalies such as NIC glitches or ECC errors, and automatically initiates corrective actions. Furthermore, the team increased RCCL timeouts to prevent transient network interruptions from aborting entire training jobs. Proactive monitoring and automated remediation are vital for long-running AI experiments.

Checkpointing, a critical process for saving model state, is distributed across all GPUs rather than being funneled through a single bottleneck. Zyphra reports a more than ten-fold improvement in save times compared to conventional approaches, which directly enhances system uptime and reduces operator workload, enhancing efficiency and recoverability.

Implications of ZAYA1 for AI Procurement Strategies

This initiative subtly highlights the emerging distinctions between NVIDIA’s ecosystem and AMD’s alternatives: NVLINK versus InfinityFabric, NCCL versus RCCL, cuBLASLt versus hipBLASLt, and so forth. The success of ZAYA1 suggests that the AMD software stack is now sufficiently mature to support serious, large-scale model development, offering a credible alternative for enterprises.

This does not necessarily imply that businesses should immediately replace their existing NVIDIA infrastructure. A more pragmatic approach involves leveraging NVIDIA for production deployments while utilizing AMD for stages where the MI300X GPUs’ memory capacity and ROCm’s open-source nature provide significant advantages. This hedging strategy reduces vendor dependency and potentially increases overall training throughput without major disruption.

In conclusion, the consortium suggests that model shape should be treated as adaptable rather than fixed, they say to design networks that align with the collective operations, they also promote the use of tolerance to protect GPU hours and modernize checkpointing.

This isn’t necessarily a revolution but a pragmatic summary highlighting the key lessons learned from training a large MoE AI model on AMD GPUs. For organizations seeking to expand their AI capabilities without being exclusively reliant on a single vendor, this collaborative effort provides a valuable blueprint and a viable alternative. The ZAYA1 project underscores AMD’s growing presence in the AI acceleration market and the potential for increased competition and innovation.