This third article in the series on Distributed MLOps explores overcoming vendor lock-in by unifying AMD and NVIDIA GPUs in mixed clusters for distributed PyTorch training, all without requiring code rewrites: Mixing GPU Vendors: It demonstrates how to combine AWS g4ad (AMD) and g4dn (NVIDIA) instances, bridging ROCm and CUDA to avoid being tied to a single vendor. High-Performance Communication: It highlights the use of UCC and UCX to enable efficient operations like all_reduce and all_gather, ensuring smooth and synchronized training across diverse GPUs. Kubernetes Made Simple: How Kubernetes, enhanced by Volcano for gang scheduling, can orchestrate these workloads on heterogeneous GPU setups. Real-World Trade-Offs: While covering techniques like dynamic load balancing and gradient compression, it also notes challenges current limitations. Overall, the piece illustrates how integrating mixed hardware can maximize resource potential, delivering faster, scalable, and cost-effective machine learning training.

The post critiques AI evaluation methods from a physicist's perspective, highlighting a troubling lack of scientific rigor compared to fields like physics. While physicists meticulously define success criteria before experiments (like CERN's specific statistical requirements for the Higgs boson), AI benchmarking suffers from three critical problems: Benchmarks are abandoned once models perform well, creating an endless cycle without measuring meaningful progress. With models training on vast internet data, benchmarks are likely contaminated, essentially giving open-book exams to models that have already seen the material. Current methods fail to properly measure generalization - whether models truly understand concepts versus memorizing patterns. The author proposes a "Standard Model of AI Evaluation" bringing together cognitive scientists, AI researchers, philosophers, and evaluation experts to create hypothesis-driven benchmarks rather than difficulty-driven ones. This framework would require pre-registered hypotheses, contamination prevention strategies, and clearly defined success criteria. The post concludes by asking whether systems potentially transforming society deserve evaluation standards at least as rigorous as those used for testing new particles.