HPC Observability: Production Monitoring, Profiling, and Site Reliability for Linux Clusters, GPUs, and Parallel Storage at Scale

HPC Observability is a hands-on guide for the engineers and administrators who keep high-performance computing systems running reliably at scale. It brings together the operational knowledge scattered across vendor documentation, conference papers, and forum threads into a practical framework for turning HPC telemetry into actionable insight. Modern HPC environments - Slurm clusters, GPU-dense AI systems, Lustre and GPFS storage, InfiniBand and Slingshot fabrics - generate more data than any team can manually interpret. ... Read More

HPC Observability is a hands-on guide for the engineers and administrators who keep high-performance computing systems running reliably at scale. It brings together the operational knowledge scattered across vendor documentation, conference papers, and forum threads into a practical framework for turning HPC telemetry into actionable insight. Modern HPC environments - Slurm clusters, GPU-dense AI systems, Lustre and GPFS storage, InfiniBand and Slingshot fabrics - generate more data than any team can manually interpret. The result is wasted node-hours, failed simulations, hidden storage bottlenecks, fabric congestion, and GPU failures that surface only after days of runtime. This book provides a complete operational approach to HPC observability through a five-layer model covering hardware, operating systems, schedulers, applications, storage, and networks. Readers learn how to build metrics pipelines for clusters from hundreds to tens of thousands of nodes; monitor GPUs with DCGM; profile MPI and OpenMP applications with PAPI and Score-P; diagnose storage and network slowdowns; create useful dashboards and alerts; and run effective incident response and post-mortems. Drawing on peer-reviewed research and real production experience, the book includes original diagrams, practical workflows, reference material, Prometheus alert examples, and a step-by-step lab environment for learning on a laptop. Written in the voice of a senior HPC engineer rather than an academic text, HPC Observability assumes readers already understand the fundamentals and focuses instead on the operational realities of running large-scale Linux, AI, and research-computing infrastructure. Read Less