Buket Benek Gursoy & Jake Mac Uilliam: Performance Engineering for Python-based AI Workflows in HPC

Performance Engineering for Python-based AI Workflows in HPC Buket Benek Gursoy and Jake Mac Uilliam The Irish Centre for High-End Computing (ICHEC) is a public body hosting Ireland’s supercomputing national service and data services supporting Irish and European research across climate, weather, health and the environment. The Performance Engineering team in ICHEC supports the optimisation of scientific codes and workflows through the development and application of tooling. Python is rapidly emerging as a key language for scientific computing and AI and its adoption in High Performance Computing (HPC) environments is accelerating. Its simplicity, readability, and expansive ecosystem make it ideal for developing and deploying machine learning workflows. However, achieving efficient performance at HPC scale requires careful profiling, tuning and the use of the right tools especially as Python introduces challenges not typically encountered with traditional HPC languages. This talk provides an overview of performance analysis and optimisation tools tailored for Python-based AI workflows in HPC. We will cover lightweight profilers like cProfile[1] as well as framework specific tools such as the PyTorch Profiler [2] which enables deep performance insight across CPU and GPU workloads which can then be visualised using TensorBoard [3] or other trace viewers. Beyond framework specific tools, standard HPC profiling and tracing tools such as Score-P [4] and Extrae [5] and visualisation tools like Paraver [6] can be applied to Python-based AI workflows. For low level performance monitoring, LIKWID [7] and PAPI [8] are useful for capturing hardware counter data. For GPU specific analysis, NVIDIA Nsight Systems [9] and Nsight Compute [10] provide in-depth performance insights. In this talk, we will present an overview of profiling and optimisation tools relevant to Python-based AI workflows in HPC. We will also share illustrative use cases to demonstrate how these tools can be applied in practice. References: [1]. https://docs.python.org/3/library/profile.html [2]. https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html [3]. https://www.tensorflow.org/tensorboard [4]. https://www.vi-hps.org/projects/score-p/overview/overview.html [5]. https://tools.bsc.es/extrae [6]. https://tools.bsc.es/paraver [7]. https://github.com/RRZE-HPC/likwid [8]. https://icl.utk.edu/papi/ [9]. https://developer.nvidia.com/nsight-systems [10]. https://developer.nvidia.com/nsight-compute