Exploring HiFloat8: A Tapered Format Complementing the FP8 Ecosystem for Robust Model T... - Y. Zhao

Standard FP8 formats suffer from frequent gradient overflows and heavy reliance on complex Delayed Scaling, which often lead to training instability or suboptimal convergence in large models. This session introduces HiFloat8 (HiF8) — a tapered precision format that offers an alternative approach to managing dynamic range. This "natural" alignment with neural network weight/gradient distributions allows HiF8 to capture high-magnitude outliers without the aggressive scaling required by standard FP8.We explore how HiF8 can works in the training and inference procedure.We will demonstrate the implementation of HiF8 within the ecosystem on vllm and deepspeed. They allow developers to evaluate performance of Hif8 on GPUs. Also, we will give an analysis of training stability and final loss parity where HiF8 provides relatively the same accuracy and 1.5-1.7 times GEMM performance than FP16. Finally, we will share insights from our ongoing collaboration on dedicated hardware support for HiF8.