Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Replacing dense layers with Sparse-IFT leads to significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL), both matching larger dense model variants with 2x or more FLOPs. To…
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models
Presented at the ICLR 2023 Workshop on Sparsity in Neural Networks. In this work, we show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training (Sparse Pre-training) and then recover the representational capacity by allowing the zeroed weights to…
