Sparsity has widely shown its versatility in model compression, robustness improvement, and overfitting mitigation by selectively masking out a portion of parameters. However, traditional methods to obtain such masks usually involve pre-training a dense model. As powerful foundation models become prevailing, the cost of the pre-training step can be prohibitive. In this talk, I will present our recent work on efficient methods to obtain such fantastic masks by training sparse neural networks from scratch, without the need for any dense pre-training steps.

Shiwei Liu is a Postdoctoral Fellow at the University of Texas at Austin. He obtained his Ph.D. from the Eindhoven University of Technology in 2022. His research interests cover sparsity in neural networks and efficient ML. He has over 30 publications in top-tier machine learning conferences, such as IJCAI, ICLR, ICML, NeurIPS, IJCV, UAI, and LoG. Shiwei won the best paper award at the LoG’22 conference and the Cum Laude (distinguished Ph.D. thesis) at the Eindhoven University of Technology. He has served as an area chair in ICIP‘22 and ICIP’23; and a PC member of almost all top-tier ML/CV conferences. Shiwei has co-organized two tutorials in IJCAI and ECMLPKDD, which were widely acclaimed by the audience. He has also provided more than 20 invited talks at many universities, companies, research labs, and conferences.