#language-modeling

#language-modeling

Dong et al. (2019) and Tay et al. (2022) train on a mixture of denoising tasks with different attention masks (full, causal and prefix attention) to bridge the performance gap with next token pretraining on generative tasks.

This work innovates by advancing beyond next-token prediction, enabling multi-token prediction, which allows the model to forecast multiple future tokens simultaneously. This is shown to enhance performance and inference speed.