Combining denoising tasks and different attention masks improves the language modeling capabilities of neural networks. New training methods, like span corruption, allow better predictions by enabling causal learning through teacher forcing. These advancements address performance gaps compared to traditional next token prediction approaches. Permuted sequences contribute to model training by requiring predictions based on mixed past and future context. However, significant limitations exist in masking more than 15% of text tokens due to information loss, showcasing the balance needed in training methodologies.
Dong et al. (2019) and Tay et al. (2022) train on a mixture of denoising tasks with different attention masks (full, causal and prefix attention) to bridge the performance gap with next token pretraining on generative tasks.
Unlike UniLM, the span corruption objective replaces spans of tokens with special tokens for the encoder and the decoder then predicts the contents of those spans, enabling full causal training with teacher forcing.
Collection
[
|
...
]