ProbSparse self-attention

ProbSparse self-attention is a variant of self-attention mechanism used in deep learning models for natural language processing tasks. It is designed to reduce the computational complexity of self-attention while maintaining the same level of accuracy.

The traditional self-attention mechanism computes a weighted sum of all the input tokens, which can be computationally expensive for long sequences. ProbSparse self-attention, on the other hand, only considers a subset of the input tokens for each query token, which significantly reduces the number of computations required.

The subset of input tokens is selected using a probabilistic sampling technique, where each input token is assigned a probability of being selected based on its relevance to the current query token. The most relevant tokens are more likely to be selected, while the less relevant tokens have a lower probability of being selected.

ProbSparse self-attention has been shown to be effective in reducing the computational cost of self-attention in various natural language processing tasks, including machine translation, text classification, and language modeling.