Scalable Softmax is Superior for Attention

Paper Link: https://arxiv.org/pdf/2501.19399

$$\text{Scalable Softmax Function}:z_i=\frac{e^{(\text{s }\text{log }n)z_i}}{{\sum }_{j=1}^n{e}^{(\text{s log }n)z_j}}$$

Pretty cool paper, talks about the advantages of Scalable Softmax over normal Softmax in attention

Uses tests like Needle in a Haystack

Concepts in the Paper §

Attention Fading - As the input vector size grows running softmax on the vector the resulting probability distribution becomes increasingly flat.