Paper Link: https://arxiv.org/abs/2104.09864

Formula:

$$f_{\{q,k\}}(x_m,m)=\left(\begin{array}{cc}\cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{array}\right)\left(\begin{array}{cc}{W}_{\{q,k\}}^{(11)}& W_{\{q,k\}}^{(12)}\\ W_{\{q,k\}}^{(21)}& W_{\{q,k\}}^{(22)}\end{array}\right)\left(\begin{array}{c}{x}_m^{(1)}\\ x_m^{(2)}\end{array}\right)$$

Key Components:

• $\theta \in \mathbb{R}$: Fixed non-zero scalar
• $x_m$: Embedding vector at position m, in 2D as $(x_m^{(1)},x_m^{(2)})$

Paper Details §

$$a_{m,n}=\frac{exp(\frac{q_m^T{k}_n}{\sqrt{d}})}{{\sum }_{j=1}^{N}exp(\frac{q_m^T{k}_j}{\sqrt{d}})}$$

In the paper when talking about attention notice how they transpose the query values instead of the key value…

Advantage over others §

Sinusoidal Positional Encoding are not preferred compared to Rotary Position Embeddings because there’s no clear pattern established as the position counter goes up

The output changes significantly (like magnitude)