Self Attention

Self Attention is an application of Scaled Dot-Product Attention

Self Attention is when you apply Scaled Dot-Product Attention to a single sequence, using the same input for Q, K, and V

Self attention lets each work “look at” other words to decide what’s important using Scaled Dot-Product Attention So:

$\text{SelfAttention}(X)=\text{Attention}(Q=X{W}^Q,K=X{W}^K,V=X{W}^V)$

$W^Q,W^K,W^V$ are learned weight matrices