About this question
Medium · Linear Algebra & Machine Learning · Quant Trader interview question · attention-mechanism, machine-learning, linear-algebra, transformer
In a self-attention mechanism, we have query (Q), key (K), and value (V) matrices. Given an input sequence represented as a matrix $X$, these matrices are derived through linear transformations: $Q = XW_Q$, $K = XW_K$, and $V = XW_V$, where $W_Q$, $W_K$, and $W_V$ are learnable weight matrices. How is the output of the self-attention mechanism computed, assuming the dimension of the keys is $d_k$?