BitNet: Scaling 1-bit Transformers for Large Language Models

BitLinear §

Quantization §

Binarization of weights §

First step of bit linear is to binarize the weights using Signum Function

A scaling factor of $\beta$ is used after Binarization to reduce the L2 Error between real and binarized weights

Mathematical steps:

$$\tilde{W}=\mathrm{Sign}(W-\alpha )$$

This centralizes the weights to 0 mean 2.

$$Sign(W_{ij})=\{\begin{array}{l}+1,\text{if }{W}_{ij}>0\\ -1,\text{if }{W}_{ij}\le 0\end{array}$$

$$\alpha =\frac{1}{nm}\sum _{ij}{W}_{ij}$$

Semantic understanding of the math:

• $W\in {\mathbb{R}}^{nxm}$ - weight matrix (for linear layer)
• $Wij$ - one specific entry of that matrix
• Sign - Signum Function
• $n$ - number of rows
• $m$ - number of columns
• $\alpha$ - The mean (average) of all the weights in W

Absmax quantization: Scales activations into the range $[-Q_b,Q_b](Q_b=2^{b-1})$ 4.

$$\tilde{x}=Quant(x)=Clip(x\times \frac{Q_b}{\gamma },-Q_b+ϵ,Q_b-ϵ)$$

$$Clip(x,a,b)=\max (a,min(b,x)),\gamma =\Vert x{\Vert }_{\infty }$$

Semantic Understanding of the math:

the clip function forces (clips) the value into the interval $[a,b]$

$$\text{Clip}(x,a,b)=\{\begin{array}{l}a,x<a\\ x,a\le x\le b\\ b,x>b\end{array}$$

What are $Q_b,b$ and the range

• $Q_b=2^{b-1}$
• Target range: $[-Q_b,Q_b]$

Example:

• If b = 8, then $Q_b=2^7=128$
• Range is roughly $[-128,128]$

This is the integer range they want the activations to lice in after scaling + clipping

The $\pm ϵ$ makes the bounds slightly inside that range $[-Q_b+ϵ,Q_b-ϵ]$ to avoid edge cases like overflow later cast or round to an integer type

$\gamma =\Vert x{\Vert }_{\infty }$

This means that gamma is equal to the maximum absolute value among all the entries of x

Eg. x = [-10, 5, 3, 2, 6] The gamma value would be 10