I8Hard★ Quantization Essential · AWQ CorePart I · Inference, Decoding & Quantization

AWQ Activation-Aware Group-Wise INT4 Quantization

⚡Flashcard Drill➔

Combine group-wise INT4 weight quantization with activation-aware per-channel scaling to reduce rounding loss on salient channels.

⏱️ Time: O(B * D_in * D_out)
💾 Space: 分组缩放因子额外开销约 0.5 bit/参数
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def groupwise_int4_quantize_dequant(w: np.ndarray, group_size: int = 4) -> np.ndarray:
"""
Symmetric group-wise INT4 quantization & dequantization over weight shape (in_dim, out_dim).
Splits input channels into blocks of `group_size` so outliers stay localized.
"""
in_dim, out_dim = w.shape
assert in_dim % group_size == 0
num_groups =
wg = w.reshape(num_groups, group_size, out_dim)
max_abs = np.max(np.abs(wg), axis=1, keepdims=True)
scales = np.where(max_abs > 0, max_abs / 7.0, 1.0)
q = np.clip(np.round(wg / scales), -7, 7)
return (q * scales).reshape(in_dim, out_dim)
def awq_salient_channel_mse(x: np.ndarray, w: np.ndarray, scale_vec: np.ndarray, group_size: int = 4) -> float:
"""
Evaluate AWQ activation-aware scaling: multiply salient input channels of W by s > 1
before INT4 group quantization and divide X by s at runtime.
"""
s = scale_vec[:, np.newaxis] # (in_dim, 1)
w_scaled_q = groupwise_int4_quantize_dequant(w * s, group_size=group_size)
x_scaled = x / scale_vec[np.newaxis, :]
y_ref = x @ w
y_awq = x_scaled @ w_scaled_q
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does AWQ differ from GPTQ in 4-bit weight-only quantization?
Next: Factorization Machine (FM) O(kd) Implementation →
💡

Core Mental Anchor / Mnemonic

Group weights in blocks of G; scale salient channels up before INT4 rounding and fold 1/s into activations.

📐 Mathematical Derivation & Core Formula

Y=XW=(X⋅diag(s)−1)(diag(s)⋅W),Q(w⋅s)/sY = X W = (X \cdot \mathrm{diag}(s)^{-1}) (\mathrm{diag}(s) \cdot W), \quad Q(w \cdot s) / s
### Protecting Salient Weights via Activation Magnitude
AWQ (Lin et al., 2023) observes that only ~1% of weight channels drive most output error—specifically those paired with high-magnitude activations.
By multiplying salient weight channels by s>1s > 1 before group-wise INT4 rounding and dividing incoming activations by ss at inference, XW=(X/s)(W⋅s)X W = (X/s)(W \cdot s) holds identically while lowering relative quantization noise on critical channels.

🔄 Tensor Dimensions & Shape Flow

w*s -> reshape groups -> INT4 round & scale -> dequant -> (x/s) @ w_q -> MSE check

🛡️ Numerical Stability & Pitfalls Checklist

  • Tune scaling factors carefully; overly large ss increases group max and hurts non-salient channels sharing that group.
  • Fuse division by ss into the preceding LayerNorm/RMSNorm scale parameter so inference adds zero runtime overhead.

💻 Reference Code Implementation

import numpy as np

def groupwise_int4_quantize_dequant(w: np.ndarray, group_size: int = 4) -> np.ndarray:
    """
    Symmetric group-wise INT4 quantization & dequantization over weight shape (in_dim, out_dim).
    Splits input channels into blocks of `group_size` so outliers stay localized.
    """
    in_dim, out_dim = w.shape
    assert in_dim % group_size == 0
    num_groups = in_dim // group_size
    wg = w.reshape(num_groups, group_size, out_dim)
    max_abs = np.max(np.abs(wg), axis=1, keepdims=True)
    scales = np.where(max_abs > 0, max_abs / 7.0, 1.0)
    q = np.clip(np.round(wg / scales), -7, 7)
    return (q * scales).reshape(in_dim, out_dim)

def awq_salient_channel_mse(x: np.ndarray, w: np.ndarray, scale_vec: np.ndarray, group_size: int = 4) -> float:
    """
    Evaluate AWQ activation-aware scaling: multiply salient input channels of W by s > 1
    before INT4 group quantization and divide X by s at runtime.
    """
    s = scale_vec[:, np.newaxis]  # (in_dim, 1)
    w_scaled_q = groupwise_int4_quantize_dequant(w * s, group_size=group_size)
    x_scaled = x / scale_vec[np.newaxis, :]
    y_ref = x @ w
    y_awq = x_scaled @ w_scaled_q
    return float(np.mean((y_ref - y_awq) ** 2))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
np.random.seed(7)
x = np.random.randn(8, 8)
x[:, 0] *= 25.0  # salient outlier channel 0
w = np.random.randn(8, 6) * 0.5
naive_mse = awq_salient_channel_mse(x, w, np.ones(8), group_size=4)
scales = np.ones(8)
scales[0] = 4.0  # boost salient channel before quantization
awq_mse = awq_salient_channel_mse(x, w, scales, group_size=4)
assert awq_mse < naive_mse * 0.5, f"AWQ scaling should cut quantization error: awq={awq_mse}, naive={naive_mse}"
print("✓ AWQ activation-aware group INT4 quantization assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does AWQ differ from GPTQ in 4-bit weight-only quantization?
GPTQ iteratively compensates quantization error per column using approximate second-order Hessians (risking overfitting to calibration samples), whereas AWQ preserves generalization by solving lightweight channel-wise scaling based purely on activation statistics.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: RadixAttention Prefix Cache Tree & LRU EvictionBack to Part I & All 105Next: Factorization Machine (FM) O(kd) Implementation →