I8Hard★ Quantization Essential · AWQ CorePart I · Inference, Decoding & Quantization
AWQ Activation-Aware Group-Wise INT4 Quantization
Combine group-wise INT4 weight quantization with activation-aware per-channel scaling to reduce rounding loss on salient channels.
⏱️ Time:
O(B * D_in * D_out)💾 Space:
分组缩放因子额外开销约 0.5 bit/参数🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def groupwise_int4_quantize_dequant(w: np.ndarray, group_size: int = 4) -> np.ndarray:
"""
Symmetric group-wise INT4 quantization & dequantization over weight shape (in_dim, out_dim).
Splits input channels into blocks of `group_size` so outliers stay localized.
"""
in_dim, out_dim = w.shape
assert in_dim % group_size == 0
num_groups =
wg = w.reshape(num_groups, group_size, out_dim)
max_abs = np.max(np.abs(wg), axis=1, keepdims=True)
scales = np.where(max_abs > 0, max_abs / 7.0, 1.0)
q = np.clip(np.round(wg / scales), -7, 7)
return (q * scales).reshape(in_dim, out_dim)
def awq_salient_channel_mse(x: np.ndarray, w: np.ndarray, scale_vec: np.ndarray, group_size: int = 4) -> float:
"""
Evaluate AWQ activation-aware scaling: multiply salient input channels of W by s > 1
before INT4 group quantization and divide X by s at runtime.
"""
s = scale_vec[:, np.newaxis] # (in_dim, 1)
w_scaled_q = groupwise_int4_quantize_dequant(w * s, group_size=group_size)
x_scaled = x / scale_vec[np.newaxis, :]
y_ref = x @ w
y_awq = x_scaled @ w_scaled_q
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does AWQ differ from GPTQ in 4-bit weight-only quantization?
💡
Core Mental Anchor / Mnemonic
Group weights in blocks of G; scale salient channels up before INT4 rounding and fold 1/s into activations.
📐 Mathematical Derivation & Core Formula
### Protecting Salient Weights via Activation Magnitude
AWQ (Lin et al., 2023) observes that only ~1% of weight channels drive most output error—specifically those paired with high-magnitude activations.
By multiplying salient weight channels by before group-wise INT4 rounding and dividing incoming activations by at inference, holds identically while lowering relative quantization noise on critical channels.
AWQ (Lin et al., 2023) observes that only ~1% of weight channels drive most output error—specifically those paired with high-magnitude activations.
By multiplying salient weight channels by before group-wise INT4 rounding and dividing incoming activations by at inference, holds identically while lowering relative quantization noise on critical channels.
🔄 Tensor Dimensions & Shape Flow
w*s -> reshape groups -> INT4 round & scale -> dequant -> (x/s) @ w_q -> MSE check
🛡️ Numerical Stability & Pitfalls Checklist
- Tune scaling factors carefully; overly large increases group max and hurts non-salient channels sharing that group.
- Fuse division by into the preceding LayerNorm/RMSNorm scale parameter so inference adds zero runtime overhead.
💻 Reference Code Implementation
import numpy as np
def groupwise_int4_quantize_dequant(w: np.ndarray, group_size: int = 4) -> np.ndarray:
"""
Symmetric group-wise INT4 quantization & dequantization over weight shape (in_dim, out_dim).
Splits input channels into blocks of `group_size` so outliers stay localized.
"""
in_dim, out_dim = w.shape
assert in_dim % group_size == 0
num_groups = in_dim // group_size
wg = w.reshape(num_groups, group_size, out_dim)
max_abs = np.max(np.abs(wg), axis=1, keepdims=True)
scales = np.where(max_abs > 0, max_abs / 7.0, 1.0)
q = np.clip(np.round(wg / scales), -7, 7)
return (q * scales).reshape(in_dim, out_dim)
def awq_salient_channel_mse(x: np.ndarray, w: np.ndarray, scale_vec: np.ndarray, group_size: int = 4) -> float:
"""
Evaluate AWQ activation-aware scaling: multiply salient input channels of W by s > 1
before INT4 group quantization and divide X by s at runtime.
"""
s = scale_vec[:, np.newaxis] # (in_dim, 1)
w_scaled_q = groupwise_int4_quantize_dequant(w * s, group_size=group_size)
x_scaled = x / scale_vec[np.newaxis, :]
y_ref = x @ w
y_awq = x_scaled @ w_scaled_q
return float(np.mean((y_ref - y_awq) ** 2))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
np.random.seed(7)
x = np.random.randn(8, 8)
x[:, 0] *= 25.0 # salient outlier channel 0
w = np.random.randn(8, 6) * 0.5
naive_mse = awq_salient_channel_mse(x, w, np.ones(8), group_size=4)
scales = np.ones(8)
scales[0] = 4.0 # boost salient channel before quantization
awq_mse = awq_salient_channel_mse(x, w, scales, group_size=4)
assert awq_mse < naive_mse * 0.5, f"AWQ scaling should cut quantization error: awq={awq_mse}, naive={naive_mse}"
print("✓ AWQ activation-aware group INT4 quantization assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does AWQ differ from GPTQ in 4-bit weight-only quantization?
GPTQ iteratively compensates quantization error per column using approximate second-order Hessians (risking overfitting to calibration samples), whereas AWQ preserves generalization by solving lightweight channel-wise scaling based purely on activation statistics.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.