Graph Attention Network

GAT (Graph Attention Network, Veličković et al. 2018) assigns learnable importance weights to neighbors via attention. After a linear transform $Wh_i$, it scores the concatenated pair with a shared attention vector: $e_{ij} = \mathrm{LeakyReLU}(a^T[Wh_i \Vert Wh_j])$ ($\Vert$ denotes concatenation, $a$ is learnable), normalizes over neighbors with softmax to get $\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}(i)} \exp(e_{ik})}$, and aggregates $h_i' = \sigma(\sum_{j \in \mathcal{N}(i)} \alpha_{ij} Wh_j)$. Multi-head attention runs $K$ independent copies of $(W^k, a^k)$: hidden layers concatenate $h_i' = \Vert_{k=1}^{K} \sigma(\sum_{j \in \mathcal{N}(i)} \alpha_{ij}^k W^k h_j)$ (output dimension ×$K$), while the final layer averages $h_i' = \sigma(\frac{1}{K}\sum_{k=1}^{K} \sum_{j \in \mathcal{N}(i)} \alpha_{ij}^k W^k h_j)$ for stable training. Unlike GCN's fixed coefficients (determined by degree alone), GAT's coefficients are data-driven and can allocate different importance to different neighbors.

graphs with heterogeneous neighborhoods (social influence, citation importance), dynamic graphs and settings with varying node counts; interview comparisons: GAT vs GCN, GAT vs Transformer self-attention, and multi-head handling.

GCN's aggregation weights $\tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$ are constants fixed by topology, limiting expressivity and making it sensitive to degree distribution. GAT makes weights dynamic w.r.t. features via LeakyReLU attention + neighborhood softmax — same-degree neighbors can still be differentiated (structurally indistinguishable node pairs become distinguishable), effectively restricting Transformer self-attention to the neighborhood subgraph ($\mathcal{N}(i)$ instead of all $L$ positions), giving both inductive capacity and stronger fitting power; the multi-head mechanism provides the same viewpoint-stabilization as Transformer multi-head attention.

Test Your Knowledge: Practice Questions for "Graph Attention Network"

Single choice pitfall questions with instant feedback and mistake tracking.