Graph Attention Network

GAT (Graph Attention Network, Veličković et al. 2018) assigns learnable importance weights to neighbors via attention. After a linear transform $Wh_i$, it scores the concatenated pair with a shared attention vector: $e_{ij} = \mathrm{LeakyReLU}(a^T[Wh_i \Vert Wh_j])$ ($\Vert$ denotes concatenation, $a$ is learnable), normalizes over neighbors with softmax to get $\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}(i)} \exp(e_{ik})}$, and aggregates $h_i' = \sigma(\sum_{j \in \mathcal{N}(i)} \alpha_{ij} Wh_j)$. Multi-head attention runs $K$ independent copies of $(W^k, a^k)$: hidden layers concatenate $h_i' = \Vert_{k=1}^{K} \sigma(\sum_{j \in \mathcal{N}(i)} \alpha_{ij}^k W^k h_j)$ (output dimension ×$K$), while the final layer averages $h_i' = \sigma(\frac{1}{K}\sum_{k=1}^{K} \sum_{j \in \mathcal{N}(i)} \alpha_{ij}^k W^k h_j)$ for stable training. Unlike GCN's fixed coefficients (determined by degree alone), GAT's coefficients are data-driven and can allocate different importance to different neighbors.

graphs with heterogeneous neighborhoods (social influence, citation importance), dynamic graphs and settings with varying node counts; interview comparisons: GAT vs GCN, GAT vs Transformer self-attention, and multi-head handling.

GCN's aggregation weights $\tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$ are constants fixed by topology, limiting expressivity and making it sensitive to degree distribution. GAT makes weights dynamic w.r.t. features via LeakyReLU attention + neighborhood softmax — same-degree neighbors can still be differentiated (structurally indistinguishable node pairs become distinguishable), effectively restricting Transformer self-attention to the neighborhood subgraph ($\mathcal{N}(i)$ instead of all $L$ positions), giving both inductive capacity and stronger fitting power; the multi-head mechanism provides the same viewpoint-stabilization as Transformer multi-head attention.

Write the GAT attention formula $\alpha_{ij} = \frac{\exp(\mathrm{LeakyReLU}(a^T[Wh_i \Vert Wh_j]))}{\sum_{k \in \mathcal{N}(i)} \exp(\mathrm{LeakyReLU}(a^T[Wh_i \Vert Wh_k]))}$ and explain the roles of $\Vert$ concatenation, LeakyReLU, and softmax.

Core difference GAT vs GCN: GCN's coefficients are fixed by topology (degree-based); how does GAT distinguish same-degree neighbors (structurally symmetric but semantically different pairs)?

Multi-head handling: why do hidden layers concatenate heads while the last layer averages them? How does multi-head differ from single-head in training stability and capacity?

Why is softmax applied only over neighbors $\mathcal{N}(i)$ instead of the whole graph? Compare with Transformer's global softmax in terms of complexity and locality.

Similarities and differences between GAT and Transformer self-attention: where are query/key/value, how is graph structure injected via masking, and when can one be converted into the other?