MMoE Multi-Gate Mixture-of-Experts

MMoE (Multi-gate Mixture-of-Experts, proposed by Google at KDD 2018) is a multi-task learning architecture designed to resolve negative transfer across weakly correlated or conflicting objectives in recommendation ranking; while traditional Shared-Bottom networks force all tasks to share monolithic lower layers, MMoE constructs a shared pool of $E$ parallel expert sub-networks ($f_1, \dots, f_E$) and equips each distinct task $k$ (e.g., Task 1: CTR, Task 2: CVR, Task 3: Watch Time) with its own Softmax Gating Network $g^k(x) = \text{softmax}(W_g^k x)$; each task dynamically aggregates weighted expert representations $y_k = h^k\left(\sum_{i=1}^E g_i^k(x) f_i(x)\right)$ before feeding task-specific output towers.

Multi-objective industrial ranking optimizing simultaneous targets (Click, Like, Share, Purchase, Dwell Time) in e-commerce and social feeds.

Shared-Bottom networks suffer catastrophic performance degradation (Negative Transfer) when task relationships are orthogonal or antagonistic; MMoE's multi-gating routes dynamic weight distributions per task, allowing expert networks to specialize flexibly.

Derive MMoE's multi-gate formulation $y_k = h^k(\sum_{i=1}^E g_i^k(x) f_i(x))$ and contrast with standard single-gate MoE and Shared-Bottom?

Why does MMoE's gating network automatically orthogonalize expert attention when tasks are negatively correlated (e.g. clickbait)?

How do dynamic loss balancing algorithms (GradNorm / Uncertainty Weighting) dynamically tune task weights $w_k$ in multi-task loss?

Analyze how the number of experts $E$ impacts parameter bloat and GPU serving latency in production?

What are the structural limitations of MMoE (all experts are shared, lacking task-exclusive isolation), leading to Tencent's PLE?