(MMoE by Google, 2018)

Multi-task learning: recommendation systems often need to optimize multiple objectives at the same time. E.g. click through rate, watch time and explicit feedback (rating).

However in practice, multi-task learning models underperform single-task models on their respective sub-tasks because they fail to model relationships among tasks (some can be conflicting). MMoE explicitly models task relationships and learns task-specific functionalities to leverage shared representations by allocating parameters instead of adding many new parameters per task.

Given $K$ tasks, model consits of a shared-bottom network ($n$ multi-layer perceptrons) $f$ which follows the input layer, and $K$ tower networks $h^k$ where $k=1,\dots,K$.

The output of the shared bottom $f$ is the output of $n$ expert networks implemented as a single layer each. $n=8$ in the paper.

Each tower network (also a single layer) is associated with its own gate $g^k$ that determines how to weigh each expert network: $g^k = \text{softmax}(W_{g,k}. x)$ where $W_{g,k}\in\mathbb{R}^{n,d}$

Thus, the re-weighted shared bottom output for tower network $k$ is:

$f^k(x)=\sum_{i}^n g^k(x)_i f(x)_i$.

And the output of task $k$ is: $y_k = h^k(f^k(x))$.

Each gating network helps the model learn task-specific information.

Question: what loss function do they use? If task-specific they would most likely use binary cross-entropy for click through rate and mean squared error for watch time.