Long-Term Retention RL

Long-term retention RL models D1/D7 retention and LTV as delayed rewards: the true return of today's action (e.g., a recommended item) is observed days or weeks later, so a discount factor $\gamma$ is introduced to model cross-period value, with long-term objective $J = \mathbb{E}[\sum_{t} \gamma^t r_t]$ — small $\gamma$ favors the near term, large $\gamma$ the far term, trading off against short-term metrics (e.g., instant CTR). Delayed rewards create a credit-assignment problem: returns are hard to attribute to specific rounds/actions; common fixes include temporal decomposition, surrogate rewards, and sequential-decision frameworks.

retention optimization for short-video/social/gaming apps, LTV modeling for subscriptions and ads, fixing the "CTR up but retention down" paradox in recsys; interviews ask why optimizing CTR alone hurts the long-term experience and how discounting/delayed rewards fix it.

purely short-term objectives (CTR) invite clickbait that destroys long-term retention; treating retention as a delayed reward discounted by $\gamma$ makes the policy trade some instant metrics for higher long-term value, while TD (bootstrapped) or Monte-Carlo attribution reduces the variance of delayed feedback — aligning short- and long-term metrics.

Why do delayed rewards make credit assignment hard, and what role does the discount $\gamma$ play?

Formula: $J = \mathbb{E}[\sum_t \gamma^t r_t]$ — what business objectives do $\gamma \to 0$ and $\gamma \to 1$ correspond to?

Why does optimizing CTR alone hurt retention, and how do you design rewards for both short- and long-term?

Attributing delayed feedback: TD bootstrapping vs MC, and how surrogate rewards speed up learning?

How do you plug retention as delayed feedback into bandits (delayed-reward bandits) or RL?