The mechanism through which position bias impacts a model is that:

• some items tend to appear higher / more frequently on screen
• thus the model will assign a higher score to those specific items (or when it sees features associated with those specific items) more than what is due i.e. confounding.

there are two things:

• correcting the model for position bias
• correcting evaluation metrics for position bias

We assume the following causal model:

Let:

• $u$ be a user and $i$ be an item. $x = (u, i)$ represents the context.
• $k \in \{1, 2, \dots, K\}$ be the rank/position of item $i$ displayed to user $u$.
• $R \in \{0, 1\}$ be the true relevance (unobserved). $R=1$ if the user would engage with the item provided they examined it.
• $O \in \{0, 1\}$ be the examination (unobserved). $O=1$ if the user examines the item at position $k$.
• $C \in \{0, 1\}$ be the click (observed).

This model asserts that a click occurs if and only if the item is both examined and relevant: $C = O \cdot R$

The probability of a click is given by:

Because $C = O \cdot R$, and because $O \perp\!\!\!\perp R \mid x, k$, the expectation over their joint distribution factors into the product of their marginal expectations:

(since $O$ and $R$ are Bernoulli variables, their expectations are their probabilities)

Thus:

I - inverse propensity weighting

Instead of learning a biased quantity $\mathbb{P}(C=1|x,k)$, learn true relevance $\mathbb{P}(R=1 | x)=\mathbb{E}\bigg[\frac{C}{p_k}\vert x\bigg]$ (which is an unbiased estimator for relevance).

We can formulate the expected loss of a single item with respect to that item’s relevance:

$\mathbb{E_R}[l(y,\hat{y}) | x] = \mathbb{P}(R=1 | x)l(1,\hat{y}) + (1-\mathbb{P}(R=1 | x))l(0, \hat{y})$ where $l(y, \hat{y})$ is the loss function.

Since we have an unbiased estimator for the item’s relevance, we can plug it into the above formula:

Because $\ell(1,\hat{y})$ and $\ell(0,\hat{y})$ are deterministic functions with respect to the model's predictions, we can pull them inside the expectation over $C$ by the linearity of expectation:$\mathbb{E_R}[l(y,\hat{y}\vert x)] = \mathbb{E}_C\left[ \frac{C}{p_k}\ell(1,\hat{y}) + \left(1-\frac{C}{p_k}\right)\ell(0,\hat{y}) \mathrel{\Big|} x, k \right]$

Finally, an estimator of the above loss function is:

Where we can replace $l$ by the log loss.

References:

• [https://decisionmaking4ir.github.io/WSDM-2022/papers/Suguru.pdf](Unbiased Recommender Learning from Biased Graded Implicit Feedback)

II - Late fusion position feature

We still assume the same model:

and split the model into two towers:

• the main tower: a deep neural network $f_{\theta}(x)$ that takes only user and item features. This parameterizes the true relevance.

2. the shallow tower: a very simple network (just an embedding layer or a single linear layer) $g_{\phi}(k)$ that takes only the position $k$ (and potentially other bias features like device type). This parameterizes the observation probability.

We fuse these two towers only at the very end, in logit space, right before the final sigmoid activation function:

$\text{logit}(C) = f_{\theta}(x) + g_{\phi}(k)$ Implementation details from Google's paper:

• 10% position feature drop-out rate to prevent over-reliance on position
• at serving time, position is treated as missing
• they actually cross position feature with device info because different types of devices exhibit different position biases

Ref: Recommending What Video to Watch Next: A Multitask Ranking System

Question: don't we still learn a biased quantity, if we learn the click label?

Instead of correcting click for position bias, we force the model's architecture to mirror the causal mechanism that generated the biased data, hoping that the network naturally disentangles the hidden variables. We are performing Maximum Likelihood Estimation (MLE) of a model that separates the relevance parameters from the position bias parameters. Mechanically, the shallow tower provides the easy gradient that items at $k=1$ are universally clicked more often, while the main tower is forced to explain whatever variation is left over.

Mesuring position bias p_k

We need to actually measure position bias to generate the corrected biased used in inverse propensity weighting

Ref: Position Bias Estimation for Unbiased Learning to Rank in Personal Search

The paper depicts 3 approaches to estimate the propensity of every position k:

• Randomize TopN (result randomization): top N documents are randomly shuffled
• Randomize Pair (result randomization, smaller intervention): randomly swap adjacent positions
• Estimation from Regular Clicks using the Expectation Maximization algorithm: user is not exposed to any randomized results, but accuracy is lower

Randomize Top N

the relevance component $r_{q,d}$ is hidden. We can average out this variable by randomizing the result list. For a randomized dataset where the top N documents are randomly shuffled before showing them to users, let $R_k$ be the subset of collected logs from position $k$.

since $\sum_{q,d\in R_k}\mathbb{P}(q,d)$ is constant across all $N$ positions (can be thought of as the average relevance of the candidate pool).

We can just normalize all propensities and return $p_k/p_0$

Randomize Pair

Let us denote $d_k$ a document that is originally at position $k$. Recall that our model assumes $\mathbb{P}(C\vert q,d_k)=p_k r_{q,d_k}$

By swapping documents $k$ and $k+1$ in the final results, the user now needs to examine position $k+1$ to see document $d_k$:

Thus:

Meaning, if we can estimate the probabilities on the left-hand side, we can estimate the position bias of $k+1$ with respect to $k$.

Now, notice that both quantities on the left-hand side are click through rates (conditioned on query, document and whether we swap or not). Let us denote $D_k$ the set of documents that were originally in position $k$, for a query $q$:

Similarly: $\mathbb{E}[C\vert D_k, \text{swap}] = p_{k+1}\sum_{q,d\in D_k}r_{q,d}\mathbb{P}(q,d)$

both expectations can be estimated by taking the sum of clicks, conditioned on the proper variables. i.e.:

The relative ratio between any two positions is given by the chaining the ratios: $\frac{p_k}{p_1} = \frac{p_k}{p_{k-1}}\dots\frac{p_2}{p_1}$

We can see the limitations of the position bias model by performing RandomizeTopN:

• after L2 scoring (L2 set)
• without L2 scoring (L1 set)

the propensity values will display different curves, showing that they are not independent of the relevance of the document set. In the L2 set, whatever item randomization drops into position 0 is likely to be good, so the user is often satisfied immediately and stops scanning. Examination at lower positions collapses which explains the steep decay. The next step would be to move to a cascade-aware or relevance-conditioned propensity model.

When applying filters for analysis (e.g. US users only, initial impressions without scrolling, 9+ results), make sure they are all unaffected by the treatment (pre-treatment filters).

Some filters are necessary. For example, some sessions might display less than $N$ candidates. This adds a bias towards earlier positions and we will only retain sessions with more than $N$ candidates displayed.

other methods for correcting position bias

1. randomized exploration

Rather than correcting observational bias retrospectively, this approach alters the data generation process. A small fraction of user traffic is subjected to a policy where items are ranked uniformly at random.

recall the original data generation process:

r -> k -> o

 \       /

  \     /

   v   v

   click 

this treatment removes the causal link between relevance (r) and position (k) meaning:

whereas before:

Under a purely random policy, the position assignment $k$ is perfectly independent of the item covariates $x$. Consequently, $p_k$ becomes a known uniform constant, and the resulting interaction logs are mathematically guaranteed to be an unbiased sample of true relevance. This data can be used as a pristine validation set or to perfectly calibrate propensity scores for the rest of the dataset.

sequential / cascade click models

Until now, our model (click = observation x relevance) was point-wise: it assumed that the probability of observing the item at position $k$ is independent of the items placed above it. Instead, cascade models use a Markovian process where the user scans a list linearly from top to bottom.

A user observes position $k$ iff they observed $k-1$ and did not find it relevant enough to terminate their search.

instead of being fixed, the propensity $p_k$ is context-dependent.

other approaches worth looking into:

• heckman sample selection (mills ratio)
• unbiased pairwise and listwise ranking: debias ranking functions like LambdaMART that optimize relative order by applying inverse propensity weighting to the pairwise gradient.

Evaluation

offline metrics (AUC)

Since click is confounded by position bias, so are offline metrics. We expect a debiased model to:

• perform worse on the regular (biased) logs
• perform better on the debiased logs

We can correct AUC inverse propensity weighting (IPS-AUC) and determine if this improves the offline / online correlation (AUC percent difference vs CTR percent difference for example).

online metrics

if the only difference between Model A and Model B is how we handle the position bias correction (IPW correction, late fusion position feature), then any engagement gains in the A/B test are strictly caused by position bias correction.

However this does not tell us the extent to which we successfully eliminated position bias from our model. I haven't found any convincing way to evaluate this.

head vs tail analysis

If we expect the change to reduce popularity bias and increase exposure to tail items, we can group items by historical exposure and verify that we see most gains on tail items.