Farid Asadi

Growth, Experiment Driven.

Simpson’s Paradox: Explained in Simple Terms

Posted on

The Simpson’s Paradox occurs when several groups of data show a direction but the effect reverses when they are combined.

A real-life example of this paradox is “Kidney Stone Treatment”. After comparing the success rates of two treatments for kidney stones, the following results can be seen:

Based on the overall success rateTreatment B is the obvious choice since it has a higher success rate. Things get nasty when we segment treatments according to stone size. Now the data are reversed, Treatment A appears to be the better treatment.

Which treatment should we choose?

The paradox can be understood by choosing Treatment A if you have a small stone, and Treatment A again if you have a large stone.

When does this paradox happen?

  1. Different sample sizes. Due to the high number of cases in groups 2 and 3, the total number heavily depends on them.
  2. Confounding variables. The stone size is a confounding variable here. Since the success rate is influenced more by the severity of the case (Stone Size) than treatment choice (Success rates are higher in small stone sizes).

The next time you’re segmenting look for:

  • The numbers/sample sizes alongside the percentages (Avinash Kaushik’s mantra).
  • Factors influencing the data that are not shown
  • Create causal diagrams or identify confounding variables.

Resources:

Read On

  1. Value–Action Loops in Growth; How Value Creation Generates User Decisions?
  2. Strategy Experimentation: How to Choose a Strategy?
  3. You Lose Your Focus Here!
  4. Price–Information Relationship in CRO
  5. Applying The Loss Aversion Principle on Pricing Pages
  6. Why Google Analytics 4 Can’t Go Far?
  7. Best Practices in Conversion Optimization