Alternative Effect Size Estimation in Randomized Controlled Trials with Heterogeneous Treatment Effects

Psychologists report effect sizes in randomized controlled trials to facilitate interpretation and inform clinical or policy guidance. Since commonly used effect size measures (e.g., standardized mean difference) are not sensitive to heterogeneous treatment effects, methodologists have suggested the use of an alternative effect size δ, a between-subjects causal parameter describing the probability that the outcome of a random participant in the treatment group is better than the outcome of another random participant in the control group. Although this effect size is useful, researchers could mistakenly use δ to describe its within-subject analogue, ψ, the probability that an individual will do better under the treatment than the control. Hand’s paradox describes the situation where ψ and δ are on opposing sides of 0.5: δ may imply most are helped whereas the (unknown) underlying ψ indicates that most are harmed by the treatment. The current study used Monte Carlo simulations to investigate plausible situations under which Hand’s paradox does and does not occur, tracked the magnitude of the discrepancy between ψ and δ, and explored whether the size of the discrepancy could be reduced with a relevant covariate. The findings suggested that although the paradox should not occur under bivariate normal data conditions in the population, there could be sample cases with the paradox. The magnitude of the discrepancy between ψ and δ depended on both the size of the average treatment effect and the underlying correlation between the potential outcomes, ρ. Smaller effects led to larger discrepancies when ρ < 0 and ρ = 1, whereas larger effects led to larger discrepancies when 0 < ρ < 1. It was useful to consider a relevant covariate when calculating ψ and δ. Although ψ and δ were still discrepant within covariate levels, results indicated that conditioning upon relevant covariates is still useful in describing heterogeneous treatment effects.