My expectation is that this regression to the mean should not apply to strong effects. Where I define strong by: the strength is enough that significance level and publication criteria are unimportant. In this case, I would expect half the results to return stronger.
The first result was a random sample, and the second result was a random sample. If there's no outside bias from publication cut-off, there should be a 50% chance that either is higher.
It's concerning if the strong results consistently re-test weaker. That shows systematic bias.
My expectation is that this regression to the mean should not apply to strong effects. Where I define strong by: the strength is enough that significance level and publication criteria are unimportant. In this case, I would expect half the results to return stronger.
The first result was a random sample, and the second result was a random sample. If there's no outside bias from publication cut-off, there should be a 50% chance that either is higher.
It's concerning if the strong results consistently re-test weaker. That shows systematic bias.