The caveat here is that it depends on what you mean by "test" and what "insights" you are interested in.
If you are interested in (for example) determining the optimal price and only 1/20 users buy something... you still probably need about 1,000 X (price points you want to test).
In that "test" you are basically trying to uncover the demand curve (or points on it). It's a statistical question.
Say you have a dating app, and you are trying a new matching algorithim. It will also take thousands of matches before you have the data to make determinations.
All that said, I totally agree with the author. I would just frame it differently.
The question you need to ask is "do I need statistics?" Statistics have become habitual, but much (most?) of the time, we don't need statistics.
If you want to learn if a user can write and publish a blog using your software or install a water filter under they sink... you don't need statistics. You need to know where most people get into trouble, and n=5 will work fine for that.
This is intuitive if we just think outside of the "testing" vocabular. You write a CV/essay/article. You ask 1-3 friends to read it and advise. You don't produce statistics.
The article talks specifically about usability testing - though it isn't in the title but it is in the first line of the article. I don't think pricing strategies or matching algorithms and such would fall under this domain.
If you are interested in (for example) determining the optimal price and only 1/20 users buy something... you still probably need about 1,000 X (price points you want to test).
In that "test" you are basically trying to uncover the demand curve (or points on it). It's a statistical question.
Say you have a dating app, and you are trying a new matching algorithim. It will also take thousands of matches before you have the data to make determinations.
All that said, I totally agree with the author. I would just frame it differently.
The question you need to ask is "do I need statistics?" Statistics have become habitual, but much (most?) of the time, we don't need statistics.
If you want to learn if a user can write and publish a blog using your software or install a water filter under they sink... you don't need statistics. You need to know where most people get into trouble, and n=5 will work fine for that.
This is intuitive if we just think outside of the "testing" vocabular. You write a CV/essay/article. You ask 1-3 friends to read it and advise. You don't produce statistics.