Let the experiment decide II | Day 105

Well-performed scientific experiments are rigorous to the extreme – because they have to be. There are several reasons for why we can’t just assume that things are the way they seem to be. The main one is both really simple and infinitely complex at the same time: it’s because we don’t always see the world in a way that corresponds to the true nature of things.

Humans operate with a wide range of mental shortcuts and rules of thumb. This is the realm where you get cognitive biases that may be subtly influencing the way you view and approach things. Amongst those might be things like wishful thinking, picking out only the data that conforms to your worldview, and similar pitfalls that end up obscuring the facts.

For example, if you generally really like drinking tea, and are in favour of herbal medicine as a principle, a weight loss tea becomes an inherently appealing idea to you. It already aligns with what you believe about the world. And because you have a positive attitude from the get-go, you may start picking up little cues that confirm your wish – or belief – that the tea simply has to work.

In science one needs to shed these biases and attitudes in order to chisel away at the truth of the matter, and reach a conclusion that is as reliable as can be. The tea either works or it doesn’t. Because humans really aren’t designed to intuit in a rigorous manner, that’s where formalised procedures – such as experiments – come in.

Here’s what a perfect experiment would look like for testing that weight loss tea I keep mentioning. Take 20 women with parameters as similar as possible – similar age, weight, metabolic rate, etc. Randomly divide them into two groups – ten get to drink the real tea for four weeks, while the other ten drink a fake tea that has absolutely no effect on weight loss.

Don’t tell them which tea they got. In fact, assign them in such a way that you don’t know this yourself. That way you won’t be subtly influencing the results by assigning participants in a way that might make a difference for the test, or even leading them onto the nature of the tea they’re getting.

In such a setup – known as “double-blind, placebo-controlled trial” – there is hardly any chance that a variable will sneak in and skew the results. You have all the criteria down pat, and only one variable to test – the use of the tea. If women in the fake tea group lose as much weight as the participants in the real tea group, well, you have your answer. That’s why this experimental design is the “gold standard” in such areas as medical drug testing.

Of course, this is not the only type of experiment, although it is the most rigorous kind. Science also relies on quasi-experiments, correlations, on data models, and on observations over time amongst other things. It’s not always possible to isolate a variable for controlled testing, but that’s ok – there are tools for this kind of stuff, too.

There’s a reason why science is considered as a reliable source of facts – it’s a process developed to be as reliable as possible, while recognising the deeply human inability to avoid bias.

And that’s why skeptics like science.


This is skeptical week. To find out why it’s skeptical week, please read Monday’s post.

Leave a Reply

Your email is perfectly safe with me.