Why you probably shouldn’t trust observational studies
1. Undersampling failure
Let’s say you’re doing research on why certain companies are more successful than others. Your data set only contains successful companies. You’ll have a lot of them taking huge risks, that combined with luck created runaway success. And you completely ignored the companies who took huge risks, but weren’t lucky enough, and ended up being bankrupt. Your sample naturally overweight the unreasonable risk takers. Compare that with a randomized controlled trial, where you start with a set of companies, and observe them throughout their lifetime, so the unreasonable risk takers with less luck will be accounted for. Then, your conclusions will be more accurate.
The more hypotheses you test on your data, the more likely it is that you’ll find a statistically significant effect. Remember, with a p-value of 0.05, on average, 5 out of every 100 randomly generated hypotheses will test positive.