Observational vs. Controlled Experiments

In many (if not all) experimental situations, the researcher hopes to determine the effects of a treatment on a measurable outcome.  For example, in his famous experiments from the Leaning Tower of Pisa, Galileo wished to determine the effects of weight (the treatment) on the time to fall (the measured outcome). (He dropped objects of different weights from the tower and noted that they took the same time to reach the ground.)  I volunteered for a study that is studying the affects of a pertussis ("whooping cough") vaccine on the incidence of pertussis.  A collection of studies that try to determine the effects of second-hand cigarette smoke on lung cancer incidence has been in the news recently.  (A federal judge ruled that there was no convincing evidence of an effect.)

A good study is always a comparison.  When someone makes a claim such as "My vaccine prevents polio", implicitly she is saying "My vaccine prevents polio when compared to people who don't get the vaccine."  So a good study must compare the results of treatments.  In a vaccine study, this means it compares people who got the vaccine with people who did not.  In a second-hand smoke study, we might like to compare people who have been exposed to varying levels of second-hand smoke.  (How would you measure this?)

One of the hallmarks that separates types of experiments is to ask yourself "Who decided which treatment a subject would receive?"  If the subjects themselves chose, then its usually what we call an observational study.  Otherwise, it is probably a controlled study.  I use the word "chose" very loosely here, because often in an observational study the subjects aren't aware that they're making a choice (or even that they're participating in a study!).  For example, if I want to determine the effects of smoking on lung cancer, I could compare people who have been smoking for 10 years or more and people who have never smoked.  In this case, the treatment is "smoking", and the outcome is whether or not a subject has lung cancer.  This is an observational study because the subjects themselves chose whether or not to smoke (long before anyone ever knew they would participate in my study!)

On the other hand, when I volunteered to participate in a vaccine study, a nurse flipped a coin (or its equivalent) and based on the outcome decided whether I would get the pertussis vaccine or a placebo.  (I wasn't told which.  Why?)  I had no say in the matter (other than my decision to participate in the study.)  So this could be classified as a controlled experiment.  (To be more precise here, a controlled experiment controls for confounding factors.  There are still many ways in which a study that starts out controlled could fail to control for other things.  For example, had the nurse told me which treatment I got, and didn't tell other people, you could argue that the experiment was not well-controlled.  But it's certainly the case that this is not an observational study.)  An example of an observational study in the same context would be one in which the nurse asked me whether I wanted the vaccine or the placebo.  Why would this affect our ability to decide whether the vaccine worked?

Why does this matter?  In a nutshell, you cannot determine whether the treatment caused the observed outcome in an observational study.  Never.  But you can in a controlled study.  So if my vaccine study is well-controlled (which is easier said than done), than if the vaccine group gets pertussis less often on average than the placebo group, we can safely conclude that the vaccine caused this difference.  However, in the artificial smoking/lung cancer study I mentioned above, even if the smokers had a higher lung cancer rate, we couldn't conclude that it was the smoking that did this.  In fact, for years, until the evidence to the contrary was overwhelming, the tobacco industry claimed there was a gene that both caused people to smoke and caused them to get lung cancer.  This illustrates a central limitation of observational studies: you can never rule out all confounding factors.

Questions
Back to Outline