THE DESIGN OF EXPERIMENTS

A. Overview

Statistics is the science of collecting, presenting, and interpreting data to answer questions. In other words, statistics is a "language" used to translate data into something we can understand.

There are four primary issues:

1. Determining the question and the data that will help you answer the question.

2. Collecting the data.

3. Summarizing and presenting the data graphically and numerically.

4. Making generalizations from the data and drawing conclusions. Frequently by making comparisons between groups.

B. Determining the Question

1. The first issue in statistics is always,

   "What question do we want to answer?"

2. Data do not speak for themselves;

   we must collect data with a purpose. You might think of data as letters of an alphabet. Statistics will help us understand the message in the data.

3. Not all questions can be answered!

   a. "Data as they are" questions can be answered.

   b. "What-if questions under replicable circumstances" can be answered

   c. Data from nonreplicable events, in general, can NOT be answered!

C. Collecting Data

1. CONTROLLED EXPERIMENTS: the researcher ASSIGNS a treatment and observes the outcomes.

   Sildenafil handout and a class exercise

   Principles of Experimentation

   • CONTROL FOR (or eliminate) potential confounding factors.
   • ASSIGN some patients to get the new drug and others not.

     a. The best method is RANDOM assignment, to eliminate bias.

     b. Patients getting the new drug form the TREATMENT GROUP; patients not getting the new drug form the CONTROL GROUP.

   • Make the experiment BLIND.

     a. Give the control group a PLACEBO (a do-nothing drug).

     b. Do not let the patients know whether they are getting the new drug or the placebo.

     c. This eliminates the "PLACEBO EFFECT", when patients get better just because they think they are getting a drug.

   • Make the experiment DOUBLE BLIND.

     a. Do not let the doctors examining the patients know which drug the patients are getting.

     b. This eliminates any bias in the doctors' judgements.

     c. Sometimes double blinding is impossible.

   • Once the experiment is concluded, COMPARE the two groups.
   • If possible, REPLICATE the experiment.
   • Don't be paranoid.

     Even if a study is not a randomized double-blind controlled experiment, it may not be worthless; it depends on how the study was done and what sources of bias may exist. The upshot of doing all of this is to eliminate all possible forms of contamination.

2. OBSERVATIONAL STUDIES: the researcher collects the data as they currently are.

D. Other Examples

• Pfizer's Sildenafil experiments 1995-1997(also see the handouts page)
• Welfare-to-Work in LA: Experimental Design
• Testing for Employment Discrimination
• Testing for Housing Discrimination in California

E. Vocabulary and Comments

Controlled Experiment

Treatment Group

Controls

Randomized/At Random/Random Assignment

Double Blind

Placebo

Confounding

Experiments need not be limited to medical trials (such as the examples in Chapter 1 of the text). School districts, police departments, transportation companies (e.g. airlines, taxis), just about anyone can conduct experiments.

Experiments are the best way of making comparisons about which we can feel confident, but sometimes they are not feasible or ethical (e.g. imagine setting up a randomized experiment involving the effects of living near nuclear power plants). In many disciplines, observational studies are more commonly found for these reasons.