MORE ABOUT HYPOTHESIS TESTING

A. Overview

Remember that there are three steps to significance testing:

1. State the Hypotheses

2. Calculate the test statistic

3. Find the P-value

B. Details

1. Calculate the test statistic

In section 5.2, you constructing tests based upon the sample mean x-bar. These normal probability calculations have a known sigma and require standardization -- so we use a Z-test.

2. Find the P-value
(Remember what a p-value is -- it is the probability -- assuming that the null hypothesis is true, that the test statistic you calculated would take a value as extreme or more extreme than what you observed. The smaller the p-value, the stronger the evidence against the null hypotheses.)

C. Hypothesis Testing Summarized

1. Clearly identify the parameter and the outcome.

2. State the null hypothesis. This is what is being tested. A test of significance assesses the strength of evidence (outcomes) against the null hypothesis. Usually the null hypothesis is a statement of "no-effect" or "no difference"

3. The alternative hypothesis is the claim about the population that we are trying to find evidence in favor of. Note there are one and two sided alternatives

4. The test statistic. It is the statistic that estimates the parameter of interest. In the above example, the parameter is mu and the statistic is x-bar. The test statistic is a Z score.

The significance test assesses evidence by examining how far the test statistic fall from the proposed null.

To answer that question, you find the probability of getting an outcome as extreme or MORE than you actually observed.

5. The probability that you observe is called a P-VALUE. The smaller the p-value the stronger is the evidence against the null hypothesis.

D. Significance Levels

Sometimes prior to calculating a score and finding it's P-value, we state in advance what we belive to be a decisive value of P. This FIXED level of significance is symbolized by alpha and all it says is how much evidence we require to decide that a test is significant. We usually say that if a p-value is less than or equal to alpha, it is significant.

The book shows how you can quickly determine significance at fixed levels by looking at "critical" values of Z. These are the same values used for the confidence intervals.

NOTE:

Yes, significance and confidence intervals are related. See pp 374-375. The hypothesized value falls outside of the confidence interval.

E. Working with Significance

1. Choosing a level of signficance
   

   You choose a level alpha in advance if you wish to make a decision. In other words, if you think an alpha of .05 would be sufficient, then state it and perform the test.

   If you wish to only describe the strength of your evidence, you don't need to state a fixed alpha. You just state the p-value you got from the test statistic.

   Note, even though you see alpha=.05 frequently. It isn't necessary holy or sacred.

   Perhaps it is better to consider that if the p-value is less than 5%, we say the results are STATISTICALLY SIGNIFICANT; if p < 1%, the results are HIGHLY STATISTICALLY SIGNIFICANT. A "significant" result means that it would be unlikely to get such extreme observed values by chance alone.

   As p-values decrease, there is increasingly strong evidence that the value observed is extreme and not possible by chance alone.

2. Very large samples and significance

   A point: if samples are very large, everything becomes significant...which may not be a good thing. See problem 5.56 in your book.

3. Appropriate use of tests of sigificance

   A point: The use of tests of significance, confidence intervals rely on the laws of probability -- which means that the ability to randomize in sampling or in experimentation must exist. If it doesn't, yes, you can still perform the test...but it is not clear what the results mean.

4. Beware of multiple analyses

   A point: If you beat the data hard enough, it will tell you anything you want it to. By doing this, we are comparing the p-value we calculate for the test statistic to some fixed value (e.g. 5%, 1%) which we consider decisive. This decisive value is usually denoted by the greek letter alpha. If you choose (or are told to choose) an alpha = .05 we are requiring that the data give evidence against the null hypothesis which is so strong that it would happen no more than 5% of the time if the null were true. If we choose alpha = .01 then we are saying this outcome would be observed less than 1% of the time if the null were true.

F. Yet another example

The technique you have been learning is appropriate when the population is known to be normally distributed and the population standard deviation is known OR the sample size is large.

The appropriate test statistic is Z and the p-values are found by using a table of standard normal values.

Example: Between 1991-1995 entering students at a major university had an average SAT score of 612 points with a standard deviation of 80 pts and is normally distributed. A simple random sample (SRS) of 100 students from 1996 entering class are selected and their average verbal SAT score is found to be 594 points.

Has there been a decline in the verbal score? Carry out an appropriate test of significance.

Null Hypothesis: NO decline
Alternative Hypothesis: decline

Z is the appropriate statistic. Z = -18/8 = -2.25
this yields a p-value of .0122

You would reject the null hypothesis at the 5% level but not at the 1% (the p-value is greater than 1% but less than 5%)