In the previous lecture you learned about confidence intervals here you will learn about tests of significance. Recall that in STATISTICAL INFERENCE, the parameters are usually not known, and we draw conclusions from outcomes (i.e. sampling outcomes) to make guesses about the underlying parameters.
Remember the basic idea: we make assumptions about the parameters, and then test to see if those assumptions could have led to the outcome we observed. We then use a probability calculation to express the strength of our conclusions.
Let's walk through the example at the beginning of Chapter 26 together.
A senator introduces a bill to simplify the tax code. His claim is the bill is revenue-neutral. Basically it won't change the amount of taxes the government collects, it just simplifies the law.
A law can be evaluated. The IRS could SAMPLE from the POPULATION of all tax returns, figure out the effect the bill would have on these revenues, and then check to see if the bill is really revenue-neutral.
In the example, the IRS samples 100 forms. The sample average comes out to -$219 which means that the government would have collected 219 fewer dollars from taxpayers. The sample standard deviation is $725.
The senator's argument (issued through an aide) is that the SD is so large, $725, that an average of -219 is inconsequential.
The IRS's argument is what you want to learn. To understand the -219 and the 725, you need to convert the sample SD to an SE for the sample average.
Remember what the SE is, it is variation association with sample statistics. The SD is the variation in a given sample (e.g. the 725 here) or the variation in a population, or in a list (see Chapter 4).
The IRS goes on to say, the senator may think/argue the that the population parameter is $0, they say it's not and it is also negative.
How do they figure that?
First, they calculate an SE for the sample average
Square Root (100) x 725
------------------------ = about $72
100
Second, they set up a "test" and use Z as the "test statistic"
-219 - 0
------------------------ = about -3
72
This test statistic says, in a way, that if the true parameter was zero dollars and the samples have a variation of $72 then the chance that you could have picked a sample of size 100 with a mean of -219 is about 0.1 of 1% which is the area to the left of -3 under the normal curve.
In previous chapters, you learned to work with the formula for a Z score and the normal curve. In Chapter 26, it all comes together.
Usually, the ALTERNATIVE is what we're setting out to prove. The NULL is like a "straw man"
where z = (observed value - expected value)/spread
All a Z does is it tells you how many SEs away the observed value is from the expected value when the expected value is calculated by using the NULL HYPOTHESIS.
p-values are always "if-then" statements:
"If the null hypothesis were true, then there would be a p% chance to get these kind of results."
1. Clearly identify the parameter and the outcome.
2. State the null hypothesis. This is what is being tested. A test of significance assesses the strength of evidence (outcomes) against the null hypothesis. Usually the null hypothesis is a statement of "no-effect" or "no difference"
3. The alternative hypothesis is the claim about the population that we are trying to find evidence in favor of. In the tax law example, you are seeking evidence that the law is not neutral. The null hypothesis would say that the average return will not change (i.e. 0) , the alternative would say it is negative . Note this is a ONE-SIDED alternative because you are only interested in deviations in one direction. (A two-sided situation occurs when you do not know the direction, you just think the evidence suggests somthing different from the null)
4. The test statistic. It is the statistic that estimates the parameter of interest. In the above example, the parameter is the population average and the outcome is the sample average and the test-statistic is Z.
The significance test assess evidence by examining how far the test statistic fall from the proposed null.
To answer that question, you find the probability of getting an outcome as extreme or MORE than you actually observed. So to test the outcome, you would ask "what is the chance of getting a -$219 or lower number?"
5. The probability that you observe is called a P-VALUE. The smaller the p-value the stronger is the evidence against the null hypothesis. If instead you had gotten a sample average of -$100 with the same SE, the senator may well be right. (A Z of about -1.4 has about 8% of the normal, so here, there was an 8% chance of getting a sample with an average of -$100 or lower).
6. On significance levels. Sometimes prior to calculating a score and finding it's P-value, we state in advance what we believe to be a decisive value of P. This is the significance level. 5% and 1% significance levels are most commonly used. If your P-value is as small or smaller than the significance level you have chosen then you would say that "the data is statistically significant at level --- "
NOTE:
Significant is not the same as important. All it means is that the outcome you observed probably did not happen by chance.
The following letter appeared in the "Dear Abby" column in the 1970s:
Dear Abby
You wrote in your column that a woman is pregnant for 266 days. Who said so? I carried my baby for 10 months and 5 days (Prof's note: that's 310 days) and there is no doubt about it because I know the exact date my baby was conceived. My husband is in the Navy and it couldn't have been conceived at any other time because I saw him only once for an hour, and I didn't see him again until the day before the baby was born.
I don't drink or run around, and there is no way this baby isn't his, so please print a retraction about the 266 day carrying time because otherwise I am in a lot of trouble.
San Diego Reader
OK....suppose it is known that pregnancy durations are normally distributed with a mean of 266.0 days and a standard deviation of 16.0 days.
A chapter 23-like question might be: what is the chance of observing a pregnancy duration of 310 days or more?
310 - 266 ----------------- = 2.75 = Z = .3% chance (SQRT(1)x 16) / 1
Remember, this is like a sample of one and you know the population parameters. Let's not pass judgment on the San Diego lady...a pregnancy as long as 310 days can happen, and the chance is about 3 in 1000 pregnancies.
In chapter 26, when we are calculating chances, we're thinking more along the lines of larger samples and of trying to make a decision -- choosing between hypotheses.
A chapter 26-like question: In a recent study of 100 pregnancy durations selected at random, the average pregnancy duration was 270 days with standard deviation of 20 days. Does this study provide evidence that pregnancy durations have increased since the 1970s? Perform a test of significance and state the p-value.
The null hypothesis is 266 days.
The alternative something longer than 266 days
This is a one-side test, we're only interested if the durations are longer now
The test statistic is
( 270 - 266) ------------------- = 2.50 = Z = .62% (SQRT(100) x 16)/100
This would suggest that the probability of getting a sample average of 270 if the true average is 266 is about 6 times in 1000. This is evidence for the alternative, that is, that durations are getting longer.
Things to note: why a SD of 16?
I've been working through examples on averages, but counting situations work too. The z statistic is the same, but beware of the SE. If you are working with averages, make sure the SE is for the average, with counts, make sure the SE is for the number, with percentages, make sure the SE is for the percentage. Page 487 of your text makes for a good summary.