1. Overview
The basic question: was the sample outcome we observed due to chance or something else? The basic question restated: one side thinks any observed difference between what you expect and what you get is REAL (so perhaps something is wrong with the expected value or with the sample process). The other side thinks the difference is just random chance operating.
If your observed value is too many STANDARD ERRORS away from the expected value, this is hard to explain by chance alone. The number of standard errors away is called a Z score and the method you use to arrive at this score is a "test of significance"
A historical example and then the example from your textbook (26.1)
2. Vocabulary
The NULL HYPOTHESIS (26.2) is that the observed results are purely due to chance alone. That is, any differences between the parameter (the expected value) and the observed (or actual) outcome was due to chance only. In this case, the null hypothesis is a statement about a parameter: the population average is 0 for the IRS example in 26.1.
The ALTERNATIVE HYPOTHESIS (26.2) is that the observed results are due to more than just chance. It implies that the NULL is not correct and any observed difference is real, not luck.
Usually, the ALTERNATIVE is what we're setting out to prove. The NULL is like a "straw man" that we wish to knock down.
The TEST STATISTIC (26.3) measures how different the observed results are from what we would expect to get if the null hypothesis were true. When using the normal curve, the test statistic is z,
where z = (observed value - expected value)/standard error of that value
All a Z does is it tells you how many SEs away the observed value is from the expected value when the expected value is calculated by using the NULL HYPOTHESIS.
The SIGNIFICANCE LEVEL (or P-VALUE) (26.3). This is the chance of getting results as or more extreme than what we got, IF the null hypothesis were true. P-VALUE could also be called "probability value" and it is simply the area associated with the calculated Z.
p-values are always "if-then" statements:
"If the null hypothesis were true, then there would be a p% chance to get these kind of results."
If the p-value is less than 5%, we say the results are STATISTICALLY SIGNIFICANT (26.4); if p < 1%, the results are HIGHLY STATISTICALLY SIGNIFICANT. A "significant" result means that it would be unlikely to get such extreme observed values by chance alone.