386 1 UNITED STATES DISTRICT COURT EASTERN DISTRICT OF NEW YORK 2 --------------------------------x THE CITY OF NEW YORK, et al., 3 Plaintiffs, 89 Civ. 3474 (JMcL) 4 v. 5 UNITED STATES DEPARTMENT OF COMMERCE, 6 et al., 7 Defendants. ---------------------------------x 8 CITY OF ATLANTA, et al., 9 Plaintiffs, 10 v. May 13, 1992 9:30 a.m. 11 MOSBACHER, et al., 12 Defendants. ---------------------------------x 13 FLORIDA HOUSE OF REPRESENTATIVES, et al. 14 Plaintiffs, 15 v. __Civ.___(JMcL) 16 BARBARA H. FRANKLIN, Secretary of Commerce, 17 Defendant. 18 --------------------------------x Before: 19 HON. JOSEPH M. McLAUGHLIN, 20 Circuit Judge 21 22 23 24 25 387 1 (Trial continued) 2 (In open court) 3 THE COURT: Good morning. Mr. Sitcov, well- 4 oiled? 5 MR. SITCOV: Well-greased, your Honor. 6 EUGENE P. ERICKSEN, 7 having been previously sworn, resumed the 8 stand and continued to testify as follows: 9 THE WITNESS: Good morning. 10 CROSS-EXAMINATION CONTINUED 11 BY MR. SITCOV: 12 Q. Dr. Ericksen, when we were finished yesterday, I 13 believe we were discussing Appendix G of Plaintiff's Exhibit 14 195. That is the loss function analyses that were done by 15 Kirk Wolter and Beverly Causey. 16 It is correct that you believe that that document 17 shows an adjusted count would be superior to the census in 18 terms of the proportional distribution of the population on 19 a state-by-state and substate-by-substate basis, isn't that 20 right? 21 A. You are talking about -- sorry. Are you 22 referring to the year 1990? 23 Q. Pardon me? 24 A. Are you referring to the year 1990? 25 Q. Yes. 388 1 A. What the analysis of Wolter and Causey does, 2 coupled with the analysis that went into our joint report, 3 that led to the conclusion that were a loss function 4 analysis calculated on the basis of 1990 data, which states 5 and similar areas to those that Wolter and Causey dealt 6 with, that such a loss function analysis would show that the 7 original enumeration would have greater error than the 8 results of the original enumeration coupled with the results 9 of the PES. 10 Since Wolter and Causey dealt with data that were 11 collected from 1980, it does not provide correct evidence 12 about 1990. It simply provides a guide as to what we might 13 have expected from 1990. 14 And so, essentially, what we were doing in our 15 report on pages 20 and 21 was prophesizing what the results 16 of the loss function analysis would show. 17 Q. I see. So the Walter and Causey paper didn't 18 even use 1990 data, did it? 19 A. No. The Wolter and Causey paper was written 20 during the decade of the 1980s, and it was written to 21 provide guidance as to what we might expect from the 1990 22 Census. 23 Q. So this was data from the 1980 census? 24 A. I believe that their data were from 1980. I 25 don't have that perfectly in mind, but I believe they did 389 1 use 1980 census data, yes. 2 Q. And neither you nor any of the other plaintiffs' 3 nominees to the Special Advisory Panel conducted a loss 4 function analysis using 1990 data before you submitted your 5 joint report to the Secretary, isn't that right? 6 A. That is correct. 7 Q. Now, another of the loss function analyses in the 8 Wolter Causey paper showed that the population shares of 9 each of the 50 states would be improved by adjustment, isn't 10 that right? 11 A. Again, I assume that you are talking about in 12 1990. 13 Q. That's right. 14 A. The Wolter and Causey paper was not based on the 15 1990 data, so the answer to your question is yes. 16 Q. Well, the loss function analysis that you 17 referred to on page 20 of the joint report that you 18 coauthored doesn't show the population shares of each of the 19 50 states would be improved by adjustment, does it? 20 A. I could just answer it doesn't demonstrate it. 21 Our analysis reported in this document, coupled 22 with the results of Professor Tukey and Appendix H and the 23 paper of Wolter and Causey Appendix G, predicted very likely 24 that a loss function analysis conducted for the 50 states 25 based on 1990 data would show that the use of the PES data 390 1 improved their estimate of the distribution population, and 2 it did. 3 Q. That's not my question. 4 My question is: The loss function analysis that 5 you referred to on page 20 of your joint report doesn't show 6 that the population shares of each of the fifty states would 7 be improved by adjustment, does it? 8 A. Our -- we were not discussing an empirical study 9 on page 20, so the answer to your question is yes. 10 Q. In fact, it doesn't tell you how many states' 11 shares would be made more accurate by adjustment, does it? 12 A. Without data, how could we do such a thing? 13 Q. So the answer to my question is yes? 14 A. The answer to your question is yes. 15 Q. And it doesn't tell you which state's shares 16 would be made more accurate by adjustment, does it? 17 A. No, it does not. 18 Q. It is the case, isn't it, that loss function 19 analysis doesn't enable you to identify any particular state 20 that's made more or less accurate by adjustment, isn't that 21 right? 22 A. With certainty? 23 Q. I'm just asking you, is that right? 24 A. It does help you identify the states which are 25 more likely to be made more accurate by adjustment, but loss 391 1 function analysis is an exercise in probabilities, and these 2 probabilities would never be a hundred percent, so the 3 answer to your question is not with certainty. 4 Q. Well, do you recall testifying at your deposition 5 in this case on page 303 at line 19: 6 "We don't, in fact, know any particular state 7 that was made more or less accurate, and loss functions are 8 generally intended to apply to the aggregate states, not the 9 individual state." 10 A. I would like to check that passage, and then I'll 11 answer your question. 12 Q. Please feel free. 13 MR. ZIMROTH: Would you tell us what page that 14 is? 15 MR. SITCOV: 303, starting on line 19. 16 A. Now, if your question is do I recall it, I 17 certainly recall having read it in my deposition, that I 18 think what I said in my deposition is about what I just said 19 to you just now. 20 Q. So it is correct then that with the loss function 21 you don't know any particular state that was made more or 22 less accurate by adjustment? 23 A. Not with 100 percent certainty. 24 Q. Okay. 25 THE COURT: Is that no with a hundred percent 392 1 certainty? 2 THE WITNESS: Right. 3 THE COURT: Or is that it wasn't made more 4 accurate with a hundred percent certainty? 5 THE WITNESS: Well, the point is that with the 6 results of loss function analysis, you might say, for a 7 certain state, it looks very, very likely that it is made 8 more accurate. But this is a sample estimate. It's never 9 going to be a hundred percent. 10 BY MR. SITCOV: 11 Q. Now, the Undercount Steering Committee originally 12 determined that loss function analysis showed that the 13 population shares of 11 states would be made worse by 14 adjustments, is that right? 15 MR. ZIMROTH: You want to refer him to a 16 particular page? 17 MR. SITCOV: Well, I'm asking him if he knows. 18 He hasn't said he doesn't know. 19 A. I don't recall particular numbers for show. 20 Sure. I do know that there was a statement there that some 21 would be more and some would be made less accurate with the 22 adjustment. But don't ask me to recall particular numbers. 23 Q. Well, do you have Plaintiff's Exhibit 55 in front 24 of you? 25 If you turn to page -- I'll leave off all the 393 1 zeros -- 1146, you'll see that there is a recommendation 2 from the Undercount Steering Committee. 3 A. Page 1146 has a title "Technical Assessment." 4 Q. Right. 5 A. Rather than a recommendation. 6 Q. Okay. "Technical Assessment." Now, could you 7 turn to page 6 of that document. 8 Now, in the third full paragraph down the one 9 that begins "loss functions," doesn't that paragraph say 10 that 39 states would have their population shares made 11 closer to the truth than 11 states would have their 12 population shares made worse after adjustment? 13 A. That is what it says. 14 Q. And then the Undercount Steering Committee took a 15 second loss function analysis in connection with the 16 adjustment decision, isn't that right? 17 A. I believe that's correct. 18 Q. And this time the Undercount Steering Committee 19 concluded that the population shares of 21 states would be 20 made worse by adjustment, isn't that right? 21 A. Like I said before, I don't recall the exact 22 numbers. I do know that I did at least that one analysis. 23 But was that number correct, I don't recall. 24 Q. Does 21 sound about right? 25 A. It could be. 394 1 Q. And in this scenario as well, that is, the second 2 loss function that the Undercount Steering Committee did, 3 you couldn't identify which state's proportional shares of 4 the population were made better by adjustment and which ones 5 were made worse, isn't that right? 6 A. Well, again, the difference is that I would say 7 that these things should be estimated within terms of 8 probability being made better or worse. And when we are 9 talking about probabilities, it isn't then saying that you 10 know which ones they are. 11 Q. But you don't know which ones they are? 12 A. In the way which I defined no knowing, the answer 13 is no. 14 Q. The true population of the United States is 15 unknown, isn't it? 16 A. We do not know exactly what the true population 17 of the United States is, that's true, right. 18 Q. And it's not possible to determine the true 19 population of any state, is it? 20 A. We can only estimate it, and the only estimate 21 that we could come up with would obviously have uncertainty. 22 Q. So is your answer to my question yes, it is not 23 possible to determine the true population of any state? 24 A. With complete certainty? 25 Q. Yes. 395 1 A. Whether answering yes or no, I will simply say it 2 is not possible to know with 100 percent certainty what the 3 true population is of any state. 4 Q. But you feel almost a hundred percent sure you 5 know the true population of any state? 6 A. I'm not in a position to quantify it. We don't 7 know for a certainty the population of any state. 8 Q. So in order for the Census Bureau to undertake a 9 loss function, it had to create a true population, didn't 10 it? 11 A. It had to create a model for a true population. 12 Q. And this so-called true population was created by 13 using data from the PES and the evaluations of the PES; that 14 is the total error amount, isn't that right? 15 A. That is correct. 16 Q. Now, if the bias in the production dual system 17 estimate were understated in the total error model, that 18 could affect the results of the loss function analysis, 19 couldn't it? 20 A. It could. 21 Q. Dr. Ericksen, I want you to consider loss 22 function analysis for the 50 states and the District of 23 Columbia. 24 Now, in order to interpret the results of a loss 25 function analysis for the 50 states and the District, you 396 1 need to know the pattern of gains and losses and not just 2 the total across the 50 states and the District, isn't is 3 that right? 4 A. There are various different interpretations you 5 would make of the results of the loss function analysis. 6 For some interpretations, the answer would be yes; for other 7 interpretations, the answer would be no. 8 Q. To determine whether or not to adjust the 1990 9 Decennial Census, you would want to know the pattern across 10 the 50 states, wouldn't you? 11 A. I'm not sure exactly what you mean by "the 12 pattern." 13 What would be mainly concerned with the results 14 of the loss function would be to know an aggregate whether 15 the errors from one set of estimates are larger than the 16 errors from another set of estimates. 17 Q. Do you recall that when I took your deposition, I 18 showed you the Bureau's loss function analysis report, and 19 we considered changing some of the numbers in the loss 20 function analysis report, and I asked you what if it came 21 out that California's number in this loss function report 22 was something other than it was, and that the total of the 23 50 states -- the 50 states and the District split the 24 remaining difference. 25 And I asked you if that would change your opinion 397 1 about loss function. 2 And you said, on page 332 of your deposition: 3 "It would depend on the pattern in the other 4 states." 5 A. Yes, I think that what I was trying to explain to 6 you was that if you were to change the results of the loss 7 function, what I would really want to do is to assess the 8 entire pattern of change in the loss function to understand 9 why the numbers were changed, what the assumptions were 10 changed to make this difference occur, what it was about the 11 data, that because of the pattern in the data that were used 12 in the loss function, to cause this result. 13 And it is a sufficiently complicated analysis 14 that when you asked me about it in my deposition, you said 15 what particular pattern would make a fair adjustment, that 16 it was too complex for me to answer just sitting there in 17 the deposition. 18 Q. Well, you are an expert in loss functions, aren't 19 you? 20 A. Within the field of statistics I do not consider 21 myself to be a particular expert in loss functions, no. 22 Q. Well, you wrote a report with Ericksen -- with 23 Estrada, Tukey and Wolter in which you recommended favored 24 adjustment in the use of loss functions, didn't you? 25 A. It was a joint report with contributions by many 398 1 people. And together this supplemented our recommendation. 2 Q. So you were not able to tell me, when I took your 3 deposition, what pattern would make a fair adjustment? 4 A. That right. 5 Q. And you couldn't tell me what pattern would make 6 a disfavored adjustment, is that right? 7 A. Not sitting there that moment, no. 8 Q. Now, in order to decide whether the results from 9 loss financial analysis favored adjustment you need to make 10 a rigorous assessment of how the calculations are done, is 11 that right? 12 A. Yes. 13 Q. And when you submitted your report with Estrada, 14 Tukey and Wolter to the Secretary on June 21st, 1990, you 15 didn't have the opportunity to review the results pertaining 16 to loss functions using data from the 1990 Census and PES, 17 did you? 18 A. We had not seen the Census Bureau's results. 19 Therefore, all we could do was write the prediction that we 20 did based on -- on pages 20 and 21. 21 Q. And prior to July 15th, 1991, you had not made a 22 rigorous assessment of the Bureau's loss function 23 calculations. 24 A. That is correct. 25 Q. Would you take a look at Defendant's Exhibit 107. 399 1 MR. SITCOV: That has already been entered into 2 evidence, your Honor, as Plaintiff's Exhibit 540 A, 16 A. 3 It is in the so-called administrative record at bates stamp 4 number 0006074. 5 Q. Do you have it, Dr. Ericksen? 6 A. Yes, I do. 7 Q. You are familiar with this document, aren't you? 8 A. I'm knowledgeable with the results of this 9 document, yes. 10 Q. And this is the report of loss function analyses 11 the Census Bureau conducted in connection with the decision 12 whether it would adjust the 1990 Decennial Census, isn't it? 13 A. That's what it is. 14 Q. Dr. Ericksen, I'd like you to take a look at 15 Table 3, which is in the back of this document, and it's at 16 bates stamp number -- without the zeros in front -- 6,087. 17 And it is entitled, "Table of Loss Function Comparisons 18 (UNWEPRS)" -- 19 A. I don't have any bates numbers on my copy. 20 THE COURT: I have the table, if you want it. 21 A. Yes, the tables aren't clear. Why don't you come 22 and show me which one you have in mind. It doesn't have any 23 table marks too. 24 (Pause) 25 BY MR. SITCOV: 400 1 Q. Do you recall testifying about this table at your 2 deposition? 3 A. I recall answering your questions about this 4 table at the deposition, yes. 5 Q. And you remember that I asked you if in your 6 mind, in your opinion as an expert in this case, this table 7 favored adjustment? 8 A. I do recall that. 9 Q. Do you remember that you said that you couldn't 10 tell me without further consideration? 11 A. Don't recall that this moment, but if it's in 12 there, I'm sure that's what I said. 13 MR. ZIMROTH: May we have the page reference, Mr. 14 Sitcov? 15 MR. SITCOV: 329. 16 BY MR. SITCOV: 17 Q. Do you recall that I -- I'm thinking now of my 18 intent on page 329. 19 MR. ZIMROTH: 329? 20 MR. SITCOV: 329. 21 THE WITNESS: Let me find the place. 22 MR. SITCOV: Okay. 23 A. What I actually said in the deposition was that I 24 couldn't recall whether this was a chart that's the actual 25 chart -- 401 1 Q. I don't think there's a question pending. 2 My question is: Do you recall me asking this 3 question and you giving this answer, Line 10: 4 "Q. Okay. Do you have the results of this 5 chart in your mind? In your opinion as an expert, does this 6 case favor adjustment? 7 "A. The question would require further 8 consideration unless you can help me remember the 9 conclusions that the Census Bureau derived from this chart 10 so I can remember if and when I asked you to look at this 11 chart in the past. 12 "Q. Does that refresh your recollection that 13 you said that it would require further consideration before 14 you could tell me whether the results of this chart favored 15 adjustment?" 16 MR. ZIMROTH: Your Honor, that question and 17 answer is in the context that I think the witness ought to 18 be able -- 19 THE COURT: I heard you so far. 20 MR. ZIMROTH: -- which I think the witness ought 21 to be able to answer as he began to answer earlier before 22 Mr. Sitcov cut him off. 23 THE COURT: Look, answer as you are comfortable. 24 THE WITNESS: Look, the point is that tables on 25 loss function analysis arrived in my office some time after 402 1 I submitted my report. 2 There was a series of tables, as I recall, six or 3 eight tables, and when you handed me this particular table, 4 I wasn't sure that this was the one that I actually based my 5 conclusions on. And why this table was also in there with 6 all the others, I could not remember at that time. 7 I certainly was aware of the fact that the 8 Census Bureau had decided on the basis of their loss 9 function analysis as a whole that the adjusted numbers were 10 more accurate than the unadjusted numbers. 11 Q. Well, do you recall that after I asked you that 12 question, you conferred with your attorneys, and then you 13 were able to tell me whether or not you thought that chart 14 favored adjustment? 15 A. I don't have any particular recollection of that. 16 Q. Well, would you turn to page 330. Do you see at 17 line 6 it says, "Witness confers with attorneys," that I 18 made a statement about that? 19 Then you asked to have the question reread. And 20 then you said: 21 "Looking just at the numbers in this chart, I 22 have not attempted to relate the numbers in this chart to 23 any commentary in any document. The sum of losses due to, 24 attached to the original enumeration by the squared error 25 loss function appear to be larger than the sum of losses due 403 1 to this series of adjusted estimates, so that would favor. 2 "Similarly, the sum of losses by the absolute 3 difference criterion for the original enumeration appear to 4 be greater than the sum of losses due to the adjusted series 5 of estimates. So in both instances, just looking at these 6 numbers, they appear to support a decision to adjust." 7 Does that refresh your recollection that after 8 you conferred with your attorneys you were able to tell me 9 that this chart favored adjustment? 10 A. This is the question and answer that occurred. 11 Q. No, I'm asking, does it refresh your 12 recollection? 13 A. Yes. 14 Q. Now, looking in column 4 in that table, do you 15 see it? 16 A. I do. 17 Q. The total in column 4 is an estimated risk, isn't 18 it? 19 A. That's one term you could use for it. 20 Q. And if we had to double the entries in the 21 variance/co-variance matrix for the smooth adjustment 22 factors to make it a more realistic estimate of the sampling 23 error, you wouldn't have been able to tell me the impact on 24 the total in column 4, isn't that right? 25 A. I have not done such analysis, so I would not be 404 1 able to tell you the impact without any numbers in Volume 2 4. 3 Q. That's because you don't know the formula for 4 estimated risk, isn't it? 5 A. As I sit here today, I could not give you the 6 formula for how to double the variance/co-variance. Having 7 looked at this column, that's true. 8 Q. In fact, you don't know whether you ever knew the 9 formula, isn't that right? 10 A. I don't recall whether I had known that or not, 11 no. 12 Q. Now, the total in column 5 is an estimated risk, 13 isn't it? 14 A. You could refer to it as that. 15 Q. And the total in column 6 is an estimated risk 16 difference, isn't it? 17 A. You could refer to it as that. 18 Q. And this estimated risk difference is estimating 19 a real risk difference, isn't it? 20 A. You could refer to it as that, yes. 21 Q. And the total in column 6 is affected by sampling 22 error, isn't it? 23 A. It's an estimate based on a sample, so it would 24 be affected by sampling error, yes. 25 (Continued on next page) 405 1 Q. In fact, when I took your deposition, you didn't 2 know how you would calculate a standard error for the total 3 in column 6 and you couldn't tell me how to do it, isn't 4 that right? 5 A. That's right. This whole report was given to me 6 sometime in July, after July 15. I was not in a position to 7 confer with the people at the Census Bureau who had done 8 this analysis. 9 The calculation of the sampling error for a 10 complex statistic like this would be highly complex. So I 11 don't know how they did it. It didn't have the information 12 in this report that would tell me how to do it. Without 13 being able to find out what the Census Bureau had done, I 14 certainly wasn't in a position to make the calculation 15 myself. 16 Q. In fact, you the didn't even know whether or not 17 someone else could have computed that standard error, did 18 you? 19 A. Without the information that the Census Bureau 20 had, I wasn't in any position to make such an assessment. 21 You recall that I was not able to find out what the Census 22 Bureau had done after July 15. 23 Q. Now, the total in column 6 might differ from zero 24 just due to sampling error, isn't that right? 25 A. Any estimate based on sample has that possibility 406 1 however low a probability it may be. 2 Q. But you hadn't specifically considered the 3 question at the time I took your deposition, isn't that 4 right? 5 A. No, I had not. 6 Q. And when I took your deposition, you testified 7 you were not in a position to give a practically informed 8 answer to the question whether the difference between the 9 total in column 6 and zero could be due to sampling error. 10 Do you remember? 11 A. A practically informed answer based on data? No, 12 because I had not made such a calculation nor seen the 13 results of such a calculation. 14 Q. Different choices in smoothing models could 15 result in different undercount estimates for any particular 16 place, isn't that right? 17 A. That is correct. 18 Q. So the choice of a smoothing model has an impact 19 on the estimated uncounted rate, isn't that right? 20 A. Sure. Any time you could have a difference, the 21 difference may be very small, but certainly will have some 22 difference. 23 Q. And uncertainty about which smoothing model to 24 use isn't reflected in the standard error, isn't that right? 25 A. Uncertainty about which smoothing model to use is 407 1 not reflected in the standard error. 2 Q. And it is also the case, isn't it, that the 3 calculation of the final standard error for the smooth 4 adjustment factors depends on the assumptions underlying the 5 smoothing model? 6 A. Along with many other things it depends on those 7 as summations, yes. 8 Q. And if the assumptions are wrong, the standard 9 errors could be too small, couldn't they? 10 A. Well, assumptions are never perfectly correct at 11 any time. The issue is how far the reality deviates from 12 those assumptions. It would perhaps cause the standard 13 errors to be different, higher or lower, I cannot give you 14 an opinion at this time. 15 Q. So you are saying it's not possible that the 16 standard error could be too small if the assumptions are 17 wrong? 18 A. I didn't think that's what you asked me. 19 Q. I asked you if the assumptions are wrong could 20 the standard errors be too small? 21 A. Oh, could? 22 Q. Yes. 23 A. They could be. 24 Q. And that would make the estimated adjustments 25 look more reliability than they actually are, wouldn't it? 408 1 A. The point is that you say it could be too small, 2 but it could be too small by a very small amount, it could 3 be too large by a very small amount, it could be different. 4 Without doing some kind of an analysis I could not give you 5 any specific opinion at this time. 6 Q. So you can't tell me as we sit here today whether 7 the estimated adjustments would look more reliable than they 8 really are if mistakes in the assumptions in the smoothing 9 model cause the standard errors to be too small, is that 10 your testimony? 11 A. If we really knew that the standard errors were 12 too small, then the answer to your question would be yes. 13 Q. The smooth adjustment factors are affected by 14 sampling error, aren't they? 15 A. Like any sample estimate, they have sampling 16 error, yes. 17 Q. And the variance of the smooth adjustment factors 18 is obtained by squaring the standard error of the smooth 19 adjustment factors, right? 20 A. That's right. 21 Q. And the variance from the smoothing model is one 22 of the inputs in the loss function analysis the Census 23 Bureau conducted, right? 24 A. That's right. 25 Q. You are aware, aren't you, that the Undercount 409 1 Steering Committee issued a report to the Census Bureau 2 director on adjustments, the one I just showed you, right? 3 A. That's right. 4 Q. And you are aware that the Undercount Steering 5 Committee issued an addendum to that original report, right? 6 A. Yes, I am aware that there was an addendum. 7 Q. And the addendum to the report states that the 8 Undercount Steering Committee had originally understated the 9 variance of the estimates from the smoothing model by a 10 range of 1.7 to 3.0, isn't that right? 11 A. Let me take a look. 12 (Pause) 13 Which one his that? 14 Q. Pardon me? 15 That is Plaintiff's Exhibit 540 A.16.A and it 16 starts at Bates number 6074. 17 MR. SOLOMON: What is the page number of the 18 document other than the page number at the top? 19 MR. ZITCOV: There are a number of different 20 pages. 21 It is the eleventh page from the back, which 22 happens to be numbered page 5. 23 (Pause) 24 MR. ZITCOV: Your Honor, I made a mistake, that 25 is not the right Bates stamp number that I gave you. It's 410 1 1146 for the Undercount Steering Committee report. My 2 apologies. 3 A. You said 540 A? 4 Q. No, I made a mistake, Doctor Ericksen. 5 It is in Plaintiff's Exhibit 540 A and it begins 6 at 1146. 7 MR. ZITCOV: Can I approach the witness and I 8 will find it for him? 9 THE COURT: Certainly. 10 MR. ZITCOV: And I will be happy to find it for 11 your Honor if you would like. 12 THE COURT: I have it. 13 (Pause) 14 MR. ZITCOV: Your Honor, it is page 1202. 15 Q. I would like you to take a look at the paragraph 16 that begins at the bottom of page 1202 of the so-called 17 administrative record and it says, "Summary reaction and 18 judgment. 19 It is likely that 1 A, 1 B and 1 C add 20 considerably to the total uncertainty. It is possible the 21 contribution from 1 D is not trivial as well, although none 22 of the evidence so far elucidates the effect of 1 D as 23 separately important." 24 As a matter of judgment, the total understatement 25 of variance from the smoothing model may be in the range of 411 1 a factor of 1.7 to 3.0 in terms of variance or 1.3 to 1.7 on 2 the standard error." 3 MR. ZIMROTH: Maybe we ought to ride the rest of 4 that so it is in context. 5 MR. ZITCOV: All I am asking, your Honor, is 6 whether or not that refreshes your recollection that the 7 addendum to the Undercount Steering Committee report 8 suggested that the estimates from the smoothing model of 9 variance might have been underestimated by a factor in the 10 range of 1.7 to 3? 11 MR. ZIMROTH: I think it is only fair if you are 12 going to read the first half of the paragraph you read the 13 second half of the paragraph in which the Undercount 14 Steering Committee says it's hard to believe that smoothing 15 did not at least produce some gains relative to the 16 unsmoothed estimates. 17 MR. ZITCOV: Your Honor, isn't that what 18 cross-examination is for? 19 THE COURT: I have no idea. 20 Can you answer it? 21 THE WITNESS: I think he is just asking me if I 22 remember that this paragraph is in the document, and the 23 answer to the question would be yes. 24 Q. And does that refresh your recollection that the 25 Undercount Steering Committee said the original report had 412 1 understated the variance of the estimates from the smoothing 2 model by a factor in the range of 1.7 to 3? 3 A. What it actually says it may be in the range of 4 1.7 to 3. 5 Since this was after the July 15 date when I 6 actually had a chance to read this document, I've not had 7 the opportunity to discuss this issue with anybody from the 8 Census Bureau to find out their reasoning for this, what 9 kind of evidence they have for it. It certainly is not in 10 this document. That's an issue that I would be interested 11 in learning more about. 12 Q. You considered at your deposition, didn't you 13 consider it a problem to be considered with regard to the 14 accuracy of adjustment? 15 A. Isn't that what I just said now? Yes. 16 Q. One of the outputs to the production smoothing 17 model was the variance co-variance matrix for the raw 18 adjustment factors, is that right? 19 A. One of the outputs? 20 Q. I'm sorry, inputs. 21 A. Oh. That was a long question. Now give it to me 22 right. 23 Q. One of the inputs to the production smoothing 24 model was the variance co-variance matrix for the raw 25 adjustment factors, is that right? 413 1 A. That's right. 2 Q. None of the P studies considered the reliability 3 of that study, did they? 4 A. None of the P studies considered the reliability 5 of that program, that's correct. 6 Q. And none of the P studies considered the quality 7 control work on that program, did they? 8 A. No. Presumably the Bureau felt it had done its 9 research elsewhere, it did not include them in the P 10 studies. 11 Q. And none of the P studies addressed the 12 reliability of variance pre-smoothing, did they? 13 A. Not in the P studies, no. 14 Q. And as of the time I took your deposition five 15 weeks ago, you had not undertaken any studies of the 16 reliability of the variance co-variance matrix for the 17 smooth adjustment factors, had you? 18 A. By reliability I'm talking about the errors in 19 that due to sampling variability, I had not conducted my own 20 study, that is correct. 21 Q. So the answer is as of the time of your 22 deposition, you had not undertaken any such study? 23 A. I believe that's just what I just said, yes. 24 Q. In your opinion as an expert, you would consider 25 a sample survey that had a fabrication rate of ten percent 414 1 to be unacceptable, isn't that right? 2 A. I would be highly concerned about a fabrication 3 rate of ten percent. If I was in a situation where I had to 4 make a decision based on the results of such a survey and I 5 had no opportunity to go out and collect more data, I would 6 perhaps look at certain things like what would happen to the 7 results if you exclude the fabricated data. But I certainly 8 would be concerned about a survey with a fabrication rate of 9 ten percent, that is true. 10 Q. Do you recall -- 11 A. Are we through with this? 12 Q. Pardon me? 13 Yes. 14 Do you recall testifying at your deposition that 15 you would consider a sample survey to be unacceptable if it 16 had a fabrication rate of ten percent? 17 A. I don't recall that particular passage. If it's 18 in there, I'm sure it's in there. If you show me where it 19 was, I could explain to you what I meant. 20 MR. ZIMROTH: Your Honor, this is a completely 21 misuse of the deposition. This witness didn't say anything 22 inconsistent with that, he didn't say that his recollection 23 needed refreshing and -- 24 THE COURT: How can we know if it is inconsistent 25 unless we allow him to ask the question? 415 1 MR. ZIMROTH: Because he is reading from the 2 deposition before he has asked him the question directly. 3 He has asked him a question about the ten percent, he 4 answered the question, now he says he needs the deposition 5 to refresh his recollection. 6 Let him look at the deposition. 7 THE COURT: Can you do it that way, Mr. Sitcov? 8 MR. ZITCOV: I thought I was doing it in the 9 normal way, I asked the question and he answered it 10 differently than he answered it in the deposition and I 11 wanted to know now if I can refresh his recollection, that's 12 all. 13 I don't think that what I am doing is unfair, 14 even unique. It's the kind of thing that happens normally. 15 THE COURT: How much longer do you have? 16 MR. ZITCOV: For this witness? 17 THE COURT: Yes. 18 MR. ZITCOV: I would say at least an hour or so, 19 maybe more. 20 THE COURT: All right. Continue. 21 BY MR. ZITCOV: 22 Q. Could you turn to page 491, please. 23 A. Don't have it. 24 Q. You don't have it? 25 A. No. You only gave me up to 459. 416 1 (Handing to the witness) 2 (Pause) 3 Q. Will you turn to page 491, please. 4 Tell me when you have it. 5 Are you there? 6 A. I'm here. 7 MR. ZIMROTH: Your Honor, this just proves the 8 point. There is nothing inconsistent in this passage -- 9 THE COURT: I don't know. I don't have it in 10 front of me. 11 MR. ZIMROTH: I'm sorry, your Honor. 12 Q. "Q. So there is no rate of fabrications that 13 in your opinion would automatically be unacceptable? 14 "A. No. 15 "Q. 50 percent? 16 "A. Would not be acceptance. 17 "Q. Okay. 40 percent? 18 "A. Forty percent would not be acceptable. 19 "Q. Thirty percent? 20 "A. Thirty percent would not be acceptable. 21 And it's also unheard of in any survey I worked on. 22 "Q. Twenty percent. 23 "A. The same. 24 "Q. Ten percent? 25 "A. The same." 417 1 Does that refresh your recollection that you 2 testified without qualification at your deposition that a 3 sample survey that had a rate of fabrication of ten percent 4 would automatically be unacceptable? 5 MR. ZIMROTH: Your Honor, Mr. Sitcov hasn't read 6 the questions immediately before or the questions 7 immediately after and that's why what he is doing now is 8 entirely unfair. 9 MR. ZITCOV: Your Honor, it seems to me if they 10 want to cover that on redirect they are entirely able and it 11 would be appropriate for them to do so. 12 THE COURT: The crucial question is whether the 13 witness understands what is going on and is comfortable 14 answering the question as being put. 15 THE WITNESS: Well, look, he is asking me 16 questions I've never dealt with in my whole career of 17 working with survey data. 18 He started asking me a series of questions at 19 which point it would be unacceptable. We got down to eight 20 percent in the deposition, I said I don't know; now it's ten 21 percent; I don't know. Ask me next month it might be 12 22 percent. 23 These are not hard and fast numbers that you sit 24 there and make automatic decisions with. I tried to answer 25 his questions as best I could at the time of the deposition. 418 1 It's a totally hypothetical question. 2 Q. So there are sample surveys that you would 3 consider acceptable that had a 12 percent rate of -- 4 A. It is possible. Like I just explained to you, 5 Mr. Sitcov, that if I had to make a decision and I knew that 6 there was a 12 percent fabrication rate and I had to make a 7 decision based on the data, I would do some kind of 8 sensitivity analysis where I would remove the fabricated 9 data to see if the results changed. If you have to make a 10 decision based on data, you have to make a decision. 11 THE COURT: You play with the cards you are 12 dealt? 13 THE WITNESS: That's right. 14 Q. Fabrication is a source of bias in the PES, isn't 15 it? 16 A. It is. 17 Q. Fabrications in the P-sample could cause an 18 overstatement of the net undercount, couldn't they? 19 A. You need to give me some more information to 20 completely answer that question, but I can imagine a likely 21 scenario where fabrications in the P-sample would overstate 22 the undercount. 23 Q. You don't know how many fabrications there were 24 in the PES, do you? 25 A. I know what the Census Bureau reported. I do not 419 1 know for a fact what that number was. Like any other 2 estimate, it's based on sample data. We know it with 3 uncertainty. 4 Q. So you don't know how many fabrications that were 5 in the PES? 6 A. Not with 100 percent certainty, no. 7 Q. Would it be fair to say based on everything you 8 heard, you don't think the number of fabrications in the PES 9 is particularly large? 10 A. Relative to the overall size of the PES sample, I 11 do not consider the number of fabrications to be overly 12 large. 13 Q. And one basis for that opinion is the P5a study, 14 isn't that right? 15 A. It's based on the Census Bureau report and P5a, I 16 believe that is the basis, one basis for that opinion, yes. 17 Q. Could you take a look at what is before you as 18 Defendant's Exhibit 109. It is part of a plaintiff's 19 exhibits, the exhibit that has all of the P studies in them, 20 and it is Bates number 4971. 21 THE COURT: What is the corresponding plaintiff's 22 exhibit number? 23 MR. ZIMROTH: It should be 540 A 5 A? 24 Is it P5a, Gene? 25 THE COURT: We will fill it in later. 420 1 MR. ZIMROTH: It would be 540 A 5 A. 2 THE COURT: 540 A 5 A. 3 BY MR. ZITCOV: 4 Q. You have Plaintiff's Exhibit 195 in front of you? 5 That is right at your right hand. 6 Can you turn to tab C. That is appendix C of the 7 report on the 1990 Decennial Census and PES that you 8 submitted with Estrada, Tukey and Walter, is that right? 9 A. That's right. 10 Q. Can you turn to page 16 of appendix C, and that 11 page begins with an underscored heading, "Imputation, 12 fabrication, census day addressed in this reporting 13 erroneous enumeration," and you say, "In this section we 14 report on four additional PES evaluation studies, P3, 15 imputation reanalysis, P4, census day address misreporting, 16 P5a, P-sample fabrication, and P5a matching reanalysis. We 17 first dismiss project P5a as a Census Bureau reports that 18 only 13 fabrications with a weighted sum of 64,667 persons 19 were found. 20 Is it your opinion that there was only 13 21 fabrications in the PES? 22 A. It's my opinion based on the fact that the Census 23 Bureau had a very extensive fabrication control procedure 24 built into the PES and based on the results of this study 25 that the number of fabrications was very small, given that 421 1 it's an estimate based on a sample, we don't know that it's 2 exactly 13, it's 13 plus or minus some small quantity, so I 3 don't know that it's exactly 13, it's probably quite close 4 to it, though. 5 Q. Okay. 6 Now, weighted to the entire population those 13 7 fabrications added almost 65,000 fabricated people to the 8 population, is that right? 9 A. That's right. 10 Q. That is a number 5,000 times as large as the 11 fabrications, isn't that right? 12 A. That's right, because of the nature -- that's 13 because it was a sample in the evaluation followup study and 14 you apply the weights due to sampling and it comes to 5,000 15 times 13, 65,000, that's right. 16 Q. So just a few mistakes in the post-enumeration 17 survey can lead to a big mistake in the adjustment, isn't 18 that right? 19 A. No. You are confusing apples and oranges here, 20 Mr. Sitcov. 21 The estimate of 65,000 is based on the evaluation 22 followup study. That was a subsample of the PES, and the 23 evaluation followup study was used to evaluate the quality 24 of the PES data. So if there was an error in the evaluation 25 followup study that P5a is based upon, it would not affect 422 1 the adjustment, it would affect our assessment of the 2 quality of the adjustment data. 3 Q. I see. 4 Now, in your report to the Secretary, you didn't 5 state anywhere that there were more than 15 fabrications in 6 the PES, did you? 7 A. I only informed the Secretary of the Census 8 Bureau reports that there were 13 fabrications. That's what 9 we did. 10 Q. Could you turn to page 9 of P5a. 11 Do you have it? 12 A. I have it. 13 Q. Okay. 14 Now, the last paragraph on page 9 says, "When 15 reviewing the results, the limitations of the study should 16 be kept in mind. The data for the study were collected in 17 the evaluation file which was not designed specifically to 18 detect fabrication. In the PES, if a case did not fall in 19 the quality control sample or if a fictitious case remained 20 undetected in the quality control effort, the case might 21 have been identified as fictitious by the after followup 22 matching operation. However, matchers must establish that a 23 number of criteria have been met before a case is determined 24 to be fictitious. These criteria depend on information 25 provided by the interviewers. 423 1 The evaluation followup matching used the PES 2 following matching procedures. No additional quality 3 control or matching procedures were implemented in the 4 evaluation followup to detect fabrication. Thus, it is 5 possible that the evaluation followup did not identify more 6 cases as fictitious because there was not enough new and 7 additional interviewer information to establish that the 8 cases were fictitious in the PES." 9 Now, you didn't advise the Secretary in your 10 report that the data for the P5a study were collected in the 11 evaluation followup which was not designed specifically to 12 detect fabrications in the PES, did you? 13 A. We did not include that in our report, that is 14 correct. 15 Q. And you didn't advise the Secretary that the only 16 cases that were even considered in P5a were those in the 17 quality control sample drawing from the PES, did you? 18 A. What I informed the Secretary is the conclusion 19 that the Census Bureau drew. The Census Bureau concluded, 20 keeping this qualification in mind without doubt, that the 21 best estimate of the number of fabrications was 13 weighted 22 up to 65,000 and that's what in their judgment they put in 23 the total error model. Therefore, relying on the Census 24 Bureau's judgment in the matter, that's what I reported to 25 the Secretary, yes. 424 1 Q. You didn't advise the Secretary that the only 2 cases that were even considered of P5a were those in the 3 quality control sample drawn from the PES, did you? 4 A. My report says what it says. 5 Q. Does it say that? 6 A. No, it does not. 7 Q. And you didn't advise the Secretary that no 8 additional quality control or matching procedures were 9 implemented in evaluation followup to detect fabrications, 10 did you? 11 MR. ZIMROTH: Is there a claim in this case the 12 Secretary didn't have the Census Bureau's own reports on 13 this subject? Is that the claim in the case? 14 I'm really puzzled by this waste of time, your 15 Honor. 16 THE COURT: The claim is that the witness did not 17 advise the Secretary and, therefore, the Secretary did not 18 know it? 19 MR. ZITCOV: That's right. 20 THE COURT: But if the Census Bureau had told him 21 he would know it? 22 MR. ZIMROTH: Right. 23 Now I am asking whether there is a claim in this 24 case whether the Secretary did not have this full massive 25 volume of reports in front of him. 425 1 Was this witness supposed to, you know, just 2 simply repeat everything in the reports? He was doing a 3 summary. He was reporting on what this bureau itself had 4 concluded. 5 THE COURT: Mr. Sitcov? 6 MR. ZITCOV: Your Honor, it is certainly relevant 7 that the witness was quite selective in what he included in 8 his report to the Secretary about fabrications. 9 The witness selected some information from this 10 report and didn't select other of the information. 11 THE COURT: And it is relevant because the 12 Secretary might then choose to ignore the report feeling 13 that it had been deliberately skewed? 14 MR. ZITCOV: Or just to take the -- 15 MR. ZIMROTH: He didn't audit the report, that's 16 the point. 17 THE COURT: He is trying to justify it. 18 Is that the idea? 19 MR. ZITCOV: That is one. 20 THE COURT: Do I have to make these ideas up for 21 you? 22 MR. ZITCOV: No. Another one is that the report 23 for panelists was skewed. 24 MR. ZIMROTH: Your Honor, this witness has 25 already testified that when he made -- when he wrote his 426 1 report on June 21, he did not have the final version of all 2 of these reports, he has testified to that effect, he had 3 underlying data. He put into the report what he thought was 4 relevant. 5 The Secretary had in front of him all of these 6 reports upon which to make his basis. The Census Bureau 7 came to precisely the conclusion that Mr. Ericksen said it 8 came to. 9 MR. ZITCOV: Your Honor, if all we needed was the 10 reports, then we wouldn't need a trial. Obviously if this 11 would be a case to be decided on the record, we wouldn't 12 need Dr. Ericksen to testify to anything -- 13 MR. ZIMROTH: It would be nice to have five 14 minutes worth of cross-examination on what Dr. Ericksen 15 spend a day and a half testifying about yesterday and the 16 day before. Not a word. 17 THE COURT: It does seem -- 18 MR. ZITCOV: He -- 19 MR. ZIMROTH: He wasn't called as an expert in 20 loss function analysis and Mr. Sitcov spends an hour 21 cross-examining about it, especially when we are going to 22 have an expert on loss function analysis. 23 He wasn't here to testify about smoothing. Mr. 24 Sitcov knows we have one of the leading experts in the 25 country here on that subject to come testify and he is 427 1 wasting Dr. Ericksen's time and our time cross-examining on 2 that subject as well. 3 MR. ZITCOV: Your Honor, I guess the plaintiff's 4 position is that Dr. Ericksen should be able to introduce 5 all of the report that he and the other three authored, that 6 he should be able to introduce all of the P studies and 7 their underlying data, that he should be able to testify 8 about adjustment and smoothing and that we should not be 9 able to question him about the things that he has introduced 10 and testified about. 11 THE COURT: I think you have carried this 12 dialogue far enough. 13 If the point you are now trying to make is that 14 the witness was selective in what he told the Secretary, 15 that point has been made. Let's move on to something else. 16 MR. ZITCOV: Okay. 17 BY MR. ZITCOV: 18 Q. Do you consider it likely that there were -- you 19 said that there may have been more than 13 fabrications, is 20 that right? 21 A. It's possible. 22 Q. Do you consider it likely there were as many as 23 30 fabrications in the PES? 24 A. Look, I have not conducted my own independent 25 study of fabrications in PES, I only know what the census 428 1 report is. 2 Q. I am asking you, do you consider that likely? 3 A. It's a possibility, but I don't have any opinion 4 as to whether or not it is likely. 5 Q. If instead of 13 fabrications in the PES, there 6 were a thousand fabrications in the PES, would your opinion 7 about adjustment change? 8 A. I would have to consider it in the context of 9 everything else that we know about the PES and the original 10 enumeration, yes. 11 Q. Do you consider it likely there were as many as a 12 thousand fabrications in the PES? 13 THE COURT: Asked and answered. Next. 14 Q. What if there were 6,000 fabrications in the PES, 15 would that cause your opinion about adjustment to change? 16 A. Look, like everything else about the PES, if you 17 present me with some new information, I would want to 18 consider it in context with everything else, I would want to 19 confer with the Census Bureau's statisticians that developed 20 these estimates. I would certainly consider it. How it 21 would all come out, I don't know at this time. 22 Q. Would you take a look at Defendant's Exhibit 102, 23 please you. This is one that has not been introduced 24 before. 25 THE COURT: Defendant's Exhibit 102, an original. 429 1 MR. ZITCOV: An original by Dr. Ericksen, your 2 Honor, a double whammy. 3 (Pause) 4 BY MR. ZITCOV: 5 Q. This is an article that you wrote about 6 estimating the population in the 1990 -- using the 1980 7 Decennial Census, is that right? 8 A. It is. 9 Q. And you advocated an adjustment of the 1980 10 Decennial Census in this article, didn't you? 11 A. By the time this article -- I certainly made an 12 argument that the 1980 census would have been improved by 13 adjustment, yes. 14 MR. ZITCOV: Your Honor, we move for admission of 15 Defendant's Exhibit 102. 16 MR. ZIMROTH: No objection. 17 THE COURT: Defendant's Exhibit 102 is admitted. 18 (Defendant's Exhibit 102 marked for 19 identification was received in evidence.) 20 BY MR. ZITCOV: 21 Q. Would you turn to page 109 of Defendant's Exhibit 22 102, please. 23 That is a comment on your article by Barbara A. 24 Bailar? 25 MR. ZIMROTH: That is another story, your Honor. 430 1 I'm sorry, I didn't look at the whole exhibit, your Honor, 2 and 102, which was represented to be an article by Professor 3 Ericksen and his colleague Joseph Kadane, has attached to it 4 a comment by some, by another person, Barbara Bailar and 5 several other comments as well. 6 THE COURT: So? 7 MR. ZIMROTH: So, therefore, I have no objection 8 to the admissibility of 102 insofar as it is the article by 9 Ericksen and Kadane, and I do have an objection with respect 10 to the others. 11 THE COURT: Mr. Sitcov? 12 BY MR. ZITCOV: 13 Q. Dr. Bailar was commenting on your -- the article 14 you wrote with Kadane, wasn't she? 15 A. She was. 16 Q. And you recognize that to be the comment that was 17 published under her name in reaction to your article? 18 A. I do. 19 THE COURT: You still object? 20 MR. ZIMROTH: Excuse me just one second. 21 (Pause) 22 MR. ZIMROTH: Your Honor, I'm just puzzled. I 23 mean, I have no objection if the witness wants to -- excuse 24 me -- if Mr. Sitcov wants to examine Dr. Ericksen about 25 anything that is in any of these comments. It's just a 431 1 puzzle to me why that makes it admissible into evidence. 2 THE COURT: It probably doesn't. 3 MR. ZIMROTH: Yes. 4 THE COURT: Are you going to withdraw the 5 objection? 6 MR. ZIMROTH: Yes, I am going to withdraw the 7 objection. 8 THE COURT: Everything is admitted in 102. 9 MR. ZITCOV: Thank you. 10 THE COURT: Maybe this would be a good point to 11 take a ten minute break. We are going to be adjourning for 12 lunch at noon, so let's keep the break to precisely 10 13 minutes. 14 (Recess) 15 THE COURT: Are we ready? 16 MR. ZITCOV: Yes. 17 THE COURT: Mr. Sitcov, you may resume. 18 MR. ZITCOV: And I will speak more slowly. 19 THE COURT: Good. 20 BY MR. ZITCOV: 21 Q. Do you have Defendant's Exhibit 102 in front of 22 you? 23 A. Yes, I do. 24 Q. Is it opened to page 109? 25 A. Yes, it is. 432 1 Q. And that is a comment on your article by Barbara 2 Bailar? 3 A. Yes, it is. 4 Q. And Barbara Bailar is one of the experts that 5 plaintiffs have included in their list of expert witnesses 6 that they intend to call in this case, is that right? 7 A. She is. 8 Q. Curbstoning is a synonym for fabrications in the 9 survey, isn't that right? 10 A. It's one synonym for fabrications, yes. 11 Q. Could you turn to page 112 of Dr. Bailar's 12 article. 13 In the paragraph in the left column at the top 14 that carries over from page 111 there is a statement that 15 says, beginning in the third line on the top, "For the other 16 system, the CES, there was no acknowledgment for erroneous 17 conclusions, yet the authors used my own estimates of .6 18 percent to 4 percent of the CES interviewers curbstone." 19 Do you consider that to be a reasonable estimate 20 of curbstoning in the CES? 21 A. I don't know. I haven't seen any real study of 22 curbstoning in the PES. Absent such a study -- 23 MR. ZIMROTH: In the CES. 24 A. You are talking about the current population 25 survey? 433 1 Q. Yes. ? 2 A. Absence seeing such a study, I don't have a real 3 opinion of that. 4 Q. You consider that to be a reasonable estimate of 5 curbstoning in the PES? 6 A. Absolutely not. The PES is totally different 7 from the current population survey. In the post-interview 8 surveys there was a procedure included to defect 9 fabrication. Where an interviewer was found to fabricate, 10 that work was done by another interviewer and that's the 11 reason why the number of fabrications is so slow in the PES, 12 why it is reasonable to believe that. 13 So whatever may be true of the current population 14 survey has nothing to do with whatever may be true about the 15 post-enumeration survey. 16 Q. The 1990 PES was a sample survey of almost 17 400,000 people, is that right? 18 A. It was. 19 Q. You don't of any other sample survey of that size 20 conducted in the United States, do you? 21 THE COURT: Ever? 22 MR. ZITCOV: Yes. 23 A. I'm quite confident that there have been, but I 24 couldn't tell you what they were. 25 Q. The CES is a much smaller survey than the 1990 434 1 PES, isn't that right? 2 A. It is a smaller survey than the PES, yes. 3 Q. How big is the CES? 4 A Well, the size of the CES has varied. The best 5 of my recollection every month they interview between 50- 6 and 70,000 households. 7 Q. And the CES uses professional interviewers, 8 doesn't it? 9 A. It does. 10 Q. Now, if a rate of fabrications in the PES was .6 11 percent, which is Dr. Bailar's low estimate, that would 12 equal about 2,400 fabrications, is that right? 13 MR. ZIMROTH: Excuse me. Are we talking about 14 the CES or PES? 15 MR. ZITCOV: PES. 16 MR. ZIMROTH: Dr. Bailar's estimate is about the 17 CES. 18 MR. ZITCOV: That's correct. 19 MR. ZIMROTH: I'm confused. 20 Q. My question is if the rate of fabrication in the 21 PES was .6 percent, which is Dr. Bailar's low estimate for 22 the CES -- 23 MR. ZIMROTH: Which this witness said he had no 24 opinion about. 25 THE COURT: Let him finish the question. 435 1 MR. ZITCOV: Thank you, your Honor? 2 Q. -- that would equal about 2,400 fabrications in 3 the PES? 4 A. That's right. 5 Q. And if we to take her high estimate of 4 percent 6 fabrications in the CES, that would be about 16,000 7 fabrications in the PES, wouldn't it? 8 THE COURT: Just doing the math? 9 MR. ZITCOV: Yes. 10 A. Look, your question doesn't make any sense, but 11 if you want me to say multiply four percent by 400,000, the 12 answer would be -- let me make sure I got my zeros right -- 13 then the -- 14 Q. That would be about 16,000, wouldn't it? 15 A. That is -- four percent times 400,000 is 16,000. 16 Q. Now I believe you testified that P5a, which was 17 -- which we talked about a moment ago was not designed 18 specifically to defect fabrications in the PES, is that 19 right? 20 A. The document said what it said. 21 Q. There was a P study that was designed for the 22 purpose of detecdting fabrications in the PES, wasn't there? 23 A. Well, I actually believe that it was the intent 24 of the Census Bureau to use P5a to get an estimate of 25 fabrications that went into the total error model. So if 436 1 that's what they did, that's my understanding of their 2 intent and I believe that was always the plan. 3 Q. Will you turn to Exhibit 110, please. 4 MR. ZIMROTH: Defendant's Exhibit 110? 5 MR. ZITCOV: Defendant's Exhibit 110, which is 6 also in Plaintiff's Exhibit 540 A, I don't know what the 7 order is. This is P6. 8 This is also, your Honor, a P study and it is one 9 of the ones in that -- I guess it is Plaintiff's Exhibit 10 540. 11 MR. ZIMROTH: 540 A series, the Census Bureau 12 studies, your Honor. 13 Q. Do you see P6 there? 14 A. I do. 15 Q. And that says, "Final report for PES evaluation 16 P6 fabrication in the P-sample interviewer effect," is that 17 right? 18 MR. ZIMROTH: Just for the record I want to say 19 it is dated July 10, 1991. 20 MR. SITCOV: Well, I guess, your Honor, you 21 better say for the record that P5a is dated July 9, 1991. 22 MR. ZIMROTH: That would clarify a lot, actually, 23 if you said that. 24 Q. So this is the report, this report you are 25 looking at, says, "Final report for PES evaluation project 437 1 P6 fabrications in the P-sample interviewer effect," right? 2 We are looking at the same one? 3 A. That's right. This was not something that went 4 into my June 21 report. 5 Q. Could you turn to page 12 of P6. 6 Do you see in the last paragraph it states, 7 "There has been speculation that in data collection such as 8 the Census Bureau's current surveys, between one half and 9 one and a half of the interviews are fabricated." It cites 10 Beamer and Stokes 1989. 11 Do you see that? 12 MR. ZIMROTH: I don't have that on that page. 13 On page 6? 14 MR. ZITCOV: Page 12 of P6. 15 MR. ZIMROTH: Page 12? 16 MR. ZITCOV: Yes. 17 MR. ZIMROTH: One second. 18 Where are you reading from? 19 MR. ZITCOV: The last paragraph. 20 MR. ZIMROTH: Would you read it began, please? 21 I'm sorry, could you read it again. 22 MR. ZITCOV: "It has been the speculation that in 23 data collection such as the Census Bureau's current surveys, 24 between one half and one and a half percent of the 25 interviews are fabricated. Beamer and Stokes, 1989." 438 1 Q. Do you see that? 2 A. Yes. 3 Q. So we have established already that the PES had 4 about 400,000 people, correct? 5 A. That's right. 6 Q. So using the range from page 12 of P6, there 7 would be between 2,000 and 6,000 fabrications in the PES, is 8 that right? 9 A. No, it's not right, because I explained to you 10 before in the current population surveys there is no 11 procedure for detecting fabrications and redoing the 12 interviewers' work. In the post-interview work there was a 13 procedure detecting fabrications and when they were found 14 the interviewers work was redone. 15 Q. I am not asking you if you think that's the 16 range, what I am asking you if we used that range and 17 applied it to the PES, there would be between 2,000 and 18 6,000 fabrications in the PES; is that right? 19 MR. ZIMROTH: Are you asking him to do the 20 arithmetic for you? Is that what you are asking? 21 MR. SITCOV: Yes. 22 MR. ZIMROTH: Okay. 23 THE COURT: Maybe we can take your word on this, 24 Mr. Sitcov. 25 A. It comes out. 439 1 Q. Good. 2 Now, according to page 12, the lowest estimated 3 rate of fabricated interviews in the PES in a regional 4 office was 2.1 percent, is that right? 5 A. This is an estimate of 2.1 percent of the -- 6 indicates the excess and not match rates that were expected 7 by a model the Census Bureau used. 8 The author of this report is inferring from that 9 that may indicate fabrications, but the evidence in this 10 report is quite indirect, and the important point is that 11 this was not the result that the Census Bureau included in 12 the total error model. 13 Q. So P6 suggests that the lowest rate of 14 fabrications in a regional office in the PES was 2.1 15 percent? 16 MR. ZIMROTH: I object. That is a 17 mischaracterization of the witness' testimony. 18 THE COURT: I think the witness can handle it. 19 A. This is exceedingly indirect evidence and I 20 believe that the authors of this report are only telling you 21 that there are certain part of the nonmatch rates of these 22 interviewers' workload they can't explain. It's only a very 23 indirect suggestion. 24 Q. But it does suggest that the rate, that the 25 lowest rate of fabrications in a regional office in the PES 440 1 was 2.1 percent? 2 A. Very indirectly it makes that suggestion. 3 Q. And that the highest rate of fabrications in a 4 regional office in the PES was 8.79 percent, is that right? 5 A. It very indirectly makes that suggestion. The 6 suggestion is so indirect that the Census Bureau did not 7 include his estimates in the total error model. 8 Q. In other words, according to P6, the range of 9 fabricated interviews in the PES extends from a rate equal 10 to the net undercount at the low end to a rate that is four 11 times the size of the net undercount rate at the high end, 12 is that right? 13 A. The Census Bureau did not include P6 as an 14 estimate of fabrications. The numbers in here say what they 15 say. That's all I've got to say about it. 16 Q. My question to you is, according to this, does 17 the range of fabrications that was found by P6 extend from a 18 low end where the rate is equal to the net undercount to a 19 range at the high end that is more than four times the net 20 undercount? 21 A. That's what these numbers are. 22 Q. That range is not consistent with 13 fabrications 23 in the PES, is it? 24 A. That's why the Census Bureau didn't use it. 25 Q. No. My question is, is that range consistent 441 1 with 13 fabrications in the PES? 2 A. Certainly not. 3 Q. And if the rate of the fabricated interviews in 4 the entire P-sample were 8.79 percent, that would have 5 equaled about 35,000 fabrications, isn't that right? 6 A. Close enough. I'll take your word for it. 7 Q. And if the rate of fabrications for the whole 8 P-sample had been 2.1 percent, that would have equaled about 9 8,400 people, isn't that right? 10 A. Fortunately, it probably wasn't. 11 Q. No, that's not my question. My question is, if 12 the rate of fabrication for the whole P-sample had been 2.1 13 percent, that would have equaled about 8,400 people, right? 14 A. 2.1 percent times 400,000 is equal to 8,400. 15 Q. And appendix C, I believe, of your report to the 16 Secretary, that is Plaintiff's Exhibit 195, I believe you 17 testified makes no mention of P6? 18 A. P6 was not included in the total error model so 19 we didn't see fit to measure it. 20 (Continued on the next page.) 21 22 23 24 25 442 1 Q. After all the PES field operations were 2 completed, including the production follow-up operations, 3 there were people in both the E-Sample and the P-Sample 4 whose match status was unresolved, is that right? 5 A. There were people in the P-Sample and E-Sample 6 whose matched status was unresolved, that is correct. 7 Q. And you have no opinion whether the unresolved 8 people in the PES were more like people who were resolved in 9 evaluation follow-up or who remained unresolved in the 10 evaluation follow-up, is that right? 11 A. I'm afraid I don't understand your question. 12 Q. Well, were the people in the PES who remained 13 unresolved more like the people who were resolved in the 14 evaluation follow-up or who remained unresolved in the 15 evaluation follow-up? 16 A. To say that someone was resolved in the 17 evaluation follow-up, people were resolved and unresolved in 18 the post-enumeration survey. 19 Q. Well, let me see if this helps you answer the 20 question. 21 Would you turn to page 538 of your deposition. 22 MR. ZIMROTH: Here's another sample, your Honor. 23 The witness has not said that he needed his recollection 24 refreshed. He just said he didn't understand the question. 25 THE COURT: I'll permit it. 443 1 Q. On line 17, I asked you the following question: 2 "Q. After all the follow-up operations in the 3 PES were completed, were there people in both the E-Sample 4 and the P-Sample whose match status was unresolved? 5 "A. You mean after all the fieldwork was 6 complete? 7 "Q. Yes. 8 "A. Yes, there were. 9 "Q. In your opinion, were those unresolved 10 people in the PES more like the people who were resolved in 11 the evaluation follow-up or who were unresolved in the 12 evaluation follow-up? 13 "Mr. Corwin: Could you read that back. 14 "The pending question was read. 15 "A. I don't have an opinion on that question 16 at this time." 17 Does that passage that I just read to you 18 correctly state that you don't have an opinion about whether 19 unresolved people in the PES are more like people resolved 20 in the evaluation follow-up or who remain unresolved in the 21 evaluation follow-up? 22 A. I guess your question was as clear to me then as 23 it is now, consequently, I didn't have an opinion. 24 Q. Okay. Well, you couldn't tell me at your 25 deposition whether you could even consider this question, 444 1 could you? 2 MR. ZIMROTH: He said he couldn't understand the 3 question, so how could he consider it? 4 THE COURT: Move on. 5 MR. SITCOV: Well, your Honor, if I just may ask 6 your indulgence, I asked him the question: 7 "Have you ever considered that question?" 8 And he answered: 9 "I may have. At this time I don't remember 10 considering that specific question." 11 BY MR. SITCOV: 12 Q. Does that refresh your recollection that you 13 don't know whether or not you considered that specific 14 question? 15 A. Sure does. 16 Q. After all the fieldwork was done in the 17 post-enumeration survey, models were used to impute 18 probabilities for the unresolved people, is that right? 19 A. Models were used to impute probabilities to the 20 unresolved people, that is correct. 21 Q. And these were mathematical models? 22 A. We can call them that, yes. 23 Q. For the unresolved people in the P-Sample, a 24 probability of match was imputed, is that right? 25 A. That's right. 445 1 Q. And for the unresolved people in the E-Sample, a 2 probability of correct enumeration was imputed, right? 3 A. Right. 4 Q. Is that right? 5 A. That's right. 6 Q. In the model used to impute match status for the 7 P-Sample unresolved was different from the model that was 8 used to impute correct enumeration status for the E-Sample 9 that was unresolved, is that right? 10 A. Would you do that one again? 11 Q. Sure. Were they two different models, the 12 E-Sample model and the P-Sample model? 13 A. To the best of my recollection, yes. 14 Q. And neither had been used before the 1990 PES, 15 had they? 16 A. The Census Bureau engaged in a decade of testing 17 those computation models. They continued to improve and 18 develop and make changes. To the best of my recollection, 19 there were changes in the model between the 1988 dress 20 rehearsal and the 1990 Census. To the best of my 21 recollection, those changes did not have a major impact. 22 Q. Well, they weren't the same models that had been 23 used before? 24 A. No, they were not the same models. 25 Q. Then were the differences major or minor? 446 1 MR. ZIMROTH: He just answered that. 2 A. The impact of the differences appeared to be 3 minor. 4 Q. Do you recall that at your deposition you 5 couldn't tell me whether or not the dissimilarities were 6 fundamental or important? 7 A. I don't remember the discussion at this time. 8 Q. Okay. Well, could you turn to page 544 of your 9 deposition. Actually, page 543. 10 Starting on line 24, I asked you the question: 11 "Q. You testified that there were models used 12 to determine the probability of match for correct 13 enumeration and that there was a different model for the 14 E-Sample and the P-Sample. I asked you: Had either -- and 15 I just want to know, had either model ever been used before? 16 "Mr. Zimroth: You mean exactly the same model? 17 That's what you want to know? 18 "Mr. Sitcov: Yes. 19 "The Witness: You do mean exactly the same 20 model? 21 "Q. Yes. 22 "A. No, they weren't exactly the same. 23 "Q. Okay. Are you thinking of a similar model 24 that had been used before? 25 "A. To the best of my recollection, there were 447 1 similarities between the models that had been used in the 2 dress rehearsal censuses in the PES. 3 "Q. Were the dissimilarities fundamental or 4 important or were they just minor? 5 "Mr. Zimroth: Objection. What do you mean 6 fundamental Mr. Sitcov? Important to whom and for what 7 purpose? 8 "Mr. Sitcov: If the witness doesn't understand 9 my question, he'll tell me. 10 "The Witness: I don't know what you mean by 11 fundamental." 12 Does that refresh your recollection that you 13 couldn't tell me whether the differences were fundamental in 14 the two models? 15 A. At that time, I could not remember the 16 differences in the two models. 17 Q. You are aware, aren't you, that during evaluation 18 follow-up, a subsample of E- and P-Sample unresolved cases 19 were reinterviewed, is that right? 20 A. That's right. 21 Q. And that was done by selecting a sample of 22 unresolved cases and sending interviewers into the field to 23 gather more information, is that right? 24 A. Yes. My understanding is that they went into the 25 field to find as many of the unresolved cases as they could. 448 1 This was about February 1991, to see if at that time they 2 could possibly get some information that would allow them to 3 resolve it. 4 Q. In the evaluation follow-up, the evaluation 5 follow-up interviewers had the original PES forms completed 6 for each of the subjects of the study, didn't they? 7 A. I believe they did. I'm not certain about that. 8 Q. Those were not independent reinterviews, were 9 they? 10 A. Well, if the intention was to try to assign them 11 a status other than unresolved, and all they knew was 12 whatever the characteristics were, then I don't think the 13 concept of independence and dependence would apply there. 14 Q. So are you saying that you don't know if it was 15 an independent reinterview? 16 A. No, what I'm saying is that it's not like a 17 matching study or some other kind of study where you want to 18 do a second test to see if you got it right the first time. 19 In this case, they knew it had been decided 20 unresolved and they were going to try to gather some more 21 information so they could assign it to a resolved or 22 unresolved status. 23 So what I'm telling you is that the concept of 24 independence doesn't really apply in this situation. 25 Q. If it did apply, would that be an independent 449 1 reinterview? 2 A. Doesn't make -- your question makes no sense. I 3 can't answer that question. 4 Q. Well, all right. 5 Could you turn to Defendant's Exhibit 111. 6 MR. SITCOV: Now, this is another P study, and 7 this is -- so I take it, it is in Plaintiff's Exhibit 540 A, 8 and I wouldn't know how to tell you any closer than that, 9 your Honor, what it looks like. 10 MR. ZIMROTH: Which P study is it? 11 MR. SITCOV: P 3. 12 MR. ZIMROTH: Then it should be A 3. And this 13 one is dated July 10, 1991, your Honor. 14 BY MR. SITCOV: 15 Q. Do you have it, Dr. Ericksen? 16 A. I have Exhibit 111. 17 Q. Okay. And is that the final report for 1990 PES 18 evaluation project P 3 evaluation of imputation methodology 19 for unresolved match status cases? 20 A. It is. 21 Q. This report contains the results of the 22 evaluation rematching study we were just talking about, 23 doesn't it? 24 A. It does. 25 Q. Would you turn to page 3, please. 450 1 And I direct your attention to the second full 2 paragraph. And the paragraph reads: 3 "The second comparison is between the overall 4 probability of match for PES unresolved cases (based on 5 production imputation results) and the results from the 6 evaluation follow-up. However, the percent of P-Sample 7 cases still resolve after evaluation follow-up" -- 8 MR. ZIMROTH: Excuse me. I think Mr. Sitcov is 9 misreading what's on the page. 10 MR. SITCOV: I am? 11 THE COURT: Yes. 12 BY MR. SITCOV: 13 Q. Let me start again. 14 "The second comparison is between the overall 15 probability of match for PES unresolved cases (based on 16 production imputation results) and the results from an 17 evaluation follow-up. However, the percent of P-Sample 18 cases still unresolved after evaluation follow-up is 58.55 19 percent which limits any further inferences. Even so, the 20 match probability for PES unresolved cases is 41.82 percent 21 compared to 37.16 percent overall for these cases in the 22 evaluation follow-up (when counting 41.82 percent of the 23 evaluation follow-up unresolved persons as matches)." 24 Now, you considered this paragraph to be evidence 25 in favor of the imputation model, don't you? 451 1 A. Within the limits of its applicability giving the 2 58.55 percent, the results of the P 3 study in general 3 supported the use of the imputation model. 4 Q. Now, this study evaluation follow-up data was 5 used to study the imputation model, is that right? 6 A. That's right. 7 Q. and the results of the imputation model were used 8 to fill in the missing data in the evaluation follow-up, is 9 that right? 10 A. That's right. 11 Q. Would you please turn to page 10 of the P 3 12 report. 13 Q. The chart or the table that is identified as 14 Table 3.1 presents the results that were reported in the 15 paragraph under page 3 that I just asked you about, is that 16 right? 17 A. That's right. 18 Q. And the numbers at the tops of the columns on 19 this chart represent the imputed match probabilities in 20 production, correct? 21 A. That's right. 22 Q. Okay. In other words, the first column contains 23 data for those unresolved P-Sample persons for whom the 24 model predicted a zero to 25 percent probability of being a 25 match, right? 452 1 A. That's right. 2 Q. Is that right? 3 A. Right. 4 Q. And the last column contains data for those 5 unresolved P-Sample persons for whom the model predicted 6 there was a 75 to a hundred percent probability of being a 7 match, is that right? 8 A. That's right. 9 Q. And if the model was working properly, the 10 percentage of cases matched in evaluation follow-up should 11 increase as the imputed match probability increases, is that 12 right? 13 A. Not completely. The problem with what your 14 question is asking is that when they went back out in 15 February 1991, which was in general about six months after 16 PES, the unresolved low of the proportion of the cases, they 17 still couldn't tell whether they should match them or not. 18 And in one column it was 77.25 percent of the cases they 19 still couldn't tell. 20 The only evidence that you can get from this 21 table would be the ratio of those determined to be matched 22 and those determined to be unmatched; and as you go from the 23 zero to 25 percent of the 6.38, and you add that to the 24 43.82, and that gives you a proportion which is between zero 25 and 25 percent. And those percentages go up as the impute 453 1 exceed match probability goes up. 2 Q. Well, my question is: If the model was working 3 properly, shouldn't the percentage of cases matched in the 4 evaluation follow-up increase as the imputed matched 5 probability increases? 6 A. The answer is yes, as long as the denominator in 7 that calculation is the sum of matched and unmatched cases. 8 You can't include the unresolved cases in the evaluation 9 follow-up because ten months after the census, they still 10 couldn't get the information they needed to make a 11 resolution. 12 Q. Well, Table 3.1 shows that the match probability 13 assigned by the imputation model -- that as the matched 14 probability assigned by the imputation model increases from 15 50 to 75 percent to 75 to a hundred percent, the match rate 16 in the evaluation follow-up declined, isn't that right? 17 A. No, it's not right. 18 Q. Well, in the 50 to 75 percent column, 16.54 19 percent of the cases were matches, is that right? 20 A. That is correct. 21 Q. And in the 75 to a hundred percent column, 15.21 22 percent of the cases were matches. 23 A. Right. But that's not the relevant statistic. 24 The relevant statistic is the ratio of the 16.54 to the sum 25 of 16.54 plus 15.38. And that ratio is just over 50 percent 454 1 which corresponds to the category 50 to 75 percent. That's 2 what you can tell; the fact that ten months after the census 3 65.94 percent of those cases was unresolved, that's all you 4 can say. 5 Q. The chart also shows that as match probability 6 assigned by the model goes up, the percentage unresolved in 7 the evaluation follow-up goes up rather than down, isn't 8 that right? 9 A. The proportion that were still unresolved is 10 greater at the right side of the table than it is at the 11 left side of the table. 12 Q. It's greater than the column where the model 13 predicted there would be the most matches, isn't that right? 14 A. That's right. 15 Q. You wouldn't use a model if you didn't know its 16 assumptions, would you? 17 A. As a practical statistician, am I ever to 18 understand the assumptions of a model. I'm sure in real 19 life there have been some that I have missed as would be 20 true of any other statistician I know. 21 Q. My question is: Would you use a model if you did 22 not know its assumptions? 23 A. In working in the real world, I'm sure I have in 24 the past. Once the assumptions are pointed out to me that 25 I've missed, I try to incorporate them in my thinking about 455 1 it. 2 Q. Well, can you turn to page 555 of your deposition 3 transcript. 4 Actually, turn to page 554, line 19: 5 "Q. Would you use a model before you knew the 6 assumption 7 "Mr. Zimroth: Mr. Sitcov, this is an area that 8 we went through in some length previously. I'm not going to 9 direct the witness not to answer. I'm just pointing it out 10 so perhaps you'll go on to another area. If you insist on 11 repeating this area, I'll allow the witness to answer once 12 again, if you want to continue with the pending line. Do 13 you want to have continue with the pending line? 14 "Mr. Sitcov: Yes. 15 "Can you read the pending question, please. 16 "The pending question was read. 17 "A. Would you use a model before you knew its 18 assumptions? 19 "Mr. Sitcov: Yes. 20 "A. No. In general you would want to consider 21 the assumptions of the model before you would use them.? 22 Does that refresh your recollection as you 23 testified in your deposition that you would want to know -- 24 MR. ZIMROTH: That's precisely what he testified 25 to on his cross-examination, your Honor. 456 1 THE COURT: I'll permit it. 2 Q. (Continuing) -- that you would want to know the 3 assumptions before you would use the model? 4 A. I think that's what I just said. 5 Q. Okay. At your deposition five weeks ago you 6 testified you didn't know the assumption in the production 7 model imputing match status for P-Sample unresolved or 8 correct enumeration status for E-Sample unresolved, isn't 9 that right? 10 A. That's because, as I told you before, I didn't 11 know exactly the model that the Census Bureau had decided to 12 use. 13 Q. And you didn't even know which, if any, of the 14 P-Studies described those assumptions, did you? 15 A. The P-Studies didn't describe the assumptions. 16 Q. I'm sorry? 17 A. The P-Studies did not describe the assumptions. 18 Q. And you didn't know those assumptions at the time 19 you wrote the report to the Secretary, did you? 20 A. As I told you before, I didn't know exactly which 21 model the Census Bureau had decided to use. 22 Q. Parameters in the imputation models and P-Sample 23 unresolved were estimated using data from the PES and PES 24 follow-up, is that right? 25 A. I need that question again. 457 1 Q. Were parameters in the imputation models for the 2 E- and P-Sample unresolved estimated using data from the PES 3 and PES follow-up? 4 A. I believe they were, yes. 5 Q. And estimating parameters in the imputation 6 models using data from PES and PES follow-up requires an 7 assumption that the final unresolved group are the same as 8 persons resolved in PES follow-up, is that right? 9 A. I don't remember that particular detail at this 10 time. 11 Q. Estimating parameters in the imputation models -- 12 okay, I think I just asked you that. 13 You are aware, aren't you, that weighted to the 14 national totals there were about 4 million unresolved cases 15 in the P-Sample and another 3 million in the E-Sample, is 16 that right? 17 A. I've thought of those quantities as percentages. 18 Those numbers don't sound far off, but I haven't actually 19 made that calculation. 20 Q. So if those numbers are right, there were a total 21 of 7 million unresolved in the E-Sample and the P-Sample? 22 A. If those numbers are right, 4 million plus 3 23 million equals 7 million. 24 Q. And that would be compared to the national 25 undercount estimate of about 5 million people, is that 458 1 right? 2 A. Well, the national undercount was 5 million. 3 Q. Okay. Do you have Plaintiff's Exhibit 502 in 4 front of you, or can you find it? That is some of the stuff 5 that was introduced by Mr. Zimroth yesterday. 6 That's an ethnographic evaluation of the 1990 7 Decennial Census report series? 8 A. Yes, it is. 9 Q. Okay. I'd like you to consider the following 10 hypothetical situation. 11 We are in a big city in the southwest, and we 12 have a community with the following groups, four groups: 13 Recent immigrants from Mexico and Central 14 America, native-born Mexican Americans, Blacks and Anglos. 15 In your opinion, which group would have the 16 highest census omission rate? 17 A. Give me the groups again, please. 18 Q. Okay. Recent immigrants from Mexico and Central 19 America, native-born Mexican Americans, Blacks and Anglos. 20 MR. SITCOV: And blacks what? 21 MR. SITCOV: Anglos. 22 MR. ZIMROTH: Sorry. 23 A. I don't know anything else about the economic 24 characteristics or anything else about them? 25 Q. No. 459 1 A. I don't think that I want to rank order them. In 2 general, I would consider the Blacks and the native-born 3 Mexican Americans and the recent immigrants especially to 4 the extent they're poor to have a high probability of 5 undercount. 6 In terms of quantifying the differences, without 7 more information, that's not something I can do at this 8 time. 9 Q. In your opinion, which group would have the 10 lowest census omission rate? 11 A. All, depending on the characteristics of the 12 Anglos, they might have the lower, but you'll have to tell 13 me more about their characteristics before I can really 14 answer that question. 15 Q. Okay. If I showed you ethnographic data which 16 demonstrated that the Anglos had the highest census omission 17 rate, would you change your opinion or would you reject the 18 data? 19 THE COURT: Would you change your opinion what? 20 MR. SITCOV: Or would you reject the data. 21 THE COURT: I don't know that his opinion is firm 22 enough to be changed. 23 MR. SITCOV: I thought his opinion was that he 24 thought that the Anglos would probably have the lowest 25 undercount rate, omission rate. 460 1 THE COURT: I don't think it was quite that 2 clear. 3 BY MR. SITCOV: 4 Q. Okay. Did you think that the Anglos would have 5 the lowest? 6 A. No. I said I'd need to have more information 7 about the other characteristics of the Anglos to really have 8 an opinion on that. 9 MR. SITCOV: Could I have just one moment, your 10 Honor? 11 THE COURT: Sure. 12 (Pause) 13 MR. SITCOV: I don't think I have any further 14 questions at this time. 15 (Pause) 16 THE COURT: Mr. Zimroth, any redirect? 17 MR. ZIMROTH: I do, and I would appreciate very 18 much that we have a few minutes to organize my thoughts. 19 THE COURT: If we have to wait until you organize 20 your thoughts, we will be here until next January. 21 How much time do you want? 22 MR. ZIMROTH: 15 minutes. 23 THE COURT: And then we'd come back and go for 15 24 minutes? 25 MR. ZIMROTH: Sure. 461 1 THE COURT: Will you finish in that time? 2 MR. ZIMROTH: No. Would it be better to just 3 adjourn now until this afternoon? 4 THE COURT: Why waste 15 minutes? 5 MR. ZIMROTH: Okay, fine. 6 THE COURT: Why don't you plough into it and then 7 sort it out later. 8 MR. ZIMROTH: Excuse me? 9 THE COURT: Why don't you just plough into it and 10 sort it all out during lunch. 11 MR. ZIMROTH: I would really prefer, if you don't 12 mind. 13 THE COURT: All right. Ten minutes. 14 MR. ZIMROTH: Thank you, your Honor. 15 THE COURT: Let's take a ten-minute break. 16 (Recess) 17 (In open court) 18 MR. MILLET: Your Honor, if the Court please, may 19 I raise another housekeeping matter? 20 THE COURT: Sure. 21 MR. MILLET: During the break, we learned from 22 the plaintiffs that they do intend to change the order in 23 which they had advised us and the Court that they were 24 calling their witnesses. I understand now that Dr. Bailar, 25 who is due to be following Dr. Wolter, their next witness, 462 1 will now be put back into some time next week, and that they 2 will be changing the order of some of their other witnesses 3 as well. 4 I can only repeat the request that I've made many 5 times before already, that the plaintiffs please provide us 6 with all of their exhibits organized by witness, as we have 7 done for them, so that we can promptly review them and have 8 the proper people look at them. 9 Plaintiffs during the break indicated to me that 10 they don't have copies available to us. I find that quite 11 surprising, quite frankly, with all the resources they have 12 at their disposal, that they are not able to spare us an 13 extra set and tell us what order they are in. 14 And I would ask the Court that they be instructed 15 to do so, especially now that they are changing the order on 16 us. 17 THE COURT: Mr. Zimroth, or Mr. Rifkind? 18 MR. ZIMROTH: Your Honor, my understanding is 19 that these exhibits are on their way. 20 THE COURT: On their way? 21 MR. ZIMROTH: My understanding is that these 22 exhibits are on their way. 23 THE COURT: On their way to where, Prague? 24 MR. RIFKIND: To here from my office. 25 MR. MILLET: For each witness? 463 1 MR. RIFKIND: You have -- 2 MR. ZIMROTH: You have the next witness. 3 MR. SITCOV: And I received Estrada at 2 o'clock 4 this morning. 5 MR. ZIMROTH: And you will have Rolph. That's on 6 its way down. 7 MR. MILLET: How about the rest of them? 8 MR. ZIMROTH: The rest of them we don't have. 9 You have the next three witnesses. 10 MR. MILLET: That's what we are asking for. 11 THE COURT: The next three? 12 MR. MILLET: We'd like to have all of them, just 13 as our people have provided them, so our people can get to 14 work on it. 15 THE COURT: Can you accommodate them? 16 MR. ZIMROTH: We are giving them as quickly as we 17 can. Very shortly they will have three witnesses in 18 advance. 19 MR. RIFKIND: Let me put it this way, your Honor: 20 By lunch they will have all the witnesses that appear this 21 week. And before the end of the week, they will have at 22 least the beginning of the following week's witnesses. 23 MR. MILLET: May we also have a representation 24 for the record the order of their expected witnesses? 25 MR. RIFKIND: The representation of the record, 464 1 which I have just given -- 2 THE COURT: Why don't you do it at the end of the 3 day. 4 MR. RIFKIND: Okay. 5 MR. MILLET: Thank you, your Honor. 6 THE COURT: Okay. 7 Mr. Zimroth, have you organized your thoughts? 8 MR. ZIMROTH: No, but I'm ready to proceed 9 anyway. 10 EUGENE P. ERICKSEN, resumed 11 REDIRECT EXAMINATION 12 BY MR. ZIMROTH: 13 Q. Professor Ericksen, could you please look at 14 P 16. That would be in 540 A.16. 15 (Discussion off the record) 16 MR. SITCOV: Your Honor, do you have the one that 17 I marked as 106? It might be easier for you to find. 18 THE COURT: Thank you. 19 (Pause) 20 THE COURT: I think we are all in sync. 21 MR. ZIMROTH: Okay. 22 Q. Will you please turn to table 14, Professor 23 Ericksen. 24 A. Okay, I have it. 25 Q. On your cross-examination Mr. Sitcov showed you 465 1 that chart, did he not? 2 A. Yes, he did. 3 Q. And will you take a look at the bottom row, which 4 is "national." 5 A. I see it. 6 Q. And do you see the numbers 2.11 and 0.49? 7 A. I do. 8 Q. Mr. Sitcov, in his cross-examination, asked you 9 to divide one number into the other. 10 A. That's right. 11 Q. He asked you to divide 0.49 into 2.11. Is that 12 right? 13 A. It's actually the other way around. 14 Q. I'm sorry. To divide 2.11 into 0.49? 15 A. That's right. 16 Q. And you accommodated him and you did that? 17 A. I did that. 18 Q. In your opinion, does that arithmetical exercise 19 tell you anything meaningful about the PES? 20 A. No, I don't believe it does. 21 Q. Does it tell you anything meaningful about the 22 differential undercount? 23 A. No, it does not. 24 Q. Could you explain to the Judge why that is? 25 A. Yes. The objective of the post-enumeration 466 1 survey was to produce adjustments of population in different 2 states and localities so that we could correct the problem 3 of the distribution of population that's occasioned by the 4 undercount of people. And the Census Bureau designed the 5 PES to do that. 6 And as part of the evaluation of the post- 7 enumeration survey, they defined categories which are known 8 as evaluation poststrata. And those are the groups that are 9 indicated on this table from 1 to 13. They group the 1392 10 poststrata into 13 categories. For example, evaluation 11 poststratum 1 is all minority poststrata in central cities 12 in the northeast. 13 And evaluation poststratum 2 is nonminority 14 people living in central cities in the northeast. And you 15 can see that the estimated undercount for the first 16 evaluation poststratum is 6.83, and the second was minus 17 0.75. 18 Now, the rest of the evaluation poststrata is 19 defined similarly for different areas. And the question is 20 whether the difference in the undercount between areas, 21 particularly between areas where minorities are 22 concentrated, where the undercounts are high, and areas 23 where minorities are few and undercounts are low, is 24 affected by the bias. So what one would need to do would be 25 to look at the effect of the bias on the differences in the 467 1 net undercounts between groups and areas. 2 Q. Did the Census Bureau do this division exercise 3 that Mr. Sitcov asked you to do? 4 MR. SITCOV: Object. I feel it would be 5 speculative, your Honor. 6 THE COURT: Overruled. 7 You may answer if you know. 8 THE WITNESS: No, this was not the -- the Census 9 Bureau, I'm sure, made that calculation at some point, but 10 that was not the major point made in the reporting of PES by 11 the Census Bureau. 12 Q. Okay. Did you do the -- did you look at these 13 figures in light of what you think -- what you have 14 testified is the appropriate purpose -- 15 A. Yes, I did. 16 Q. -- about these evaluations? 17 A. Yes, I did. 18 Q. And what did you find? 19 A. What I found was that the use of the PES to 20 supplement the original enumeration increased the population 21 share of areas of minority areas. And when one incorporated 22 the estimated bias for those areas from the PES, it made a 23 very small effect on the correction provided by the 24 post-enumeration survey. 25 Q. And did the Census Bureau do a similar analysis? 468 1 A. Yes, it did. 2 Q. And what did it find? 3 A. That was a loss function analysis that we've 4 talked about, and it found that when the bias and the PES 5 was taken into account, the effect on the distribution of 6 population was such that it deemed the original enumeration 7 supplemented by the PES to be more accurate than the 8 original enumeration by itself. 9 Q. Now, could you look at page -- I'm sorry. One 10 other thing. 11 Could you look at PX 9, which is the Secretary's 12 decision. Page 2-23. 13 Do you have it? 14 A. I do. 15 Q. Do you see the sentence that talks about 1.4 16 percent and the 2.1 percent? 17 A. I do. 18 Q. Is the Secretary there doing this arithmetical 19 task that Mr. Sitcov asked you to do? 20 MR. SITCOV: I object. He can testify if he 21 knows what it is. I don't know that he can say what the 22 Secretary did. 23 Q. What is that there? 24 A. The sentence that reads: 25 "The model does show that the PES is biased to 469 1 have overestimated the undercount, and that a biased 2 estimate of the undercount would be about 1.4 percent rather 3 than the production estimate of 2.1 percent." 4 That's making a statement that one-third of 2.1 5 percent is biased and deducting it from the national 6 undercount. 7 Q. And is that, in essence, what Mr. Sitcov asked 8 you to do in this exercise? 9 A. It is. 10 Q. And, in your opinion, does that say anything 11 meaningful about the PES or about the net undercount or -- 12 excuse me -- or about the differential undercount? 13 A. No, it does not. 14 Q. Could you look at page 20 of your report, which 15 is, I believe, PX 195. 16 A. Okay. 17 Q. Is that a loss function analysis that you have 18 there? 19 A. No. It's a discussion of what we anticipate a 20 loss function analysis would show. 21 Q. And what did you anticipate a loss function 22 analysis would show? 23 A. That the distribution of undercount for states 24 and other areas would be improved by use of the PES. 25 Q. And at the time you wrote this, did you have 470 1 available to you the results of the Census Bureau's loss 2 function analysis? 3 A. No. 4 Q. But, subsequently, you have seen those results? 5 A. That is correct. 6 Q. And do they comport with what you predicted that 7 they would show? 8 A. Yes, they do. 9 Q. Now, Professor Ericksen, Mr. Sitcov asked you 10 about the report, I think it was, P5a. Is that right? 11 Do you remember that? 12 A. Yes, I do. 13 Q. And he accused you of selective incorporation? 14 A. Yes, he did. 15 MR. SITCOV: Object, your Honor. 16 THE COURT: Overruled. 17 Q. Did you have the P5a, the final report of P5a 18 available to you at the time you wrote your report? 19 A. No, I did not. 20 Q. When you wrote your report, did you understand 21 that the Secretary would have available to him information 22 that was not available to you? 23 A. Yes, I did. 24 MR. ZIMROTH: I have nothing further, your Honor. 25 THE COURT: Any recross? 471 1 MR. SITCOV: I think I may. It's 5 of 12. Could 2 we break now? 3 THE COURT: Well, I think they are trying to get 4 him on an airplane. 5 MR. ZIMROTH: Yes, I'm trying to get him out of 6 here, your Honor. 7 MR. SITCOV: Could I have just one second, your 8 Honor? 9 THE COURT: Yes. 10 (Pause) 11 RECROSS-EXAMINATION 12 BY MR. SITCOV: 13 Q. Dr. Ericksen, you had the tables for all the 14 P-Studies available to you before the Secretary's decision, 15 didn't you? 16 A. I did have tables. Whether I had all the tables, 17 I have not made that check, to answer your question yes or 18 no. 19 MR. SITCOV: No other questions. 20 MR. ZIMROTH: That's it, your Honor. 21 THE COURT: Pleasure, Doctor. 22 THE WITNESS: I have enjoyed it. Thank you. 23 (Witness excused) 24 THE COURT: All right. We'll recess and resume 25 at 2:15. 472 1 MR. RIFKIND: I'd like to take a minute of 2 housekeeping and use these three minutes. 3 THE COURT: All right. 4 MR. RIFKIND: I was asked to make a 5 representation and I'd like to give a notice, if I may. 6 You may step down. 7 The order of our witnesses, is Dr. Wolter, Dr. 8 Rolph, Professor Estrada, Dr. Fisher, Dr. Cain, Dr. Bailar, 9 Professor Tukey, Dr. Fienberg. 10 The change resulted because we were running a 11 little behind of what we hoped to be. And one witness had 12 an engagement that we were going to smash into, so we moved. 13 Let me give a notice then, if I may. 14 We have a stipulation that I referred to in 15 opening covering the supposed confidentiality of the 16 adjusted tapes, the tapes that give you the corrected data 17 for every block of the country. 18 For reasons not apparent to us, although the 19 government was finally persuaded by the Magistrate to 20 produce those tapes to us, they did so only under a 21 confidentiality agreement which we reluctantly acceded to in 22 the interest of getting them. That confidentiality 23 agreement provides that on three days notice, we may move to 24 lift the confidentiality associated with it, and I give 25 notice now that on Monday morning I will so move. 473 1 I think I want to avoid arguments that some of 2 the references we may wish to make in next week's testimony 3 would breach the confidentiality if uttered in this open 4 courtroom. I'd like to avoid having to seal the courtroom, 5 or make other strange arrangements for dealing with that 6 sort of evidence, and I will assert that I see no plausible 7 basis for keeping the stuff confidential at this point. 8 But that can wait until Monday. I'm required by the 9 stipulation to give three days notice. 10 THE COURT: May I ask that you confer with the 11 government in the hopes that you can resolve this amicably 12 among yourselves? 13 MR. MILLET: I was just going to suggest that. 14 MR. RIFKIND: Okay. I raise it now -- actually, 15 we asked them if it could be waived by letter of last week, 16 and we were told no. But I'm willing to try again. 17 THE COURT: Well, I find -- 18 MR. RIFKIND: And I adhere their suggestion. 19 THE COURT: I find you to be a reasonable, 20 amicable and congenial group. If they think it will we 21 done, we'll do it. 22 MR. RIFKIND: We'll do our best. 23 (Luncheon recess) 24 25 474 1 A F T E R N O O N S E S S I O N. 2 2:30 o'clock p.m. 3 MR. SOLOMON: Plaintiffs call Dr. Kirk Wolter. 4 KIRK M. WOLTER, 5 called as a witness by the plaintiff, having 6 first been duly sworn, was examined and 7 testified as follows: 8 MR. SOLOMON: We have given to your clerk one 9 binder of documents that we will be referring to and I would 10 like to give Dr. Wolter a set as well. 11 We have given them to the government a couple of 12 days ago. 13 DIRECT EXAMINATION 14 BY MR. SOLOMON: 15 Q. Dr. Wolter, what is your occupation? 16 A. I'm a statistician and a manager. 17 Q. Where are you currently employed? 18 A. I'm employed at the A.C. Nielsen Company in 19 Northbrook, Illinois. 20 Q. How long have you been employed at Nielsen? 21 A. Roughly three and a half years. It will be four 22 years this September. 23 Q. Since about September of 1988, is that right? 24 A. That's correct. 25 Q. Prior to September of 1988, where were you 475 1 employed? 2 A. I was employed full-time at the U.S. Bureau of 3 the Census, I was employed part-time at the George 4 Washington University. 5 Q. For how long were you employed at the Census 6 Bureau? 7 A. Fourteen years. 8 Q. Dr. Wolter, have you served as a member of the 9 special advisory panel that was appointed pursuant to the 10 stipulation and order entered by Judge McLaughlin? 11 A. Yes, I did. 12 Q. What is your position at A.C. Nielsen? 13 A. I am vice president statistical design worldwide. 14 Q. And as vice president statistical design 15 worldwide, what generally are your responsibilities? 16 A. I am responsible for statistical methodology 17 worldwide. 18 We operate in 28 countries. No statistical or 19 other related quantitative procedures are used in the 20 company without my approval. The buck stops with me. 21 Q. Describe briefly for the court the business of 22 A.C. Nielsen? 23 A. Nielsen is in two general areas of business. One 24 is media research, the other is marketing research. 25 We are perhaps better known for our media 476 1 research. These are the well-known Nielsen television 2 ratings. 3 In the marketing research area, we produce 4 information on the total sales volume of all consumer 5 packaged goods sold in the United States and in 27 other 6 countries worldwide. 7 We also publish considerable information about 8 the promotional conditions or causal conditions under which 9 consumer sales occur. 10 Q. What is the chief statistical technique that you 11 as the person where the buck stops at Nielsen employ for the 12 business of Nielsen that you just described? 13 A. The field of survey sampling really is the 14 bedrock or foundation underlying all of the work done at the 15 Nielsen Company in both media research and marketing 16 research. 17 Q. Dr. Wolter, are you familiar with the 18 post-enumeration servey or the PES that was conducted by the 19 Bureau in connection with the 1990 census? 20 A. Yes, I am. 21 Q. We will be discussing the PES in great detail. 22 Let me ask you now, is that a sample survey? 23 A. Yes, it is. 24 Q. Would you give the court some idea of the 25 dimension of the use of sample surveying in the media 477 1 research business of Nielsen? 2 A. This is a, this is a huge business bringing in 3 hundreds of millions of dollars in revenue for the Nielsen 4 Company. 5 For the television industry generally, the 6 Nielsen television ratings provide a kind of currency 7 whereby business is transacted between advertisers and the 8 television networks. 9 The television industry in the United States 10 amounts to tens of billions of dollars and it literally 11 could not function as a business without this kind of 12 information produced by Nielsen. 13 Worldwide, of course, the impact of Nielsen data 14 on the television industry is even larger. 15 Q. And the statistical bedrock of all of that 16 commerce is what, sir? 17 A. Well, is, again, a series of sample surveys 18 conducted by the Nielsen Company. 19 Q. Can you give the court some idea of the dimension 20 of commerce generally that depends on survey sampling with 21 respect to the other part of the Nielsen business? 22 A. The marketing research area is an even larger 23 component of the Nielsen business, and without wanting to be 24 very precise, overall worldwide generates somewhere between 25 half a billion and a billion dollars in revenue for us. 478 1 But more than that, the entire food distribution 2 system and distribution system for consumer packaged goods 3 in the United States and in other countries is driven by 4 Nielsen data. 5 I once heard it said that during the height of 6 the military establishment in the United States, America 7 spent about as much on guns as on butter, meaning that 8 America spent as much on food as on the military, and that 9 entire food industry literally is driven by the Nielsen data 10 and is made more efficient as a consequence. 11 Q. Dr. Wolter, where were you educated? 12 A. I took my undergraduate degree at St. Olaf 13 College in Minnesota, my masters and Ph.D. are from Iowa 14 State Universsity in Ames. 15 Q. What was the subject of your doctoral 16 dissertation? 17 A. Statistical models of measurement error. 18 Q. And your Ph.D. is in statistics? 19 A. Yes, it is. 20 Q. At the time that you were working on the subject 21 of your Ph.D., was the field of sample surveying a new one? 22 A. No, no, definitely not. 23 Q. Can you trace the current history of the field as 24 of the time you were, in fact, looking toward your Ph.D.? 25 A. The modern field of survey sampling was codified 479 1 in the 1930s and 1940s. It emerged as a formal discipline 2 in statistics at that point in time. And, of course, there 3 has been very lively research and implementation of sample 4 surveys ever since, not only in this country, but in all 5 countries of the world. 6 There were informal surveys conducted well before 7 the thirties, even into the last century and perhaps before. 8 One can recall history lessons where in ancient 9 times or Biblical times there were censuses and surveys. 10 But as a modern scientific discipline, codification occurred 11 roughly from the 1930s onwards. 12 Q. Do you consider the field of survey sampling to 13 be a separate subfield within the field of statistics? 14 A. Yes, it is. 15 Q. You had mentioned when you were at the Bureau you 16 did some teaching. Where did you teach? 17 A. I taught at the George Washington University, 18 which is in Washington, D.C. 19 Q. Describe generally for the court what you -- did 20 I cut you off? I'm sorry. 21 Describe generally to the court what you did at 22 George Washington University when you taught there. 23 A. I taught course in survey sampling, I directed a 24 Ph.D. student, I served on dissertation committees of other 25 Ph.D. students, I drafted a master's degree program in 480 1 survey sampling and carried out other departmental 2 activities. 3 Q. How many years did you teach at George 4 Washington? 5 A. It was 13 or 14 years. 6 THE COURT: Full-time faculty? 7 THE WITNESS: No, it was part-time. 8 I taught one course per semester. 9 Q. How many courses per year? 10 A. Typically, two courses per year, one per 11 semester. 12 Q. Over the period that you were at George 13 Washington University, how many students did you teach 14 survey sampling to? 15 A. I would estimate between 180 and 200 students. 16 Q. Does George Washington -- 17 A. These are, these are Ph.D. level students, I 18 might add. This was not -- these were not undergraduate 19 courses. 20 Q. Did the George Washington University have a 21 subspecialty or a specialty in survey sampling? Was that 22 one of the disciplines that it specialized in in statistics? 23 A. Yes, it did, in part because of the very large 24 federal statistical establishment that was located in the 25 area. George Washington was a kind of training ground for 481 1 federal statisticians. 2 Q. In the field of survey sampling, do you consider 3 Professor Gene Ericksen to be an expert? 4 A. Yes, sir, I do. 5 Q. To your knowledge, does the University of 6 California at Berkeley have a known survey sampling 7 department? 8 A. I'm, I'm uncertain. 9 Q. Do you hold any memberships in statistics related 10 professional organizations? 11 A. Yes, I do. 12 Q. Could you tell the court which ones? 13 A. I am a member of the American Statistical 14 Association, I am an elected member of the International 15 Statistical Institute, I am a member of the Institute of 16 Mathematical Statistics, I am an executive member of the 17 American Marketing Association, I am a current scientific 18 secretary of the International Association of Survey 19 Statisticians. 20 Q. What is the job of a scientific secretary of the 21 International Association of Survey Statisticians? 22 A. This is an elected post that by and large is 23 responsible for the intellectual well-being of the 24 membership. 25 Q. Are you a Fellow of the American Statistical 482 1 Association? 2 A. Yes, I am. 3 Q. When were you elected to be a fellow? 4 A. I don't recall. It was 80, 81, 82, somewhere in 5 that general time frame. 6 Q. I am going to ask you about the work you did at 7 the Census Bureau a little more in a moment. 8 When you were there, did you receive any awards? 9 A. At the Census Bureau? Certainly I received 10 awards, yes. 11 Q. Can you describe to the court the awards you 12 received? 13 A. I received the Census Bureau's highest award, 14 which is called the Bronze Medal. 15 I received the -- 16 THE COURT: The best they can give is bronze? 17 THE WITNESS: Unfortunately, yes. At that point, 18 their parent agency takes over and reserves all the better 19 awards. 20 But I am pleased to report that I also won one of 21 their awards, their Silver Medal Award. 22 I received other certificates of recognition, 23 cash awards and the like, and I was awarded one of the 24 agency's annual EEO awards. 25 Q. The parent award that you were just talking about 483 1 was an award conferred by the Department of Commerce, is 2 that right? 3 A. Yes, that's correct. 4 Q. Since you received your Ph.D., have you received 5 any awards from Iowa State? 6 A. Yes, yes. 7 Q. Tell the court which ones, please. 8 A. Well, at the time of my Ph.D., I received an 9 award as the most outstanding Ph.D. recipient that year, and 10 then later in my career I received an award from Iowa State 11 as their most outstanding young alumnus. 12 Q. Have you written any books? 13 A. Yes, I have. I have written one book in survey 14 sampling, specifically, a specialized topic within survey 15 sampling known as variance estimation. 16 Q. Do you know whether that work is used in the 17 field of survey sampling? 18 A. Oh, yes, indeed. It's used around the world. 19 Q. Have you written or published articles on 20 subjects of survey sampling or more generally in statistics? 21 A. Yes, I have. I have written a large number of 22 articles. 23 Q. Would you look, please, at the binder that is in 24 front of you, which I think you will find in tab 1 we have 25 marked as PX 537 for identification a document that, for the 484 1 record, is the curriculum vitae of Dr. Wolter. 2 Would you look at that and tell me if you can 3 identify that? 4 A. Yes. This is a recent vitae. 5 Q. Of yours? 6 A. Yes, that's correct. 7 MR. SOLOMON: We would offer Plaintiff's Exhibit 8 537. 9 MR. MILLET: No objection, your Honor. 10 THE COURT: 537 is admitted. 11 (Plaintiff's Exhibit 537 marked for 12 identification was received in evidence.) 13 BY MR. SOLOMON: 14 Q. Dr. Wolter, returning to your career at the 15 Census Bureau, describe to the court the positions you held 16 beginning in 1974, just briefly. 17 A. From 1974 I was designated a mathematical 18 statistician in one of the Bureau's divisions known as the 19 research center for measurement. 20 From roughly 1978 I was appointed to a 21 supervisory post as assistant division chief in the Bureau's 22 statistical research division. 23 Then somewhere around 1980 I assumed the highest 24 technical post, staff post within the Bureau, that was known 25 as the chief or senior mathematical statistician. 485 1 And then by 1982, roughly, or 83, I have 2 forgotten which, I became chief of the Bureau's statistical 3 research division. 4 Q. Did you hold the position of chief of the 5 statistical research division from 1983 until you left the 6 Bureau in 1988? 7 A. Yes, I did. 8 Q. Describe, please, generally what the statistical 9 research division of the Bureau of the Census was during the 10 time you were its chief? 11 A. The stat research division was one of the major 12 research divisions within the Census Bureau. It was a group 13 of roughly 100 scientists, most with masters and Ph.D. 14 degrees in such fields as statistics, mathematics, computer 15 science, demography, political science and so forth. 16 Our mission, quite simply, was to monitor the 17 progress of the work in the rest of the Bureau, to evaluate 18 the quality of work, to work with other census executives in 19 identifying problems and opportunities in census programs, 20 to develop new methodologies and transfer them into 21 production so as to reduce cost or improve quality or 22 timeliness of Census Bureau data products. 23 We served as a kind of conduit, also, between the 24 outside world and the Census Bureau and brought developments 25 outside of the Census Bureau to the attention of Census 486 1 Bureau executives and conversely brought internal problems 2 and opportunities to the attention of outside researchers 3 and other constituencies. 4 Q. During the time that you were at the Bureau, what 5 was the reputation of the statistical research division? 6 A. I think it was a very high one. 7 Earlier you asked about the history of survey 8 sampling and I spoke about the codification of the field 9 beginning in the 1930s and to a very substantial extent the 10 statistical research division was the place or one of the 11 preeminent places in the world where this codification 12 emerged and took place. 13 The division has come to be regarded around the 14 world as one of the preeminent contributors to this field of 15 science. 16 Q. Describe for the court the role that you played, 17 generally, as the chief of the statistical research division 18 at the Bureau? 19 A. That, that is a post of administration with 20 personnel responsibilities, budget responsibilities, space 21 responsibilities, union responsibilities; it's a post of 22 professional leadership, it's a post of policy development, 23 it's a post of -- it's a major point of interface between 24 the Bureau and other organizations outside the Bureau. 25 Q. To what particular projects, if any, did you in 487 1 particular devote yourself while you were the chief of the 2 statistical research division? 3 A. I devoted my activities to all of the above. 4 Certainly one of the technical or program areas that I 5 devoted considerable attention to was the problem of 6 undercounts and miscounts in decennial censuses and, indeed, 7 of other censuses, too. 8 Q. Did you, while you were at the Bureau, devote 9 yourself to the study of the potential amelioration of 10 undercounts and miscounts in decennial censuses? 11 A. Yes. That was one of our very important 12 activities during the 1980s. 13 MR. SOLOMON: Your Honor, we would proffer Dr. 14 Wolter as an expert for any questions. 15 MR. MILLET: I'm sorry, an expert for any 16 questions? 17 MR. SOLOMON: Yes. 18 MR. MILLET: Certainly we have no objection to 19 Dr. Wolter's expertise in statistic, but I don't think I 20 could quite go as far as Mr. Solomon. 21 MR. SOLOMON: The issues that are relevant to 22 this lawsuit. 23 MR. MILLET: We have no objection to that, as 24 long as that is so. 25 THE COURT: Dr. Wolter is qualified and may 488 1 express his opinions. 2 BY MR. SOLOMON: 3 Q. Dr. Wolter, when you were at the Bureau, did the 4 Bureau conduct economic censuses? 5 A. It conducts economic censuses each five years. 6 Q. Describe very briefly to the court what economic 7 censuses are? 8 A. The economic censuses cover the economic census 9 activities of firms in the United States, firms as opposed 10 to households. 11 There is a census of retail trade, there is a 12 census of wholesale trade, there is a census of selected 13 service industries, there is a census of manufacturers, of 14 governments, of agricultural, and so forth. These censuses 15 attempt to enumerate all firms engaged in those economic 16 endeavors. 17 Q. What role, if any, did you play in connection 18 with the evaluation of any of the economic censuses taken at 19 the Bureau? 20 A. From the late 1980s I was assigned responsibility 21 for a crosscutting task force charged with the 22 responsibility for evaluating the quality of the economic 23 censuses. 24 In those years we were completing an evaluation 25 program of the 1978 round of economic censuses. 489 1 When I say 1978, incidentally, I mean that the 2 census actually takes place in 1979 regarding economic 3 activity during the calendar year 1978. 4 Then subsequent to that we began the planning of 5 the evaluation of the 1982 economic censuses. Subsequently, 6 we approved those plans, executed the plans, published the 7 results and moved on towards the 1987 economic censuses. 8 Q. What use, generally, is made of the economic 9 census figures published by the Bureau? 10 A. Massive uses. First of all -- 11 THE COURT: Shape up. 12 A. They are used by the federal government for the 13 national accounts, they are used throughout the Census 14 Bureau for sampling purposes for subsequent monthly and 15 annual surveys, they are used by private industry for sample 16 surveys, they are used by my company for weekly surveys. 17 They are a very important component of the 18 American economy. 19 Q. Is the gross national product one of the economic 20 pieces of data that rely on the economic censuses published 21 by the Bureau? 22 MR. MILLET: Objection; relevance. 23 THE COURT: Overruled. 24 You may answer. 25 A. The national accounts are produced by our sister 490 1 agency, the Bureau of Economic Analyses based on inputs from 2 the Census Bureau, the economic census being a very key 3 preeminent input. 4 Q. And the gross national product, how is that 5 derived from that data? 6 A. That's a very elaborate process. 7 Q. All I want to know is if you know whether the 8 gross national product is one of the data outputs that rely 9 on the Census Bureau inputs taken from the economic 10 censuses? 11 A. Yes, it relies on input from the economic 12 censuses. 13 Q. Were evaluations carried out with respect to the 14 1982 economic census? 15 A. Yes. 16 Q. Were those evaluations based, in part, on 17 sampling techniques? 18 A. They were based extensively on sampling 19 techniques. 20 Q. As a result of the evaluations that were carried 21 out by the Bureau, did the Bureau communicate to I think you 22 said the department, the Bureau of Economic Affairs, 23 potential improvements that could be made to the census 24 counts based on the evaluations that were based on sampling? 25 A. We published evaluations of the quality of the 491 1 1982 economic censuses. Those evaluations were made 2 available to the Bureau of Economic Analysis. 3 Q. Of what use did the Bureau make of those 4 evaluations? 5 A. The Bureau determined that it was possible to use 6 the results of those evaluations to improve or correct 7 deficiencies in the 1982 economic censuses, and so the 8 economic censuses were, in fact, corrected or adjusted for 9 those identified difficulties. 10 Q. Since the time you left the Bureau in 1988, have 11 you stayed generally abreast of the work the Bureau has done 12 in connection with its undercount research and PES? 13 A. I have stayed generally abreast, yes. 14 Q. How have you done that? 15 A. Through a variety of means. 16 First and foremost, I was a member of the special 17 advisory panel that was created as a result of your Honor's 18 stipulation and order in 1989. 19 Secondly, I have attended a range of meetings, 20 annual meetings, local meetings of professional 21 organizations, such as the American Statistical Association, 22 whereby discussions of this matter took place. 23 And finally, I've kept generally in touch to the extent that I could with former colleagues and staff members 24 at the Census Bureau. (Continued on the next page) 25 Q. Have you testified before on the subject of 492 1 census accuracy and correction of census figures? 2 A. Yes. I testified in a lawsuit in 1984 in New 3 York City concerning the completeness of the 1980 Decennial 4 Census. 5 Q. On that occasion, you were a witness called by 6 the Bureau of the Census, is that right? 7 A. That's correct. 8 Q. And have you testified under oath on deposition 9 in this matter? 10 A. In the instant matter. 11 Q. Yes. 12 A. Yes, for five days. 13 Q. In connection with those depositions or any of 14 the work that you have done in preparation for your 15 testimony here, have you received any remuneration? 16 A. No, I have not. 17 Q. Have you been promised any remuneration? 18 A. No, I have not. 19 Q. As a member of the Special Advisory Panel, can 20 you estimate how much time you spent devoted to the subject 21 of the post-enumeration survey in the correction of the 1990 22 census count? 23 A. The intensity of the work varies, of course -- it 24 did vary, of course, from the time to time. In the early 25 months of the panel, the panel tended to meet on the order 493 1 of once a month, and I devoted perhaps on the order of two 2 or three days per month to my role as a panel member. 3 Later, once all the census activities and 4 evaluation activities were under way, we tended to meet less 5 often, and I tended to devote no more than a day per month 6 to my activities on this panel. 7 Then moving beyond 1990 and into the spring of 8 1991, my activities intensified considerably, and I devoted 9 several days per month to this responsibility. There were 10 times in June of that year when I devoted very considerable 11 time to this responsibility. 12 Q. As a member of the Special Advisory Panel, did 13 you make a recommendation to have Secretary Mosbacher on the 14 question of whether or not the 1990 Census counts should be 15 statistically adjusted? 16 A. Yes, I did. 17 Q. What was your recommendation? 18 A. I recommended that the corrected counts were more 19 accurate and absolute in distributional terms than the 20 original raw census counts. And thus, on those grounds 21 principally, I recommended that he certify the corrected 22 counts as the official 1990 Census figures. 23 Q. I would like you to turn to the next tab in your 24 binder, tab 2, where we have marked for identification as PX 25 538, for the record, a June 21st, 1991 letter from Dr. 494 1 Wolter to Secretary Mosbacher. 2 Have you seen that document before, sir? 3 A. Yes, I have. 4 Q. What is it? 5 A. It's the letter that I wrote to Secretary 6 Mosbacher conveying my recommendations. 7 MR. SOLOMON: We would offer PX 538. 8 MR. MILLET: Your Honor, I have no objection to 9 all but the last page. I just showed this to Mr. Solomon 10 just before the examination began. 11 As your Honor can see, Exhibit 530 is bates stamp 12 510 and I believe it's 525. And then there's a table at the 13 end which appears to not have a bates stamp. This does not 14 bear the telefax legend that is on the bottom of the other 15 pages. I do not believe it is part of the administrative 16 record. 17 If Mr. Solomon can lay a foundation for the last 18 page, we don't object. 19 We don't object to 510 to 525. The objection is 20 limited to the last page. 21 MR. SOLOMON: May I ask a question of this 22 witness? 23 THE COURT: Yes. 24 BY MR. SOLOMON: 25 Q. Dr. Wolter, on page 14 of your letter of 495 1 recommendation, in the paragraph beginning on Tuesday, June 2 18, on the fourth line, you have a reference there to "this 3 table." And you say "See table 2 attached." 4 A. Yes, sir. 5 Q. Do you see that? 6 A. I see that passage. 7 Q. And I want you then to look at the last page of 8 this exhibit. 9 A. Yes. 10 Q. PX 538, and tell me whether that is the table 11 that was attached. 12 A. Yes, it was. 13 Q. Identify what that table is, sir. 14 A. It's an analysis performed by the Census Bureau 15 regarding the accuracy of the congressional proportion based 16 on corrected and uncorrected census counts. 17 Q. Was that attached to the letter that you sent to 18 Secretary Mosbacher? 19 A. My secretary inadvertently deleted this table 20 from the original FAX. I noted the error on the order of an 21 hour after the error occurred. I believe it was immediately 22 faxed to the Secretary at that point; so, at that point, it 23 was detached from the letter. But then we mailed a hard 24 copy of the letter to the Secretary; and, unless I'm 25 remembering this incorrectly, the entire package, with the 496 1 table, was mailed at that point in time. 2 MR. SOLOMON: We renew our offer. 3 MR. MILLET: I appreciate the explanation. I 4 withdraw the objection. 5 THE COURT: Exhibit -- what was the number? 6 MR. SOLOMON: 538. 7 THE COURT: -- 538 is admitted. 8 (Exhibit PX 538 for identification was received 9 in evidence) 10 BY MR. SOLOMON: 11 Q. Dr. Wolter, did you also participate in the 12 preparation of the joint report to the Secretary? 13 A. Yes, I did. 14 Q. Who were the coauthors on that report? 15 A. Gene Ericksen, John Tukey and Leo Estrada, 16 co-panel members. 17 Q. Can you give the Court a brief sketch of the 18 reasoning that underlay your recommendation to the 19 Secretary. 20 A. Well, generally, there is a fundamental problem 21 of differential undercount in the decennial censuses. This 22 problem was first measured for the 1940 Decennial Censuses. 23 It's been measured for each and every decennial census since 24 then. 25 For the 1990 Census, however, for the very first 497 1 time the Census Bureau was well-positioned to do something 2 about that differential undercount. For the very first 3 time, it has a very precise measurement of the differential 4 undercount. And for the very first time, it has means of 5 adding the missed and miscounted people back into the raw 6 undercount -- excuse me -- back into the raw census so as to 7 correct that raw census for the missed and miscounted 8 people. 9 It would be a great service, in my judgment, to 10 customers of census data to have the more accurate data. 11 That, essentially, was the basis for my recommendation. 12 Q. What did you rely on in making that recommenda- 13 tion to the Secretary? 14 A. Well, I relied on many inputs. 15 First of all, as I just stated, there was a 16 record of undercount measurements and methodology since the 17 1940 census. I am knowledgeable of all of that activity. 18 I have personally been involved in undercount and 19 correction-like work at the Census Bureau since roughly the 20 1978-79 time frame. 21 And I am -- I have worked considerably ever since 22 that time frame on this matter. So I brought to bear all of 23 this history and knowledge that the Bureau generally and 24 that I personal have developed over these many years. 25 Furthermore, there is a record of undercount 498 1 measurement, and in other countries, such as Canada, 2 Australia, the U.K., and other countries, and I have 3 generally enjoyed exchanges with colleagues in other 4 countries. And so I am aware of their activities and 5 experiences in this matter. 6 Finally, of course, I am very intimately involved 7 or knowledgeable about the 1980 Decennial Census. I took 8 part in that undercount measurement process and brought that 9 experience to bear on 1990. 10 Then there was an extensive research and 11 evaluation effort throughout the decade of the 1980s, 12 planning for the census of 1990. And that entire experience 13 base was brought to bear on the decisions in 1990. 14 And finally, of course, from my perspective as a 15 member of the Special Advisory Panel in 1990, I had access 16 to considerable information on the conduct of the original 17 enumeration, on the conduct of the coverage measurement 18 processes, and on their quality, and I brought all of that 19 current information to bear on this decision as well. 20 Q. At the time that you made the recommendation to 21 the Secretary, based on the information and the background 22 that you have just described, did you believe that 23 reasonable or knowledgeable statisticians could disagree 24 about whether the PES should be used to adjust the census 25 counts? 499 1 A. No, there was a clear-cut matter. It was 2 absolutely clear cut that considerable improvement in the 3 raw counts could be achieved by utilization of the 4 correction or of the corrected data. 5 Q. After the time that you made your recommendation, 6 -- after the time you made your recommendation, did you 7 learn that the Undercount Steering Committee at the Bureau 8 and the Director of the Bureau also made either findings or 9 recommendations concerning whether the PES should be used? 10 A. Yes, I learned of their recommendations. 11 Q. And what were their recommendations that you 12 learned? 13 A. They recommended to the Secretary the use of the 14 corrected counts. 15 Q. After that time, did you read the Secretary's 16 decision? 17 A. Yes, I did. 18 Q. Do you recall about when you read the Secretary's 19 decision? 20 A. On the 15th of July or shortly thereafter of 21 1991. 22 Q. In a word, what was his decision? 23 A. In a word, he decided that the raw counts were 24 the preferred counts in his judgment. 25 Q. Do you agree with the conclusion or his 500 1 reasoning? 2 A. No, of course not. 3 Q. Having read his report, have you changed your 4 mind with respect to your position? 5 A. No, I have not changed my mind. 6 Q. Okay. I'd like you to look at the third tab of 7 your book. That has been marked as PX 9. And what we have 8 here, just for the convenience of the Court and witness, are 9 excerpts of the Secretary's decision that I may be referring 10 to with Dr. Wolter. 11 MR. SOLOMON: If the Court wishes to have the 12 complete copy, I'll find it. 13 Q. But there are some specific places that I'd like 14 to call your attention to. 15 To begin, I would like you to return to page 266. 16 That's 2-66. 17 And the discussion of guideline 6 which runs from 18 266 through 270 I want to call your attention to a couple of 19 statements made by the Secretary. 20 At the bottom of 2-66 it says: 21 "The Census Bureau produced highly technical 22 research on a very tight production schedule, using tools 23 that were on the cutting edge of statistical theory and 24 survey methods." 25 Then over on 2-70, in the conclusion, in the 501 1 second paragraph, he says: 2 "To the Census Bureau's great credit, the 3 statistical tools used to calculate and evaluate the 4 adjusted counts are at the cutting edge of statistical 5 research. But such cutting edge research is not tried and 6 true -- it requires more thorough scrutiny before it can be 7 used to affect the allocation of political representation 8 and Federal funding." 9 Would you comment on those statements. In 10 particular, do you believe them to be fair and accurate? 11 A. No. That's -- they are neither fair nor 12 accurate, and represent a distortion of what I know to be 13 the facts. 14 If I can take the statements one by one, perhaps 15 that might be an efficient way of proceeding. 16 First of all, I don't agree that the Bureau 17 produced work on a production schedule. I wouldn't say it 18 was necessarily a very tight production schedule, it was 19 certainly an appropriate production schedule, certainly of 20 the kind and similar to the production schedule that the 21 Census Bureau always operates under in the context of the 22 decennial census. 23 The Secretary also remarks about tools that were 24 at the cutting edge of statistical theory and survey 25 methods, and I have to tell you that this is an incomplete 502 1 statement and tends to mislead as it's worded. The fact of 2 the matter is -- 3 MR. MILLET: I'm going to object to that, your 4 Honor. Now he's characterizing the Secretary's intent and 5 not discussing the document. 6 THE COURT: Overruled. 7 You may continue. 8 A. (Continuing) Well, the fact of the matter is 9 that the -- that the tools that are used in this process 10 were, to my knowledge, first used as early as the 1940s, 11 perhaps even earlier than that. They were used then by Ed 12 Demming and others. Demming is the patriarch of the U.S. 13 quality movement. These tools were used generally for the 14 first time in connection with the decennial census in 1950. 15 They have been used in connection with U.S. Decennial 16 Censuses periodically ever since. They've been used by 17 other countries ever since. 18 And so, in a certain sense, what was done in 1990 19 is an outgrowth, is an evolutionary outcome of work that was 20 begun in 1940. And to say that somehow suddenly these 21 methods are at the cutting edge I believe is inappropriate. 22 At the same time, enormous improvements and 23 refinements, and so on, were carried out and tested during 24 the 1980s. But even there, the improvements and refinements 25 were tested in real field tests repeatedly during the decade 503 1 of the 1990s. And so by the time of the 1990 Census, the 2 tools were improved, they were an evolutionary outcome of 3 earlier work that was done as early as the 1940s, and in 4 that sense, I see nothing particularly cutting edge about 5 them. 6 Q. Dr. Wolter, let interrupt you for a moment. 7 You said that statistical tests had been carried 8 out in the '90s. I assume you meant the '80s? 9 A. In the 1980s, that's correct. 10 Q. Okay. Thank you. 11 A. The next passage I had from you was on page 2-70? 12 Q. Yes. 13 A. He says, "To the Census Bureau's great credit, 14 the statistical tools" -- and I think the Census Bureau does 15 deserve great credit, I might add. That's one point on 16 which I would agree with the secretary. 17 (Continuing) -- "...are at the cutting edge of 18 statistical research." I have just spoken to that issue. 19 But -- 20 THE COURT: You prefer stone age tools. 21 THE WITNESS: What's that? 22 THE COURT: You prefer stone age tools, sharpened 23 a bit. 24 THE WITNESS: Only if they are better than 25 cutting edge tools. That isn't often the case. 504 1 A. In any event, such cutting edge tools or research 2 is not tried and true. This is nuts. 3 The work that was done, as I said, was developed 4 as a part of an -- as a four- or five-decade long 5 improvement process. It was a part of a process of 6 iterative refinement of testing, problem identification, 7 methodological problem-solving, further testing, further 8 refinement, and so on and so on. 9 And even after the gains of the 1980s were 10 achieved, all of the new improved methods were tested fully 11 in a number of tests, real field tests throughout the 1980s. 12 So in my judgment, the work was tried and true. 13 Q. Have you completed your answer? 14 A. Yes. 15 Q. Okay. Tell the Court what role you personally 16 played in connection with undercount research at the Bureau. 17 Describe it generally, at first, if you would. 18 A. From the time that I was appointed Chief of the 19 Statistical Research Division, and even before then I 20 assumed considerable responsibility for undercount 21 measurement within the Census Bureau, and retained that 22 position and responsibility throughout the decade of the 23 1980s. 24 Q. Did you have personal involvement during the 25 period 1978 prior to 1983? 505 1 A. Yes. I became involved in planning for the 1980 2 Census Coverage Measurement Program. I had staff involved 3 in that, and I was involved and directed their work. 4 Subsequently, I was asked to lead a cross-cutting 5 Census Bureau Task Force charged with evaluating the 1980 6 census and evaluating the quality of the 1980 census 7 undercount estimates. 8 Q. What was the problem that the Bureau was trying 9 to address in conducting the research and the tests that you 10 just generally made reference to? 11 A. Well, the basic problem here is differential 12 undercount. Differential undercount causes some groups in 13 the country to be undercounted more than other groups in the 14 country. 15 Census data is inaccurate as a result. Customers 16 of census data, as a result, are making business decisions 17 in light of flawed census data. 18 It is incumbent on the Census Bureau to try to 19 improve that process, if at all possible. That's the basic 20 problem we were trying to address. 21 Q. Why does that problem arise? 22 A. It arises as a result of the fundamental 23 interaction between census-taking methodology and the 24 American social fabric. 25 As early as 1940, as I said, it was learned that 506 1 there is a differential undercount between blacks, for 2 example, and non-blacks; between men and women; and between 3 various age groups. That differential undercount remarkably 4 has remained constant or very nearly constant in every 5 decennial census since then. In spite of the Census 6 Bureau's good efforts. And they are very good and they are 7 very enormous efforts to try to eliminate the differential 8 undercount by enumeration techniques, that is, by way of the 9 original enumeration. 10 And yet, in spite of all the Bureau's good 11 efforts, and in spite of decades of work, in spite of 12 billions of dollars over the many decades, the differential 13 undercount has remained persistent, constant, almost 14 intractable, at least intractable for the kinds of 15 conventional enumeration methods the Bureau has utilized. 16 Q. Would you give the Court a thumbnail sketch of 17 the design of the PES as used in the 1990 Census, because I 18 would like to use that as a backdrop to ask you to retell 19 the history of the projects at the Bureau leading up to the 20 PES of 1990. 21 A. Okay. PES, of course, stands for Post- 22 Enumeration Survey. And from that name, as you might 23 imagine, this is a sample survey and it's a sample survey 24 that follows the original enumeration. It's am large sample 25 survey of households and people. For each of the 507 1 individuals in this sample survey, a determination is made 2 as to whether the person was counted in the original 3 enumeration or not counted in the original enumeration. 4 That determination of and enumeration status is 5 made by way of what the Bureau calls matching. It matches 6 the person based on name, address, and other characteristics 7 of the individual, to the census returns, and thereby 8 determines whether the person was enumerated or not in that 9 original enumeration. 10 Then the Bureau takes the original enumeration, 11 together with this post-censal survey, and the information 12 on which of those individuals were counted and which were 13 not, and develops an estimate of the size of the total 14 population of the United States; not only nationally, but by 15 various demographic groups and for various geographic areas 16 of the United States. 17 The difference between these PES-based total 18 population estimates and the original enumeration based 19 population estimates is commonly referred to as the 20 decennial census undercount. 21 Q. You mentioned that the process by which the PES 22 arrives at the determination of whether someone was counted 23 or not was done by matching. Do you have personal knowledge 24 of the matching rules that were used as part of the 1990 25 census? 508 1 A. Yes, generally. 2 Q. Can you describe to the Court how the matching 3 went forward. 4 A. Well, the matching went forward in two or three 5 basic layers. 6 First of all, there was developed during the 7 1980s what's called a computer expert system that implements 8 matching automatically. This is matching based on the 9 names, the addresses, the race, the marital status, age, 10 et cetera, of the individuals enumerated in the post-census 11 survey. 12 The automated matcher, if I can use that term, 13 matches about 75 percent of the persons counted in the post- 14 censal survey. 15 All other individuals are referred to human 16 beings for matching, and there is a multi-stage human 17 matching process that starts with a staff of well-trained 18 clerks in census processing offices who compare the 19 post-censal survey people versus the original enumeration 20 returns to determine who was -- who is in the original 21 enumeration and who is not. 22 Then the results of that matching are referred to 23 further teams of better clerks, better in the sense that 24 they exhibit greater skill, better in the sense that they 25 are trained more, better in the sense that they are under 509 1 tighter control, administratively. 2 And really to cut a long story short, there are 3 several additional layers of human matching whereby better 4 and better clerks and technical personnel are applied to 5 this matching operation. There are further field follow-ups 6 to gather additional information, and then there is further 7 layering of additional human matching effort which 8 ultimately culminates in determining which of the PES people 9 are counted and which are not counted in the original 10 enumeration. 11 Q. Did the 1990 PES use imputation? 12 A. Yes, it did. 13 Q. Describe briefly for the Court what imputation 14 is? 15 A. In all sample surveys of households and people, 16 there is inevitably some missing data. 17 There is inevitably some segment of the 18 population that doesn't cooperate or which supplies faulty 19 data. 20 This isn't true just in the United States. It's 21 true around the world. And it isn't just true in PES-like 22 surveys, but it's true in every sample survey known to me. 23 Inevitably, there's always some -- some missing data. And 24 always, survey statisticians must decide how to handle this 25 missing data. There's really no issue. It's not a question 510 1 of should we handle the missing data or shouldn't we; it 2 always must be handled in any sample survey. It's often 3 handled by estimating what the missing data would have been 4 had it been observed. Statisticians and others in the 5 survey profession often refer to this process as imputation. 6 And the Census Bureau imputes missing data in the 7 original enumeration as well as in the PES as well as in 8 every other census and survey that the Census Bureau 9 conducts. 10 Q. Do you have an opinion whether with respect to 11 the 1990 PES there was a relatively large, or relatively 12 small amount of imputation that was necessary by reason of 13 missing data? 14 A. There was an unprecedentedly small level of 15 missing data in the 1990 Post-Enumeration Survey requiring 16 the imputation process, as I have just described it. 17 Q. Would you just briefly finish off the description 18 of how the PES gets to the adjusted numbers, and now I 19 understand we've had matching and we've had any imputation 20 that might be necessary. 21 A. Okay. Well, this can be a long story or a short 22 story. 23 Q. Take the short story. 24 A. Take the short story. Okay. 25 Q. Leaving nothing out, sacrificing no accuracy when 511 1 you are speaking, what he (indicating) used to tell me. 2 A. Okay. In broad general terms, we at this point 3 in the explanation, we have the PES data as reported, and 4 then we have the imputed data in the case where reported 5 data was missing. And in that sense our data is now 6 complete, it's either reported or it's imputed, but we can 7 now say that the data set is complete. 8 We now have the original enumeration and we now 9 have the complete PES data, and the Census Bureau then 10 initiates its estimation process. The Bureau estimates two 11 proportions from these various data. First of all, it 12 estimates the proportion of people enumerated in the 13 original enumeration correctly, and, of course, the 14 complimentary proportion is the proportion of the original 15 enumeration that was, in fact, counted erroneously. 16 I don't know if you are familiar with the term 17 "erroneous enumerations" or not, but some individuals 18 regrettably are counted erroneously because they were born 19 after census day but were counted nonetheless, or they died 20 before census day and were counted nonetheless. Or they 21 were counted in the wrong geographic location, or census 22 returns were duplicated or triplicated for an individual, 23 and so on and so on. 24 So that's the first piece of the PES estimation 25 process, is to estimate the proportion of the original 512 1 census enumeration that was counted correctly versus 2 erroneously. 3 Then the Census Bureau also estimates a second 4 proportion. This is the proportion of the U.S. population 5 that was counted in the original enumeration. 6 Then the original enumeration itself, together 7 with these two proportions, as I have described them, allow 8 the Census Bureau to estimate the total size of the 9 population. As I said, not only at the U.S. level, but also 10 for various demographic segments of the United States, and 11 also for various geographic areas, like states and cities 12 within the U.S. 13 Once these direct estimates of total population 14 are produced, then the Census Bureau goes through another 15 refinement in the estimation process, and removes what's 16 called sampling error, or removes as much sampling error as 17 it can from these direct estimates the size of the 18 population. 19 At that point, the Bureau has its best estimates, 20 the size of the population, at, shall we say, aggregate 21 levels of geography, and for aggregate demographic groups 22 within the U.S., and it then has the means of carrying the 23 missed people down to lower levels of geography, and 24 allocating those missed people to the various small areas -- 25 blocks, counties, enumeration districts, and the like -- 513 1 within the country. 2 Q. Did the Bureau use something called 3 stratification in calculating the undercount estimates as 4 part of the 1990 PES? 5 A. Yes, they do. 6 Q. Just describe generally what the stratification 7 scheme was? 8 A. Are you referring to the post-stratification 9 scheme? 10 Q. Yes. 11 A. Well, that's a fancy word that statisticians use. 12 All it means, that we are going to chop up the country into 13 mutually exclusive and exhaustive pieces, and these pieces 14 are called poststrata. 15 In this particular case, there were 1392 pieces 16 that the overall United States was divided into. These 17 poststrata or these pieces were in part defined by 18 geography. And in part, they were defined by demographic 19 areas, such as age, sex, race, ethnicity, housing tenure, 20 type of place. Whether it's a central city or not. 21 And so you wind up with a piece of the country 22 called or referring to, for example, all black males in New 23 York City that rent their housing unit and that are age, 24 say, 20 to 29. That is an example of one of the 1392 25 pieces. 514 1 And together these pieces or poststrata 2 completely span the entire resident population of the United 3 States. 4 Q. You spend your life doing survey sampling or 5 inspecting survey sampling. Is stratification common in -- 6 A. Stratification of this kind is pervasive 7 throughout all the work at the Census Bureau. 8 Q. Why did the Bureau stratify the 1990 PES? 9 A. It does so to reduce sampling variability, which 10 is one of the limitations on any sample survey, and it also 11 does so to produce the very best possible corrected census 12 product, a product that meets the needs of census customers. 13 Q. You made reference in your description before to 14 the fact that the Bureau after arriving at the estimate 15 removed sampling variability. Is that also called 16 smoothing? 17 A. Yes. Smoothing, and a variety of other names are 18 used in technical publications to refer to this process. 19 But cutting through all that jargon, it's really just a 20 matter of attempting to strip out this one or a portion of 21 this one limitation on survey results called sampling error. 22 Q. Is that a common technique in survey sampling? 23 A. Is what a common technique? 24 Q. Smoothing, or the reduction of sampling 25 variability? 515 1 A. It's used in a number of surveys, yes, but it 2 isn't as pervasive as post-stratification itself is, but 3 nonetheless, it's been used time and again. 4 MR. SOLOMON: I don't know if your Honor wants to 5 take an afternoon break. I'm happy to move on at this time. 6 THE COURT: All right. Let's take a 15-minute 7 break. 8 (Recess) 9 (Continued on next page) 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 THE COURT: All right, you may resume. 516 1 MR. SOLOMON: Thank you, your Honor. 2 BY MR. SOLOMON: 3 Q. Dr. Wolter, will you please describe to the court 4 in overview how the Census Bureau approached the problem of 5 the differential undercount and the design of the PES? 6 A. In the 1980s or before? 7 Q. Yes, in the 1980s. 8 A. In the 1980s. 9 Surely. 10 Starting in roughly in 1984, 85 time frame, there 11 was, first of all, a period wherein I gained support from 12 the very highest executive levels within the agency that 13 this problem, the problem of differential undercount, would 14 be one of the two or three preeminent objectives for the 15 agency in the decade of the 1980s. That agreement from the 16 top was reached in roughly 1984-85 time frame. 17 Subsequent to that, a staff was developed, 18 budgets were acquired and real work got underway. 19 There was a period of new methods development, 20 then there was a period of testing, then there was a period 21 of further refinement of methods based on the testing, and 22 it went back and forth between testing and methodological 23 development. 24 And eventually decision processes were put in 25 place and finally the 1990 enumeration and PES itself took 517 1 place. 2 Q. As part of that effort, were special or 3 particular organizations within the Bureau formed to address 4 the question of a differential undercount and the design of 5 the 1990 PES? 6 A. Yes. There were two principal organizations 7 formed within the Census Bureau to address this issue which 8 I have now said is one of the two or three preeminent 9 executive objectives for the decade. 10 One of the organizational units came to be known 11 as the undercount research staff. This was a staff of 12 professional statisticians, demographers and the like that 13 reported to me and the statistical research division. 14 I proposed creation for this staff, I hired this 15 staff, I trained this staff, I directed the staff and that 16 went on until the time that I left the Census Bureau. 17 The second principal organization within the 18 Census Bureau came to be known as the undercount steering 19 committee, and this was a kind of board of directors, if you 20 will, that oversaw policy development and the work of the 21 undercount research staff. 22 The undercount steering committee was composed 23 of, oh, roughly half a dozen top census executives and took 24 part in every critical decision throughout the decade, from 25 1984 until, I suppose, current points in time. 518 1 Q. Who headed the undercount research staff directly 2 under you? 3 A. A gentleman who I hired by the name of Howard 4 Hogan, Dr. Howard Hogan. 5 Q. Is Dr. Hogan still at the Bureau? 6 A. Yes, he is, to my knowledge. 7 Q. What's role, if any, did you play as a conduit or 8 liaison in between the work of the undercount research staff 9 and the undercount steering committee? 10 A. Well, again, the undercount steering committee, 11 as I said, was and is like a board of directors setting 12 policy, reviewing developments and the like. I have served 13 on that board, on that committee. 14 At the same time, staff work and coordination 15 work, detailed work, planning work, implementation work is 16 being accomplished by the undercount research staff which 17 reported administratively to me. 18 So obviously I was in the middle. I was the 19 conduit. I was reporting findings from the staff to the 20 board, I was communicating policy from the board back to the 21 staff, and there was a continual feedback loop. 22 Q. Were there particular areas within the undercount 23 research staff that got a particular attention or focus? 24 A. Every aspect of this issue got exceptional focus 25 throughout the decade. Some of the areas that we focused on 519 1 especially in early years were activities like the matching 2 process that I mentioned before, such as limiting the amount 3 of missing data which I mentioned before, and such as the 4 process of removing sampling error, what you called 5 smoothing earlier, all of that, among many other areas, 6 received considerable attention by this staff and by various 7 other staffs that the undercount research staff coordinated. 8 Q. Were there other areas of the Census Bureau that 9 were involved in the research and design efforts besides the 10 undercount research staff and the undercount steering 11 committee? 12 A. Yes. There were many other offices within the 13 Bureau that participated in this overall enterprise. 14 Let me start with the population division. The 15 population division is a group of demographers and other 16 social scientists that study the population of the United 17 States. They have been instrumental in producing undercount 18 estimates for every census since 1940. Their method of 19 undercount measurement is known as the method of demographic 20 analysis. They participated with us. 21 The field division within the Census Bureau 22 participated with us. That is the group charged with 23 responsibility for actually carrying out the interviewing 24 process on all surveys and censuses within the agency, PES 25 and decennial census included. 520 1 We participated with an organizational group 2 known as the decennial operational division that has 3 responsibility for all of the computer work and all of the 4 office processing associated with a large enterprise like 5 the decennial census and PES. 6 There was a crosscutting funding group within the 7 Census Bureau known as the decennial planning division, and 8 we certainly worked extensively with that group. 9 We worked extensively with colleagues in the 10 statistical methods division, which is a division charged 11 with operational responsibility for surveys of households 12 and people within the Census Bureau. And, of course, 13 various other executives, too. This was an activity that 14 drew in many people from almost every corner of the Census 15 Bureau. 16 Q. What level of resources did the Bureau as a whole 17 commit to the project of ameliorating the differential 18 undercounting by sample survey? 19 A. An extraordinary high level of effort and 20 commitment was put into this enterprise. 21 As I said, census executives, early in the 22 decade, decided that this was to be the primary focus or one 23 of the primary focuses for the 1990 census, reduce the 24 differential undercount, and it put people, it put staff and 25 it put commitment behind this. 521 1 I would estimate on the order of perhaps a 2 hundred person years of headquarter's time was devoted to 3 this activity throughout the period from 1985 through 1990, 4 and at least $10 million effort was put behind this. And 5 that's not counting the time of all the interviewers and 6 matching personnel and other people that took part in this 7 overall process. 8 This was a large endeavor that occupied a major 9 initiative by this agency and by many other groups that 10 participated with this agency. 11 Q. Did outside professional organizations or 12 associations play any role in connection with the study of 13 the adjustment issue during the 1980s? 14 A. Outside organizations played a key role by design 15 and that was only right and proper. 16 I'm thinking of such organizations as the 17 Standing Advisory Committees from the American Statistical 18 Association, the American Economics Association, the 19 American Marketing Association, the Pop Association, I'm 20 thinking of a special advisory panel of, a blue ribbon panel 21 of experts appointed by the National Academy of Sciences to 22 review this matter and to offer advice and counsel to the 23 Census Bureau; I'm thinking of colleagues at Stat Canada in 24 Ottawa, Canada and other organizations, too, that 25 participated with us either in a formal or informal role 522 1 throughout the entire decade. 2 Q. I think you mentioned that it was important. 3 Why was it important to have outside advisors or 4 experts reviewing this work or commenting on it or 5 communicating with the Bureau? 6 A. As a matter of science, as a matter of corporate 7 culture, as a matter of what's right and proper. 8 The Census Bureau makes every effort on all 9 critical matters to expose its methods, its procedures, its 10 theories to open public comment. 11 We wanted the very best advice, we wanted the 12 very best counsel, we wanted to use the very best possible 13 procedures and methodologies to solve this very critical 14 important national problem of decennial census differential 15 undercount. 16 We felt it was important that we establish some 17 kind of a consensus of support among professional groups and 18 other communities that used decennial census data. We felt 19 that was important as we move into the 1990 census period 20 and as we make or made the very important decisions that 21 were, in fact, made regarding that census. 22 Q. Dr. Wolter, John Rolf will be testifying here on 23 behalf of the plaintiffs in this case. 24 Was he on the panel of the National Academy of 25 Sciences, the blue ribbon panel that you made reference to? 523 1 A. Yes, he was. 2 Q. And Dr. Feinberg, Steve Feinberg will also be 3 appearing on behalf of the plaintiffs in this case. 4 Do you know whether he had a position at the 5 National Academy of Sciences? 6 A. I, I, I have forgotten the exact years of his 7 tenure, but for many years Steve was the chair of a parent 8 committee within the National Academy of Sciences. 9 In other words, there was a parent committee 10 which is commonly known as the committee on national 11 statistics. For a number of years early in the decade Steve 12 chaired that parent committee. It was that parent committee 13 that formed and organized the blue ribbon panel on decennial 14 censuses that I referred to earlier. 15 Q. How did the Bureau communicate with these outside 16 agencies the thoughts and design concepts and the testing 17 procedures that it was coming up with? 18 A. It communicated by a variety of means. 19 There were frequent meetings with these various 20 organizations where verbal or oral presentations took place, 21 where testimony was given, where there was oral exchange 22 between the committee and the Census Bureau. Often written 23 documents were presented to these various committees by the 24 Census Bureau. 25 Of course, there was widespread one-on-one 524 1 interaction, not only with the individual members of these 2 committees, but with the professional community at large. 3 And then finally, professional organizations like 4 the American Statistical Association, like the Population 5 Association of America, I presume like the American Bar 6 Association, hold periodic annual national meetings and 7 regional meetings and the Census Bureau staff presented 8 dozens of papers and other kinds of presentations at these 9 various national and regional meetings, and that drew in a 10 variety of participants from these organizations from around 11 the country and, indeed, other countries. 12 Q. As a result of the Bureau's internal analysis and 13 its communications with the outside agencies, the 14 presentations of papers that you just made reference to and 15 the one-on-one communications, do you believe that over the 16 course of the decade of the 1980s, consensus was formed 17 concerning whether or not the PES could be used to 18 ameliorate the problem of differential undercounting in the 19 decennial census? 20 MR. MILLET: I object to that. It calls for 21 speculation. It is outside the area of hisexpertise. 22 MR. SOLOMON: The the witness testified that the 23 purpose for communicating with the outside agencies and 24 experts was to form a consensus. 25 THE COURT: I will permit the question. The 525 1 objection is overruled. 2 A. One of my most important responsibilities at the 3 Census Bureau was to interact with these outside groups, was 4 to stimulate, if you will, the exchange between the Bureau 5 and the outside community on this very issue. That was my 6 responsibility, among others. 7 I stimulated that exchange. I believe that 8 consensus was reached on this matter, that PES based 9 procedures could ameliorate the differential undercount. 10 I'm not saying that there was unanimity of 11 opinion. There were a few views that I would regard as 12 extreme, quite frankly. 13 Most of the advisors, most of the panels, the 14 committees, the groups that stayed with this issue for a 15 period of time, became familiar with it, took time to learn 16 the issues and offer serious advice and counsel were part of 17 the consensus that I just described. That was quite clear. 18 Q. Why if you are talking about adjusting the 1990 19 census did work begin I think you mentioned in the 1982, 83, 20 84 time frame? 21 A. There are probably two dimensions to that issue. 22 First of all, in government there are enormous 23 lead times. The government just doesn't move suddenly, it 24 moves. 25 THE COURT: The record won't reflect the meaning, 526 1 but I have it. 2 A. It takes time to acquire a budget, it takes time 3 to receive approval to acquire computer equipment, it takes 4 time to acquire temporary space and staff needed to execute 5 a decennial census, it takes years to put in place all of 6 the apparatus needed to handle an operation this large. 7 That's one dimension. 8 But an even more important dimension of this is 9 the fact that the Census Bureau, again, by its scientific 10 orientation, by its public service orientation, but its 11 corporate culture seeks to constantly improve, refine, test, 12 further improve, refine, further test almost everything the 13 Bureau does. That's been a pervasive characteristic of the 14 agency since the 1940s. And it was that corporate culture 15 that led us to want to begin this work as early as possible 16 in the 1980s, starting with the experience in 1980 itself, 17 learning from that experience, learning what we could about 18 how to improve that experience, putting in place 19 improvements, improving, testing, improving and further 20 testing. 21 All of that takes years and that's why we started 22 early in the 1980s. 23 Q. In the 1984 time period, after the undercount 24 steering committee got together and formed, did it issue a 25 general plan to the relevant scientific community letting 527 1 them know of this plan of attack? 2 A. Yes. The undercount steering committee, the 3 board, if you will, as one of its very earliest initiatives 4 made extensive efforts to set policy for the entire 5 undercount measurement enterprise throughout the 1980s. 6 This policy was drafted and it was presented not only to 7 professional communities, but to other political and other 8 interested parties who have an interest or a stake in the 9 decennial census. 10 Q. Let me ask you, please, to turn to the next tab 11 in your notebook, tab 4, which we have marked for 12 identification PX 541. 13 MR. SOLOMON: For the record, it is an October 14 1984 document entitled, "Requisite planning and research 15 relating to a decision on census adjustment for 1990," by 16 KIRK M. Wolter, Friday, October 12, 1984. 17 Q. Can you identify PX 541? 18 A. Yes. It is an article I wrote setting forth the 19 policy that was developed by the undercount steering 20 committee. 21 MR. SOLOMON: We would offer Plaintiff's Exhibit 22 541, your Honor. 23 MR. MILLET: No objection. 24 THE COURT: 541 is received. 25 (Plaintiff's Exhibit 541 marked for 528 1 identification was received in evidence.) 2 BY MR. SOLOMON: 3 Q. This document outlines what is said on page 2 of 4 the document to be five major planning goals. 5 Can you go through those for the court and 6 identify them and tell the court whether those planning 7 goals were achieved? 8 A. Yes, sir. 9 MR. MILLET: Can you identify the page again? 10 MR. SOLOMON: I am on page 2 of the document that 11 bears the production number 16211. 12 MR. MILLET: Thank you. 13 A. There is a paragraph headed number 1, "Complete 14 the 1980 coverage evaluation studies." 15 This was the first of the goals. 16 There were two major coverage evaluation studies 17 of the 1980 decennial census. One was a study based on the 18 method of demographic analysis. I mentioned that in passing 19 earlier. And secondly there was a study based on a PES 20 conducted in 1980. That was also known in 1980 as the 21 post-enumeration program. 22 Both of those undercount measurement activities 23 were, in fact, completed, and by the 1985-1986 time frame 24 the Bureau published a formal report summarizing the 25 information and findings about the coverage of the 1980 529 1 census. That goal was, therefore, satisfied. 2 I'm now on page 3, top of the page, goal 2, which 3 reads, "Determine criteria for deciding when census adjusted 4 estimates are more accurate than basic census estimates and 5 whether adjustment would have any practical effect." 6 That goal was achieved by the, by the late 86-87 7 time frame. It was at that point in time that the Bureau 8 issued a first draft of its criteria, that is, the criteria, 9 the scientific criteria and other criteria that it would 10 utilize in deciding whether to correct or not to correct the 11 1990 decennial census undercount. 12 Subsequently, that draft was pushed to a back 13 burner, quite frankly, in lieu of or as a result of the 14 Commerce Department's decisions in October of 1980 not to 15 implement a PES and correction process in 1990. 16 Q. That was in October of 1987? 17 A. Yes, sir. 18 Later, of course, when your Honor reinstated the 19 PES in the stipulation and order of 1989, the Commerce 20 Department was under obligation to resurrect these criteria 21 which, indeed, it did, and which, indeed, it published as 22 its guidelines for census correction, and I would gather or 23 guess that that date was somewhere in the 89-90 time frame. 24 MR. MILLET: Objection and move the witness' 25 characterization of the Commerce Department's obligations 530 1 under the stipulation be striken. 2 THE COURT: The document speaks for itself, so 3 avoid characterizing the obligations contained in the 4 document. Okay? 5 THE WITNESS: Sorry. 6 Q. Please look at the third of the goals on that 7 page. It says, "Develop direct measurement technique to be 8 used for measuring undercount for 1990." 9 Was that goal achieved by the Bureau? 10 A. Yes. The Bureau considered an array of 11 undercount measurement methods, tested each one of them 12 thoroughly, determined which were good, which weren't so 13 good, narrowed the field, tested the better ones further and 14 ultimately narrowed down to one principal method that would 15 be used as the preeminent measure survey based measurement 16 technique for undercount in 1990. That was the so-called 17 PES or post-enumeration survey method. That was goal 3. 18 Q. Goal 4 is on page 6. 19 A. Okay. Goal 4 is, and I'm quoting again, "Decide 20 on adjustment methodology for 1990 if decision is made to 21 adjust." 22 And this issue has a decade long history. 23 Early in the decade, and, again, I'm thinking of 24 the 1984 time frame, I assigned this issue to one of the 25 staffs within the statistical research division, and that 531 1 staff remained in place throughout the decade and its 2 principal responsibility was this very goal; it was to, it 3 was to develop methods for correction or adjustment if the 4 decision was made to adjust. Its goal was to test those 5 methods, its goal was to evaluate those methods under a 6 variety of criteria and to recommend to census executives 7 the very best possible methodology for use in 1990. That 8 goal was certainly achieved. 9 Q. The last goal. 10 A. The last goal on page 9, this is number 5, and, 11 again, I quote, "Decision on adjustment." 12 The decision process that we envisioned was that 13 the decision about whether to actually correct or not to 14 correct the decennial census was to be data driven. In 15 other words, this decision wasn't to be based on assertion, 16 this wasn't to be based on speculation, this wasn't to be 17 based on what somebody thought, this was a decision that was 18 to be data driven based on real experiences with the 1980 -- 19 excuse me -- 1990 enumeration and 1990 coverage measurement 20 process, and the decision was to be made by the Census 21 Bureau's director in late 1990 or early 1991 in light of the 22 data and in light of the decision criteria that I discussed 23 earlier under goal number 2. 24 Q. I notice, actually, that you have in item 5, 25 decision 5, that decision would be made by the director of 532 1 the Census Bureau. 2 In 1984 was it your assumption that this data 3 driven decision, in your words, would be made by the 4 director of the Bureau of the Census and not the Secretary 5 of Commerce? 6 MR. MILLET: Objection, relevance. Plaintiffs 7 have stipulated this decision was to be made by the 8 Secretary in the stipulation and order. 9 MR. SOLOMON: Your Honor, the defendants, in the 10 stipulation and order, said what they were going to do and 11 the plaintiffs said they are going to do at least that much, 12 and in the last paragraph of your order it says we are 13 reserving all of our rights. 14 Here I simply want the witness to tell me -- Mr. 15 Millet and I can argue about that interesting question some 16 other time. Here I would like to know what the witness' 17 assumption was, because it obviously bears on whether the 18 Census Bureau was proper in making the recommendations that 19 it made and whether the Secretary of Commerce acted 20 improperly in overruling them. 21 THE COURT: For those limited two purposes, I 22 will overrule the objection and admit the testimony. 23 A. It was certainly my view, the view of the 24 undercount steering committee that the decision would be 25 made by the census director. This decision was made by the 533 1 census director in 1980. 2 In all census history that I'm aware of decisions 3 about the conduct of decennial censuses was made at the 4 Census Bureau by the Census Bureau staff and it was my 5 assumption, it was our assumption at that point in time that 6 once again the census director would be making this decision 7 among many others. 8 Q. In light of the fact that you believed the 9 decision was going to be a data driven one, were you aware 10 of anyone at the Department of Commerce level who had the 11 requisite knowledge to make a data driven decision? 12 MR. MILLET: I will object. That calls for 13 speculation. 14 THE COURT: If you know you may answer. 15 A. There was, to my knowledge, there was no one at 16 the Commerce Department with, with, with adequate knowledge 17 of the theories, methods and operations of survey sampling 18 and of evaluation and measurement studies. 19 That isn't to say there weren't individuals who 20 were knowledgeable about other aspects of statistical 21 methodology, but not this particular methodology, no. 22 Q. Did the undercount steering committee set for 23 itself a schedule by which it would attack each of the five 24 planning items that we just made reference to and achieve 25 them? 534 1 A. Yes. A schedule of broad milestones was 2 established covering all of the work to be done on this 3 matter from the time of this document through the 1990 4 period itself. 5 Q. Did the Bureau call to the attention of the 6 outside experts and the relevant scientific community the 7 schedule so that anyone who wanted to participate in this 8 decision would? 9 A. Absolutely. The schedule, the milestones were 10 widely brought to the attention of all interested 11 communities. 12 Q. Would you please turn to the next tab in the 13 book, tab 5, where we have marked for identification PX 545 14 of October 1986, a document entitled, "Introduction to the 15 decennial census undercount and correction." 16 Can you identify that document? 17 A. Yes. This is another document that I wrote and 18 presented to the Census Bureau standing advisory committees. 19 That was in October 1986 time frame. 20 Q. In particular, looking at page 5, is this one of 21 the places where the Bureau identified through the relevant 22 statistical community what the schedule was of the key 23 milestones that it was following? 24 A. Well, yes. In section 3.3 on pages 5 and 6 there 25 is a discussion of decision points. So the answer is yes, 535 1 this is one of the documents. 2 Q. What were the major decision points, for the 3 court's benefit? 4 A. Well, aside from narrow decisions regarding 5 detail and aside from decisions of implimentation of field 6 tests and the like, there were two major decision points 7 concerning this matter: 8 One was to be a decision in the spring of 1987, 9 the other a decision to be made in late 1990 itself or early 10 1991. 11 First the 87 decision point. That was to be a 12 key decision as to whether the Bureau felt confident that 13 the PES correction process was likely to be successful if 14 implemented in 1990. 15 Further, this was also a decision as to whether 16 to seek funding for a PES in 1990. 17 These two aspects of that decision, of course, go 18 hand and hand. If you don't think you are going to be 19 successful, then you don't ask for the money. 20 So the Bureau was to make, in that time frame, a 21 decision about feasibility of the undercount measurement 22 program and the correction program and along with that a 23 decision as to whether to seek funding to support the 24 undercount measurement program. 25 Q. Let me turn your attention to the first paragraph 536 1 of this paper, in particular a sentence that reads, "Just as 2 we could not conduct the 1990 enumeration without widespread 3 support for our plans and goals, neither can we contemplate 4 adjustment without establishing a consensus of support for 5 our techniques and standards." 6 Is this one of the places where you advised the 7 outside scientific world that what the Bureau was seeking to 8 obtain was a consensus of the relevant scientific opinion? 9 A. Yes, sir. But, I mean, moreover, all of the 10 presentations that took place in these meetings themselves 11 formed the basis for establishing a consensus of support or 12 indicated our general intent to seek such support. 13 MR. SOLOMON: Plaintiffs offer Exhibit 545. 14 MR. MILLET: No objection, your Honor. 15 THE COURT: Exhibit 545 is admitted. 16 (Plaintiff's Exhibit 545 marked for 17 identification was received in evidence.) 18 BY MR. SOLOMON: 19 Q. Describe for the court how the Bureau received 20 feedback from the outside agencies, outside expert 21 professional panels that it was communicating with. 22 A. There was a variety of feedback. There was 23 certainly one-on-one feedback from individual members of the 24 committees that I have described. 25 There was one-on-one feedback from other 537 1 scientists around the country and in other countries. 2 There was feedback from the committees as a 3 committee. 4 There was feedback in the form of oral and 5 written recommendations from the American Statistical 6 Association, from the POP, that is the population committee, 7 from the National Academy of Sciences panel, and so forth. 8 Of course, there were exchanges of views and 9 feedback from the professional community at large by way of 10 the national and regional meetings that I described earlier 11 and by way of our exchange with our sister agency in Ottawa, 12 Stat Canada. 13 Q. You make mention in your last answer to 14 particular recommendation, written recommendations by the 15 outside agencies. 16 Would you turn, please, to the next tab in your 17 notebook, tab 6, where we have marked for identification PX 18 550, which is entitled, "Report of the ASA technical panel 19 on the census undercount." 20 Can you identify that document? 21 A. Yes, sir. This document is known to me. 22 This is, this is a final report produced by yet 23 another panel that I haven't described yet. 24 There are myriad panels and committees, there was 25 really a massive, really massive oversight and examination 538 1 of this issue and this is yet another one of the panels. 2 This panel existed in the roughly 82-83-84 time 3 frame, I don't recall exactly when. It was constituted to 4 review what was done in 1980. It was constituted to review 5 the 1980 census, the 1980 undercount measurement program, to 6 comment on the quality of a potential correction at that 7 point in time, and they issued these final recommendations 8 to us, again, in the 82-83-84 time frame, whenever. 9 Q. Did the Bureau read and consider if not adopt the 10 recommendations of the ASA technical panel? 11 MR. MILLET: That is a compound question, your 12 Honor. 13 Q. Did the Bureau read and review the 14 recommendations of the ASA technical panel? 15 A. Well, many of us read it and reviewed it and 16 certainly relevant executives did, yes. 17 MR. SOLOMON: We would offer PX 550. 18 MR. MILLET: Your Honor, I'm sure this is 19 probably an oversight on Mr. Solomon's part. My PX 550 is 20 two documents, it's not merely one. 21 MR. SOLOMON: Let me see it. 22 MR. MILLET: You may. 23 (Pause) 24 MR. MILLET: My objection is that both parts of 25 these have not been authenticated, first of all, and 539 1 secondly, at least to the part that has been authenticated 2 and I would object to that first half that is hearsay. 3 THE COURT: The first half is the part you just 4 discussed with the witness, right? 5 MR. SOLOMON: I thought, in fact, I was 6 discussing the entire document, but I will lay the 7 foundation for the second half. 8 THE COURT: Would you, please? 9 MR. SOLOMON: Sure. 10 BY MR. SOLOMON: 11 Q. Was it the practice from time to time at the 12 Bureau to respond in writing to the recommendations made by 13 the outside expert agencies? 14 A. Yes. We often responded in writing. There may 15 have been cases where we didn't, but certainly as a matter 16 of course we always responded in writing formally to the 17 standing advisory committees. 18 Q. Look, please, at this document further in. I 19 believe it is published in the same place in the journal of 20 the American Statistical Association. 21 I cannot tell what page it is on, but it begins 22 with a comment from the Census Bureau by Barbara A. Bailar. 23 Do you see that? 24 A. Yes. 25 Q. Can you identify that? 540 1 A. Sure. 2 Barbara Bailar way my supervisor at the Census 3 Bureau. She wrote this article. 4 In this article she comments on the final report 5 of the ASA technical panel, that is, the report marked 550 6 that we just looked at. 7 THE COURT: To whom did she write this? 8 THE WITNESS: Well, I, I suppose to the technical 9 panel itself and to all interested parties that may read the 10 technical panel's report and may wish to know what the 11 Bureau's views are regarding the technical panel's report. 12 THE COURT: Mr. Millet, you don't object, or do 13 you, to the first half? 14 MR. MILLET: I don't object to the authentication 15 of the first half, your Honor. I don't think we still have 16 sufficient authentication for the second half, and I do 17 maintain my hearsay objection to both. 18 THE COURT: The hearsay problem stops me a bit, 19 Mr. Solomon. Can you help me there? 20 MR. SOLOMON: Sure. 21 Of course, with respect to the authentication, it 22 is a periodicaL and it is self-authenticating. 23 MR. MILLET: I don't think it has been 24 established that it is a -- 25 THE COURT: Let's leave that on the back burner. 541 1 Talk about hearsay. 2 MR. SOLOMON: Your Honor, this document is being 3 offered to show the growing consensus that occurred within 4 the relevant statistical community during the 1980s and it 5 would be sufficient if, as I've already established through 6 this witness, that the Bureau, in fact, heard and read and 7 took account of, whether or not it agreed with, the 8 particular recommendations that were made. The witness has 9 already agreed to that. 10 Second, this is, in fact, part of a plan that 11 leads up to the 1990 census as my next question will, in 12 fact, demonstrate. 13 MR. MILLET: I hear Mr. Solomon saying he is 14 offering it for the truth of the matter contained therein 15 and, therefore, I maintain my objection. 16 THE COURT: I hear him saying it's a nonhearsay 17 purpose, state of mind, a relevant effect on the state of 18 mind on anybody that would read it. 19 MR. SOLOMON: That's correct, your Honor. 20 THE COURT: All right. You said it better. 21 THE COURT: For that purpose, I shall admit it, 22 both documents, under the one rubric Plaintiff's Exhibit 23 550. 24 The authentication problem, I think, is minor. 25 I take this is some kind of -- something is 542 1 blotted out on the bottom, but it looks to be some sort of 2 periodical, the American-something. 3 MR. SOLOMON: It is. I may have misspoke. Mine 4 is legible enough and we will get a more legible copy for 5 the court. It stays the American Statistician, November 6 1984. 7 THE COURT: Is that a learned publication of some 8 sort? 9 THE WITNESS: It's one of the principal 10 publications of the American Statistical Association. 11 THE COURT: I'm satisfied. I will admit it. 12 (Plaintiff's Exhibit 550 marked for 13 identification was received in evidence.) 14 BY MR. SOLOMON: 15 Q. Dr. Wolter, look, please, at recommendation 16 number 12 of the ASA technical panel, which says, the second 17 to the last page of the first of the parts, "That the Bureau 18 of the Census sponsor an outside technical advisory group on 19 undercount estimation and related problems." 20 A. Yes, sir, I see that. 21 Q. Did the Bureau follow that recommendation? 22 A. Yes, it did. 23 Q. What was the outside technical advisory group 24 that was impaneled? 25 A. I described earlier a committee of the National 543 1 Academy of Sciences. 2 We sought out the National Academy, we asked for 3 their help. Under their ausppices, a panel was constituted 4 and reviewed all census methodology, including especially 5 the issue of undercounts and miscount. 6 Q. As compared to the time that was spent by other 7 outside advisory groups, can you describe for the court what 8 kind of time the NAS panel put in the project of studying 9 the census undercount? 10 MR. MILLET: No foundation for that. 11 THE COURT: If you know, you may answer. If you 12 don't, don't guess. 13 A. I can't speak to any one individual, but I can 14 speak to time frame. 15 The panel existed over a several year period. I 16 don't remember exactly when their work got underway, but it 17 was probably in late 84, early 85, and their work continued 18 throughout the next several years. 19 I recall meeting with them just before I left the 20 Census Bureau in May of 1988. So it was a, it was a 21 prolonged period of time where, I don't know for sure, but 22 my impression was that most panel members spent a 23 considerable amount of time immersed in this issue, learning 24 the details of this issue, asking questions about this 25 issue, commenting on this issue and so forth. 544 1 Q. Please turn to the next tab in the book we have 2 marked as PX 2, a document entitled, "The bicentennial 3 census, new directions for methodology in 1990." 4 You have here just an excerpt of a very long book 5 I supplied you with and the court obviously has copies of 6 the entire book. 7 Can you identify that? 8 A. Yes, I can. It's probably the most important or 9 one of the most important work products of the National 10 Academy panel. It was a book and a kind of interim report 11 that they issued regarding 1990 decennial census 12 methodology. I believe this was published late in 85 or 13 early 86, somewhere in there. 14 Q. Does the book contain recommendations to the 15 Bureau? 16 A. Yes, it does. 17 Q. Do you know whether the recommendations of the 18 panel of the NAS, the National Academy of Science panel 19 reflected in the book, reflect the input made to the panel 20 by Census Bureau? 21 A. Certainly. 22 MR. MILLET: I think that question assumes a fact 23 not established. 24 MR. SOLOMON: The witness testified at length 25 about discussions the Bureau had with the NAS panel. 545 1 THE COURT: I will permit it. The objection is 2 overruled. 3 A. Let me try to put it this way: 4 We met repeatedly with the panel, we presented 5 documents to the panel, written documents, we presented 6 data, information to the panel, and the panel digested, 7 deliberated, worked with the data and worked with all of the 8 information that we gave to them. 9 Of course, the panel brought their own 10 experiences and skills to bear on these critical national 11 issues as well and ultimately, based on their own 12 experiences and skills, together with the various 13 information that we presented, they arrived at this, at this 14 report and recommendations. 15 Q. Did the Bureau take into account the 16 recommendations made by the NAS panel in determining how to 17 proceed through the 1980s with respect to the design of the 18 PES? 19 A. Oh, very definitely yes. This became some kind 20 of guiding force for us. There were a number of very useful 21 recommendations in this document. 22 This isn't to say that every last recommendation 23 was adopted. They were all subjected to tests of 24 feasibility and so forth and so on, but in philosophy, in 25 spirit if not to the letter, we followed many of the 546 1 recommendations. 2 MR. SOLOMON: Your Honor, the plaintiffs would 3 offer PX 2. 4 MR. MILLET: Assuming this is for state of mind 5 as well, I won't object. 6 THE COURT: All right. Plaintiff's Exhibit 2 is 7 admitted on the same rationale, that is, state of mind. 8 MR. SOLOMON: That is both the Bureau's state of 9 mind and the Commerce Department's state of mind, is that 10 correct? 11 MR. MILLET: I don't think there is a Commerce 12 Department state of mind. 13 MR. SOLOMON: It is a perfectly obvious the 14 Commerce Department state of mind was in the administrative 15 record and we are going to argue to your Honor that all of 16 this relevant history should have been in the administrative 17 record and wasn't. 18 MR. MILLET: Then, your Honor, I do object. Are 19 we are showing the absent of state of mind by showing the 20 document exists? 21 There is no foundation that someone looked at it 22 or not looked at it. I think we are getting pretty far 23 afield. 24 MR. SOLOMON: We were promised the so-called 25 administrative record that had all the relevant documents to 547 1 it. The witness testified that this is one of the key 2 documents and it is nowhere to be found. 3 MR. MILLET: The administrative record contains 4 the documents that were relevant to the Secretary's 5 decision. This particular document may have been relevant 6 to Dr. Wolter's work as he has testified. That has nothing 7 to do with state of mind of the Commerce Department or any 8 decision makers within the Commerce Department. So 9 producing documents that show the absence of state of mind, 10 particularly when Dr. Wolter left the government in 1988, I 11 think we are getting pretty far afield here. 12 THE COURT: Well, it may be a bit unusual, but 13 just as you can admit a business record to show that there 14 is not an entry there when you expect it to be there, it 15 seems to me that when you are dealing with nonhearsay 16 admitted to show a state of mind, you can admit a document 17 to show that this document was out there, it should have 18 been taken into account and it apparently was not. 19 Therefore, on the state of mind of the Commerce 20 Department, arguably evincing a decision to stay in 21 invincible irnorance, on that rationale I will admit it, the 22 goal of invincible ignorance. 23 (Plaintiff's Exhibig 2 marked for identification 24 was received in evidence.) 25 BY MR. SOLOMON: 548 1 Q. Will you look at page 2 of the book, the second 2 to the last bullet on the page says, "As in every census 3 since 1940, statistical sampling methods will be used to 4 obtain responses to many census items," and the paragraph 5 goes on from there, I need not read all of it. 6 At the time that you received this from the NAS 7 panel, did you agree that statistical sampling methods had, 8 in fact, been used since 1940 in the decennial censuses? 9 A. Well, sure, they had been used. 10 Q. And was statistical sampling methods also used in 11 the 1990 decennial census, putting aside the PES for the 12 moment? 13 A. Yes, surely. 14 Q. On the top of page 3 in the first bullet it says, 15 "As has been true since 1960, large computers will be used 16 to process the census returns in a relatively short span of 17 time." 18 Do you see that, sir? 19 A. Yes, sir. 20 Q. Are you familiar with the use that the Census 21 Bureau has made of computers? 22 A. Yes, sir. 23 Q. Are computers foreign to the census-taking 24 process? 25 A. No. In fact, in the early 1950s, Univak 1, 549 1 serial number 1 was delivered to the Census Bureau. This is 2 the world's first electronic computer used for commercial 3 purposes. 4 This is one of the most important facts in the 5 history of the Census Bureau. The Census Bureau was the 6 first organization in the world to use computers. 7 So this statement was true insofar as it goes, 8 but actually the Univak 1, serial number 1 processed parts 9 of the 1950 census, not merely 1960. 10 Q. Do you recall seeing in the Secretary's decision 11 a reference to computers? 12 A. Yes, I recall a passage early in his report. 13 Q. What was the thrust of the Secretary's statement? 14 A. The thrust of his statement was something to the 15 effect that a computerized process should not overturn a 16 tried and true process that has been used for 200 years. 17 Q. In light of what you knew at the Bureau, did you 18 agree with that statement? 19 A. No, of course not. It misses the mark on two 20 grounds: 21 First of all, computers have been used since 22 1950, as I just said. 23 In fact, one of the other principal goals of the 24 Census Bureau during the 1980s was to automate the 1990 25 census to the extent practicable. Automation means 550 1 computing equipment. So he was just off the mark altogether 2 regarding computing. 3 And secondly, he seemed to, at least in my view, 4 I don't know if this was in his state of mind or not, but in 5 my view he seemed to disparage computers as somehow being 6 less than the, less than the other techniques, presumably 7 manual techniques, used or that he believed were used in 8 prior censuses, and that just isn't so, either. 9 Q. Last stop on this book, will you please turn to 10 page 17, the last several -- 11 A. Tab 17? 12 Q. Page 17 of that book. 13 Toward the bottom of the page there is a sentence 14 that begins, "However, the panel," speaking of the NAS 15 panel, "also believes that the ultimate goal of the census 16 should be the accuracy of the census figures. The evidence 17 is overwhelming no counting process, however diligent, will, 18 in fact, enumerate everyone. Hence, the panel recommends 19 that the Census Bureau carry out the vigorous program of 20 research on coverage evaluation and adjustments methods 21 that, if successful, would permit adjustment of census 22 figures as part of the methodology of the 1990 census." 23 Do you see that, sir? 24 A. Yes, sir. 25 Q. Do you agree with the statement that the ultimate 551 1 goal of the census should be the accuracy of the census 2 figures? 3 A. In the final analysis at the end of the day, it 4 almost has to be. 5 There are thousands of uses of census data, but 6 at the end of the day, it seems to me the Constitutional 7 requirement for the census is the one that must prevail. 8 Q. Did you agree that the evidence was overwhelming 9 that no counting device, however diligent, will, in fact, 10 enumerate everyone? 11 A. Absolutely. The differential undercount as I 12 described earlier is just somehow fundamental, fundamental 13 to the way in which conventional census methods interact 14 with the U.S. population. 15 If you can eliminate that differential, you got 16 to use different technique. Trying to do the wrong 17 technique better isn't going to make it, and 40, 50 years of 18 experience have taught us that. 19 Q. Do any histories exist that describe the sampling 20 used by the Bureau in the 1940s and fifties and sixties? 21 A. Yes, surely. There are dozens of papers in the 22 statistical literature authored by Census Bureau staffers, 23 some of them pioneering papers in this area. There are some 24 overviews, some books written, some interviews published by 25 eminent statisticians from the Census Bureau from that 552 1 period documenting the work. 2 Some of the names that come to my mind include 3 Ross Eckler, Morris Hansen, Bill Hurwitz and others. These 4 are historical names in this field. These are people who 5 worked at the Census Bureau, these are people who preceded 6 me in my post at the Census Bureau. 7 Q. In particular, who is Dr. Eckler? 8 A. Eckler was a statistician at the Census Bureau, 9 later he rose to assume the post of director. 10 Q. I would like you to turn to the next tab in the 11 book -- 12 THE COURT: Maybe we can break at this point and 13 resume tomorrow. 14 MR. SOLOMON: Whatever is convenient for your 15 Honor. 16 THE COURT: My attention span has just about run 17 its limits. 18 MR. SOLOMON: Sure. 19 THE COURT: All right. We will recess for the 20 day. 21 We will start again tomorrow at 9:30. 22 (Adjournment taken to Thursday, May 14, 1992, at 23 9:30 o'clock a.m.) 24 25