2449 1 UNITED STATES DISTRICT COURT EASTERN DISTRICT OF NEW YORK 2 ------------------------------x THE CITY OF NEW YORK, et al., 3 Plaintiffs, 88 Civ. 3474 (JMcL) 4 v. 5 UNITED STATES DEPARTMENT OF COMMERCE, 6 et al., 7 Defendants. ---------------------------------x 8 CITY OF ATLANTA, et al., 9 Plaintiffs, 10 v. 92 Civ. 1566 (JMcL) 11 MOSBACHER, et al., 12 Defendants. ---------------------------------x 13 FLORIDA HOUSE OF REPRESENTATIVES, et al. 14 Plaintiffs, 15 v. 92 Civ. 2037 (JMcL) 16 BARBARA H. FRANKLIN, Secretary of Commerce, 17 Defendant. 18 --------------------------------x 19 May 28, 1992 20 9:30 a.m. 21 22 Before: 23 HON. JOSEPH M. McLAUGHLIN, 24 Circuit Judge 25 2450 1 THE COURT: Mr. Sitcov, you look poised to 2 strike. 3 MR. SITCOV: That is just how I feel, your Honor. 4 I neglected to offer into evidence a couple of exhibits that 5 Dr. Freedman testified to yesterday and I want to make that 6 offer now. They are Defendants' Exhibit 66 -- 7 THE COURT: Why don't you take them in order. I 8 think there were some earlier numbers. 9 MR. SITCOV: Yes, there were. The earlier 10 numbers are 62, 63, and 64. Those are the reports that Dr. 11 Freedman prepared that he testified about yesterday. 66 is 12 a letter with enclosures that Dr. Freedman testified about, 13 the letter that Dr. Freedman received from the Census 14 Bureau, addressed to him, and signed by a Census Bureau 15 official. Enclosure 3 was the smoothing model that he 16 testified about yesterday. 17 Then Exhibit 68 was Dr. Freedman's modification 18 to the second assumption in the smoothing model. 19 THE COURT: That's it? 20 MR. SITCOV: That's it, yes. 21 THE COURT: Mr. Solomon? 22 MR. SOLOMON: Good morning, your Honor. 23 THE COURT: Good morning. 24 MR. SOLOMON: We don't object to 66, which is the 25 letter. 2451 1 THE COURT: 66 is admitted. 2 (Defendants' Exhibit 66 for identification was 3 received in evidence) 4 MR. SOLOMON: Exhibit 68, it is a one-page 5 re-mark of a -- 6 THE COURT: That was the one we had in the 7 charts, yes. 8 MR. SOLOMON: Right. 9 MR. SITCOV: That is the one with V sub 3. 10 THE COURT: Yes, V sub 3, I remember it well. 11 MR. SOLOMON: It is one altered page of a report. 12 We don't have any objection to its coming in for 13 illustrative purposes, but I don't think there is anything 14 else to it. I don't think it is otherwise evidentiary. 15 THE COURT: We have enough evidentiary testimony 16 about it in the witness, so I will take it in as an adjunct 17 to his testimony, illustrative only, 68. 18 (Defendants' Exhibit 68 for identification was 19 received in evidence) 20 MR. SOLOMON: With respect to 62, 63, and 64, we 21 have the same objections that we did yesterday and then 22 some. Mr. Sitcov got through his examination and pointed 23 the witness to what he believed were the relevant parts 24 without having these documents in evidence. The parts that 25 weren't discussed really do contain not just double and 2452 1 triple hearsay -- 2 THE COURT: Your problem is with the stuff that 3 was never discussed? 4 MR. SOLOMON: That is correct, your Honor. 5 THE COURT: I agree with you. I will take those 6 parts of the documents 62, 63, and 64 which contain charts 7 and diagrams and things about which the witness testified. 8 All of the rest I will exclude. 9 (Defendants' Exhibits 62, 63 and 64 for 10 identification werr received in evidence) 11 THE COURT: You are ready to resume, Mr. Solomon, 12 or to start? 13 MR. SOLOMON: Ready to start. 14 MR. RIFKIND: It just feels that way. 15 DAVID FREEDMAN resumed 16 CROSS-EXAMINATION 17 BY MR. SOLOMON: 18 Q. Good morning, Professor Freedman. A very 19 important part of the testimony that you gave yesterday 20 concerned your discussion of the standard errors surrounding 21 the undercount estimates in your opinion, is that correct? 22 A. That was one major topic, yes. 23 Q. I would like you to look at DX-64, which is the 24 paper on loss functions, page 4, Table 2. 25 When we were on this chart yesterday, I think in 2453 1 one of my few objections I observed that it didn't have the 2 standard error information. I believe you said, on page 3 2372, "They have been given the data. I gave it to them 4 myself. It's on that diskette." 5 I would like to show you a copy of the letter 6 that was given to us when we got the diskette. 7 MR. SOLOMON: May I, your Honor? 8 THE COURT: Yes. 9 Q. We have marked that for identification as PX-766. 10 I believe you testified that the standard errors 11 here were computed using variance/covariance matrices, 12 correct? 13 A. Yes. 14 Q. Would you please identify for us on this list of 15 the files that you gave us which variance/covariance matrix 16 you used in the calculation of the standard errors that 17 appear on page 4, Table 2? 18 A. The major ingredient for those standard errors is 19 the Bureau's final covariance matrix for the smoothed 20 adjustment factors which appear at disks 1 through 10 dot 21 COV on the last page. COV, C O V. You also need the 22 evaluation -- sorry -- you need the adjustment factors 23 themselves, which are in the SMTH1392.DAT file, and you need 24 the population counts for the various states. Those appear 25 earlier as state dash two digits dot DAT. 2454 1 There is yet another file which I think is called 2 STRESID dot DAT. These, by the way, are also files that are 3 listed with different names in Dr. Fisher's exhibits. Not 4 the covariances, but the population files. 5 Q. Let's stick with the covariances, because I think 6 that is what you told me yesterday I could look at to derive 7 these standard errors. The covariance matrices that are in 8 the files that you gave us are covariances for the 9 adjustment factors, not for the shares, correct? 10 A. That's right. 11 Q. You created a separate covariance/variance matrix 12 in order to calculate the standard errors that appear on 13 page 4, Table 2, correct? 14 A. Right. 15 Q. You didn't give us that covariance matrix, did 16 you? 17 A. I did not. 18 Q. What formula did you use to calculate the 19 covariance/variance matrix that you didn't give us but that 20 you needed to prepare these standard errors? 21 A. You, first of all, derive a variance/covariance 22 matrix for the adjustments to the states. Then you derive 23 by what is called the delta method the variance/covariance 24 matrix for the shares. 25 Q. Is the delta method the only method that could be 2455 1 used to calculate a variance/covariance matrix? 2 A. I don't know that it is the only one. I would 3 think it is the standard one to get from the numbers to the 4 shares. 5 Q. What was given to us you didn't tell us that you 6 used the delta method, did you? 7 A. May I look at my report for a second? 8 Q. Of course. 9 A. Would you please turn to page 31. There is an 10 equation 9 on that page. Below that equation on the page 11 there is a block of type which refers to G. That is the 12 variance/covariance matrix that I believe you were referring 13 to that I had computed from the Bureau's data. It says that 14 I computed X, those are the adjustments, from the Bureau's 15 smoothed adjustment factors, and G from the Bureau's 16 covariance matrix for these factors -- that would be the 17 stuff in the dot COV files -- via the delta method. 18 Q. Here you are telling us how you computed the 19 variance/covariance matrix for Table 9, not for Table 2, 20 correct? 21 A. It is the same matrix and the same method. 22 Q. Where did you tell us that, sir? 23 A. I don't know that I did in the report. 24 Q. When you calculated the variance/covariance 25 matrix -- 2456 1 A. Forgive me. If I may continue. There is a 2 parenthesis in there which explains that the Bureau provided 3 state level census counts by poststrata. Those are the 4 state dash two-digit files. And counts for the residual 5 population, that is the STRESID file. 6 Q. I just want to focus on what you told me to look 7 at yesterday, and that is the variance/covariance matrix. 8 We have now established that we didn't have what you told me 9 to look at. 10 MR. SITCOV: I object. 11 THE COURT: I will sustain the objection. 12 Q. I want the record to be perfectly clear. You did 13 not give us the variance/covariance matrix which you 14 calculated and which you needed in order to calculate the 15 standard errors on Table 2, isn't that right? 16 A. I did not give you the matrix G, that's correct. 17 Q. Did you, in calculating the matrix G, utilize the 18 share data that you had or did you do it analytically based 19 on the Bureau's numbers? 20 A. I may not be answering your question. Let me 21 try. If I don't, just -- 22 THE COURT: He will tell you. 23 THE WITNESS: I actually wasn't afraid he 24 wouldn't, your Honor. 25 A. I recomputed the shares myself using the smoothed 2457 1 adjustment factors that the Bureau provided. I did not use 2 the published shares. 3 Q. There are a number of other tables that you 4 looked at in this report yesterday, including Table 9, which 5 have plus or minus numbers, risk differences, you also do it 6 on Table 10. We couldn't calculate any of those based on 7 the information that was on these files, isn't that correct? 8 A. Oh, I think that is totally wrong. I computed 9 those numbers from the data in these files, and I tried hard 10 to explain how I did it in these technical pages in the 11 report. If I failed, forgive me, but I did try. 12 Q. The fact of the matter is, unless you tell us 13 that you are using the delta method for calculating the 14 variance/covariance matrix, we would have to guess to know 15 which one you used, correct? 16 THE COURT: Except that he said that was the 17 standard method. 18 Q. There is more than one standard method, isn't 19 there? 20 A. For computing the covariance matrix for shares? 21 Q. Yes. 22 A. I suppose there is. I find it hard to think what 23 the alternative method would be just right now. 24 Q. Yours doesn't agree with what the Bureau did, 25 does it? 2458 1 A. I beg your pardon? 2 Q. The yours, the variance/covariance matrix that 3 you calculated, doesn't agree with the Bureau's does it? 4 A. Let me refer to my report again, please. 5 May I please refer to page 33 of my report. 6 There is a Table 15 in the middle of the page, which I did 7 mention yesterday. Below that table there is a discussion 8 which I believe is responsive to your question. Freddie 9 Navarro, if I may refer to him in that way, and I, it says 10 here, are using the same formula for the delta method, and I 11 know that because I checked it with him, and we agree on the 12 variance/covariance matrix G for state population share 13 changes. 14 I know that because we spent an interesting hour 15 on the telephone reading each other numbers from his matrix 16 and my matrix. So I am using pretty much exactly the same 17 variance/covariance matrix that they are. 18 THE COURT: That is your idea of an interesting 19 hour? 20 THE WITNESS: Your Honor, statisticians have very 21 odd ideas sometimes. 22 A. I don't want to say that the numbers were 23 identically the same. But to about six significant figures, 24 they were. 25 Q. You can calculate these data by use of a Monte 2459 1 Carlo method, and you can calculate these data by use of an 2 analytic method. You talked about both of these things 3 yesterday, correct? 4 A. No, I don't think that is right, at least in the 5 way I am using the language here. If you want me to do 6 something else, please tell me. 7 The matrix G is an analytical object which he 8 computed and which I computed analytically. He went on to 9 do a Monte Carlo and actually so did I. That Monte Carlo is 10 discussed at the bottom of page 33 and the top of page 34. 11 There is some difference between the Monte Carlo 12 that I did and the Monte Carlo which he did. That is 13 reported on page 34 in that picture and in the prose. His 14 number came out substantially below the number that I got. 15 The difference has no practical significance whatsoever. It 16 would be of interest only to a pair of statisticians gabbing 17 on the telephone. 18 But his number is, I think, substantially below 19 what you could reasonably get by a Monte Carlo, and I think 20 he agreed with me on that point. But we agreed totally, 21 insofar as you can agree on these things when you exchange 22 numbers between computers, we agreed on the analytical 23 matrix G. 24 Q. Both of you knew what each of you was talking 25 about when you were talking about the analytical matrix, 2460 1 right? 2 A. I am rather sure that that is the case, or I 3 would not testify that way. We talked to each other on the 4 telephone. We exchanged faxes. I may be wrong. Anybody 5 can make a mistake, starting with me, but I am rather -- 6 Q. Just so -- 7 MR. SITCOV: Your Honor. 8 THE COURT: He is finishing. 9 Go ahead. You are finishing your answer. That 10 is going to be his objection. 11 MR. SITCOV: Yes. 12 A. I am rather confident that we were talking about 13 the same matrix, yes. 14 Q. Is it fair to say that the bulk of time that you 15 have spent studying the census and issues of adjustment has 16 been as a retained expert for the government? 17 A. I think that's fair. 18 Q. You would agree that sampling survey is not a 19 subspecialty of yours? You would agree with that, wouldn't 20 you? 21 A. No, I don't think so. I have done work on sample 22 surveys. 23 Q. Do you consider yourself a specialist in sample 24 surveys? 25 A. No, I don't consider myself a specialist in 2461 1 sample surveys. But I have certainly done work in sample 2 surveys. I have published papers about them. I have done 3 consulting on it. 4 Q. With respect to the papers that you published, 5 you mentioned yesterday that you published two papers, only 6 two papers, on the issue of census or issues of adjustment. 7 Only one of those was in a scholarly journal, isn't that 8 right? 9 A. I think that is quite wrong. I published one 10 paper in Statistical Science, which is a good journal in my 11 field. I published one paper in Science, which is one of 12 the great journals in North America if not the world. I 13 have a third paper in press in a journal called Methodology 14 which is a house journal of Statistics Canada, which is the 15 census bureau in my country of birth. 16 Q. Do you consider science one of the learned 17 journals of the statistics profession? 18 A. I consider it one of the great learned journals 19 which covers all fields of science, including mine. 20 Q. Is it accurate to state that when you looked at 21 the smoothing model with respect to the litigation in 1984, 22 that that was your first real encounter with the smoothing 23 model in an applied context? 24 A. That is fair. 25 Q. You, in fact, never structured or performed or 2462 1 analyzed a loss function analysis in order to make any 2 real-life decision, have you? 3 A. That is true, and I think that would be true not 4 only of me but of many others as well. 5 Q. As a general matter, you don't publish on 6 population estimates or projections, is that a fair 7 statement? 8 A. That is true. 9 Q. Generally, in your work, you don't usually use 10 population projections, is that right? 11 A. I use them sometimes. I think I testified at my 12 deposition, and I will certainly testify today, that I have 13 used population projections. But they certainly don't do 14 not form a major basis for most of the work I do. 15 Q. Do you agree, Professor Freedman, that with 16 respect to your writing about the uses to be made of 17 statistics, that you have tended to take a hard-line or 18 extremist position? 19 A. I will accept the former as a characterization, 20 but I will certainly not accept the latter. When people 21 come to me with a model, my question is, and it is my 22 question in this case -- I am not trying to ask you a 23 question. I know you can't do that. 24 Q. All I want is a answer to my question. You have 25 answered it. 2463 1 MR. SITCOV: No. I think the witness was in the 2 midst of answering it. 3 MR. SOLOMON: He already answered it. He said he 4 could accept with respect to hard-line, he wouldn't accept 5 with respect to extremist. 6 THE COURT: Let's move on. 7 Q. It is also fair to say that you are described by 8 others in the field of statistics as being one who takes 9 hard-line or polar positions, is that a fair statement? 10 MR. SITCOV: I object. 11 THE COURT: Overruled. 12 A. I rather think that some of the people in your 13 side of the case would describe me that way, and I think 14 many other people in the profession would not describe me 15 that way. 16 Q. Let me ask you about Professor Meier. He is on 17 your side of the case ostensibly, correct? 18 MR. SITCOV: I object. 19 MR. SOLOMON: He has testified on the behalf of 20 defendants. 21 THE COURT: That is a better way to put it. 22 MR. SOLOMON: You're right. 23 A. I think he is on our side of the case both 24 ostensibly and for real. 25 Q. The defendants offered, and it was admitted into 2464 1 evidence, an exhibit, an article by Professor Meier, DX-80, 2 in which he describes your position under the heading of 3 absolutionist or strict constructionist. Do you think that 4 is a fair label? 5 THE COURT: Absolutionist or absolutist? 6 MR. SOLOMON: Absolutist and strict 7 constructionist, right. 8 THE COURT: Only I give absolution here. 9 A. I guess generally I should say I want to know the 10 context, I want to look at the article, I want to see that. 11 But I think I am not going to do that. I am just going to 12 say no, I don't think Paul was right about that. Whatever 13 it was, he said it, and I wish he hadn't. 14 But if that is his opinion, which he would have 15 found out by asking questions in court, I wasn't hear for 16 that, I haven't read his transcript, if that is his opinion, 17 that is his opinion. What can I do? He and I are still 18 good friends. 19 Q. In another article offered by Professor Meier, 20 DX-81, Professor Meier says, "I do not quite share David 21 Freedman's hard-line defense against foremost statistical 22 inference for observational data." Is that a fair 23 characterization? 24 MR. SITCOV: Of what? 25 A. If that is Paul's position, that is Paul's 2465 1 position. 2 Q. Do you agree with it? 3 MR. SITCOV: Mr. Solomon asked if it was a fair 4 characterization. I am wondering, is he asking is it a fair 5 characterization of what Dr. Meier said or is it a fair 6 characterization of Dr. Freedman's position? 7 THE COURT: Would you clarify that? 8 A. If Mr. Solomon is quoting from Paul Meier's 9 article and Paul Meier said it, then I am sure it is a fair 10 statement of what Paul was thinking at the time. Whether he 11 thinks that in general, I do not know. 12 I think I testified earlier that I take a rather 13 hard-line position on models. My question about a model is, 14 why is it right? If that is what Paul is saying, well, yes, 15 that is my position. I am not doing anything special for 16 the model in this case. 17 Q. In one of the other articles that were offered by 18 the defendants, DX-92, there was a comment to your paper, 19 the one paper on the 1984 litigation that you wrote, by 20 Professor Madansky. Do you remember that comment? 21 A. I sure do. 22 Q. Do you remember that he likens you to the Neturei 23 Karta in Jerusalem? 24 A. A great crack, yes. 25 Q. You actually taught in Jerusalem. You know who 2466 1 the Neturei Karta are, don't you? 2 A. I should, but I don't think I do. I just 3 remember the crack. 4 Q. They describe them as the ultra-extremists who 5 live mainly in Jerusalem, whose quest, if successful, will 6 mean that the state of Israel will cease to exist. Is there 7 some fair analogy between you and the Neturei Karta? 8 A. One of your experts testified in my presence that 9 if my kind of position was adopted, then a lot of 10 statisticians would have out of work. My response to Al's 11 comment in that article was, yes, there is a lot of bad 12 statistics out there, and there is also a lot of good 13 statistics, including some done by Al Medanski. He has a 14 famous paper on how to find the best pastrami sandwich in 15 New York. Bill and I did mention him. 16 Q. When you were retained by the government in this 17 case in 1989, that was in connection with the plaintiffs' 18 motion for a preliminary injunction, isn't that right? 19 A. I believe that's right, yes. 20 Q. Since that motion was resolved, you have been 21 retained more or less on an ongoing basis by the government, 22 correct? 23 A. That is true. 24 Q. Even after the motion was resolved, you continued 25 to advise the government, correct? 2467 1 A. Right. 2 Q. Up to the time of your deposition, I think you 3 told us you had earned, and I just did the calculations -- 4 the last one I am going to do, in fact -- about $150,000 5 from this retention? 6 A. That's right, yes. 7 Q. How much time have you spent since then? 8 A. I haven't added it up, but at a rough estimate 9 maybe 200 hours. 10 Q. When you were talking with Mr. Sitcov yesterday, 11 you said that you visited the Bureau in 1987. At the time 12 it was really before you had formed much of an opinion on 13 the adjustment question. Do you recall testifying to that? 14 A. Yes let me be clear. I had a very clear opinion 15 about '84 or '80, but I had not really formed an opinion 16 about '90. 17 Q. So prior to that time you had, in fact, published 18 your article about the smoothing in the 1984 litigation, 19 correct? 20 A. Yes. 21 Q. It was also in 1987 that you testified before 22 Congress, correct? 23 A. Right. 24 Q. At that time you said, among other things, for 25 one thing, people aren't fish, in trying to differentiate 2468 1 the problems you thought there would be in the dual-system 2 estimation in the 1990 census. Do you recall that? 3 A. I do, and I would even be willing to testify to 4 that today. 5 Q. I don't think we have to disagree on that. 6 A. That is another point of contact between me -- 7 Q. Also in 1987 you expressed concern about the 8 matching that might go on in the 1990 census, didn't you? 9 A. No. I expressed concern about the matching that 10 would go on in the post-enumeration survey following the 11 census. 12 Q. Following the 1990 census? 13 A. Following the 1990 census, right. 14 Q. You also complained about the smoothing 15 techniques that were then available in 1987 -- 16 MR. SITCOV: Objection. 17 A. I won't accept the term "complained." 18 Q. You also criticized the smoothing techniques that 19 were being talked about for the 1990 census, is that fair? 20 A. Yes. 21 Q. Before Congress you, in fact, made a 22 recommendation in 1987 that we should decide now not to 23 adjust, is that right? 24 A. That is correct. 25 Q. Was that right around the time that you hadn't 2469 1 formed your opinion? 2 A. Oh, no. That was after the time I had spent at 3 the Bureau and after the time I had received the documents, 4 had read them, had had some exchange of views with Kirk 5 Wolter. In fact, when I testified in front of Congress that 6 time, the committee wanted those letters, and Kirk and I 7 agreed to turn them over. 8 Q. It was you who suggested Professor Meier as a 9 witness for the defendants in this case, isn't that correct? 10 A. I believe that's right. 11 Q. It was you who suggested Professor Breiman as a 12 witness for the defendants in this case, correct? 13 A. I believe that's right. 14 Q. It was you who suggested Professor Wachter as a 15 candidate for the Special Advisory Panel, correct? 16 A. Again, I think that's right. 17 Q. In fact, I saw you sitting at counsel table for a 18 good part of the first couple of weeks of the trial, didn't I? 19 A. Your eyes do not deceive you. I was there. 20 MR. SOLOMON: With apologies, I have to talk 21 about smoothing. 22 Q. In connection with your own work, you don't 23 perform smoothing analysis, correct? 24 A. I think that's right, but I would say that with 25 respect to the 19484 case or the case about the 1980 census 2470 1 and the case with respect to this case, I have certainly 2 done a lot of smoothing and presmoothing as well. 3 Q. It is your opinion that smoothing is a 4 controversial issue because its operation depends on 5 statistical models whose assumptions have not been validated 6 in your opinion, isn't that right? 7 A. I am not sure that the operation depends on it, 8 but certainly the conclusions you draw from the operation of 9 the Bureau would depend on the assumptions, and that is, 10 indeed, a reason why I think it is controversial. 11 Q. Are you aware that the Bureau used smoothing in 12 connection with the generation of revenue sharing estimates? 13 Are you aware of that? 14 A. I became aware of it in the 1984 litigation when 15 one of the counsel representing plaintiffs told me about it 16 on the stand. I am perfectly happy to accept that they do, 17 if they do. 18 Q. According to you, I take it that the assumptions 19 underlying that use of smoothing also around validated, is 20 that right? 21 A. I can't being make any blanket statements about 22 models that I really haven't looked at. It could easily pe 23 the case that the assumptions are not validated, and I would 24 find it problematic if I looked at it. But I certainly do 25 not want to testify about that particular model. I think 2471 1 that is probably the Fay-Herriot model, and I do not have an 2 informed expert opinion about it. I have opinions about 3 classes of models perhaps, but I can't really testify about 4 that particular model. 5 Q. In your opinion, have you ever met a smoothing 6 model that you liked? 7 THE COURT: Like Will Rogers? 8 A. I met two, and I sure don't like them. 9 Q. You are not aware of any that you do believe are 10 validated, correct? 11 A. Well, I'm not sure that is quite right. I heard 12 John Rolph's testimony about his fire alarm caper. As best 13 you can tell about a model with a five-minute introduction 14 in a courtroom setting, especially from a witness on the 15 other side, I liked what I heard, because he checked it 16 against reality. 17 Q. Thank you. We don't know the true undercount 18 estimate. We don't know the true undercount, correct? 19 A. We certainly do not. 20 Q. So if what you were requiring was testing and 21 validation in the way you described what Dr. Rolph talked 22 about, then we would never have validation of the Bureau's 23 smoothing techniques, is that a fair statement? 24 A. I'm not sure it's totally fair. You can try to 25 validate these models by using them in situations which seem 2472 1 quite analogous to the situation that you are primarily 2 interested in. So I think it would be hard to validate 3 these smoothing models in any strict way. That is certainly 4 part of the reason why I am skeptical about them. In that 5 respect, I think you are fairly characterizing my position. 6 But I have seen rather little work in any attempt 7 to validate these models in analogous contexts. 8 (Continued on next page) 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 2473 1 Q. But you were not -- you didn't work at the 2 Bureau in the 1980s, correct? 3 A. It seems to me I did a tiny bit of consulting for 4 them in the sense of going there a few times and giving them 5 talks and discussing issues with them. 6 Other than that, certainly not. 7 Q. Although you talked yesterday with Mr. Sitcov 8 repeatedly about what the Bureau did and what the Bureau was 9 thinking, you can't really do anything other than speculate 10 about what they were talking about and what they were 11 analyzing within the Bureau, is that a fair statement? 12 A. Oh, I think that's quite wrong, because I do talk 13 to them on the telephone and I do visit there and I do know 14 all those peoples and I see them at professional meetings. 15 If I were speculating here, I would try to label it as 16 speculation. 17 I may be wrong about this or that, but I have not 18 testified to anything which in my opinion can fairly be 19 described as speculation. I testified as to what I had 20 heard. 21 Now, you may not like to admit that kind of 22 testimony and I can't speak to that, that's a legal matter, 23 but what I am testifying to yesterday and today is what I 24 have heard from people at the Bureau and what I have read. 25 Q. There were some articles published about the 2474 1 smoothing that was done as part of the tests, field tests 2 prior to the 1990 census. Did you read those articles? 3 A. I can recall an article by Don Ylvisaker that I 4 read. There were some papers on the smoothing in TARO. I 5 cannot remember the authors. I read some of those. 6 I cannot recall very much about the St. Louis 7 test and I do not recall sharply whether or not I read 8 papers about the smoothing there. 9 Q. You talked yesterday about smoothing and 10 presmoothing and then benchmarking as if they were all 11 different and acceptable techniques. 12 You are aware that each of those was prespecified 13 in the way we have been using that term in this litigation 14 prior to the 1990 census, is that correct? 15 MR. SITCOV: I object. I'm not sure that Dr. 16 Freedman knows how Mr. Solomon is using the term 17 prespecified. 18 MR. SOLOMON: He spent the day yesterday 19 commenting on the evidence by all the witness. It has been 20 defined and used by all those witnesses. 21 Q. If you have an understanding, you can answer it. 22 If not, I will clarify it. 23 THE COURT: The objection is overruled. You may 24 answer. 25 A. I have an understanding and I will answer. I 2475 1 can't believe that the presmoothing and benchmarking is 2 prespecified and I will even tell you why. 3 Q. I didn't ask you why. Thank you. 4 I want to read you from the testimony of Dr. Fay 5 and ask you whether you have any basis to disagree with 6 that. 7 "And the smoothing and the presmoothing 8 operations were almost completely prespecified, weren't 9 they? 10 "A. Close to completely." 11 The question that I have for you is whether you 12 have any basis in fact to disagree with Dr. Fay? 13 A. I do have a basis and I do disagree with him. 14 I'm sorry to disagree with Bob Fay, who is a person I 15 greatly like and respect, but I disagree with him and I have 16 a basis for it. 17 Q. Tell us what parts were not prespecified and then 18 tell us the basis. 19 A. I do not believe that the presmoothing was 20 prespecified -- look, before you push the button, it's 21 prespecified. Everything is prespecified before you push 22 the button. 23 But the presmoothing was not prespecified any 24 great time in advance of the publication of the June 13 25 press release. I have a document in evidence here that 2476 1 shows that in about March the Bureau was ascribing a 2 presmoothing model which was not the model that they used. 3 In fact, there is testimony about it in court. I think we 4 pointed to a logarithmic equation that this document was 5 referring to as presmoothing and that is not the equation 6 that was ultimately used. 7 The benchmarking was not prespecified. There 8 were some extensive rangle about that, to my understanding, 9 in the Bureau and with special advisory panel and I base 10 that on two facts: 11 First of all, Ken Wachter told me about it and 12 second of all they were floundering around trying to compute 13 the variances after benchmarking. If benchmarking were 14 prespecified they would have known how to compute the 15 variances. 16 Q. Did you hear the testimony here that the Bureau 17 had decided upon a method for presmoothing seven months 18 before the time that you are now talking about ? Did you 19 hear that testimony? That's my question? 20 A. I'm going to try to answer it. 21 I heard, I think I heard to my recollection John 22 Rolph explain the general idea of presmoothing was 23 prespecified. That is the only testimony that I can recall 24 that speaks to that issue. 25 Q. You testified yesterday that you believed that 2477 1 bias is introduced by smoothing, is that correct? 2 A. I believe it and so do many of the other experts 3 who testified here. 4 Q. You have never attempted to verify if smoothing 5 introduces bias, isn't that a fair statement? 6 A. That is not a fair statement. If you look at the 7 corrections to my deposition transcript, which we supplied, 8 I explained to you that in my 1986 statistical science 9 article, Bill Navidi and I offered mathematical prove that 10 smoothing can introduce bias. 11 I think I also added to the corrections -- I 12 believe, I hope in the corrections I said that some of the 13 computer experiments we did with variable selection 14 algorithms points to the possibility of bias introduced by 15 smoothing. 16 What is I think fair to say is that we have not 17 quantified the amount of the bias in 1990. 18 Q. Just so the judge understands, the deposition 19 that you took and the answers that you gave are on page 303. 20 "Q. Do you have a conclusion about how much 21 bias is introduced by smoothing? 22 "A. No. 23 "Q. Did you do any studies to attempt to 24 quantify how many biases is introduced by smoothing? 25 "A. Not as a separate item. 2478 1 "Q. So you have never attempted to varify 2 whether, in fact, smoothing introduces bias? 3 "A. I don't remember having done so." 4 MR. SITCOV: Your Honor, if this is used as 5 impeachment, which is what I think Mr. Solomon is doing, 6 then he should also read the correction page that Dr. 7 Freedman testified about just a moment ago and that was 8 provided to plaintiffs within the 30-day period. 9 MR. SOLOMON: The long answer that the witness 10 says he added was his correction to perfectly clear 11 questions that were made. I don't have any need to do more. 12 If you want to do it on redirect, you are welcome to it. 13 THE COURT: You can do it on redirect, but we are 14 here now, let's take care of it. There may not be any 15 redirect. 16 MR. SITCOV: That's right. If Mr. Solomon has 17 the correct -- 18 THE COURT: Would you read me the corrected 19 version? 20 MR. SOLOMON: I will be happy to. Let me get a 21 version. 22 I assumed it was correct when he said it. 23 MR. SITCOV: I move to strike it, your Honor. 24 THE COURT: I will strike that. 25 MR. SOLOMON: Bear with me one moment while I get 2479 1 it. 2 THE COURT: All right. 3 (Pause) 4 MR. SOLOMON: I am prepared to take the witness' 5 recollection, he just testified at length as to what he said 6 he did. I don't have any problem with that. 7 THE COURT: All right, let's move on. 8 BY MR. SOLOMON: 9 Q. One of the points that you were making yesterday 10 is that in your opinion additional sources of variance exist 11 in the dual system estimation process other than are capture 12 in the total error model. Is that a fair statement? 13 A. I'm not sure it's quite fair. 14 I think what I was saying was that, first of all, 15 I thought the variances as measures of sampling error were 16 substantially underestimated by the Bureau. That was point 17 one. 18 And that point two, there are sources of error in 19 the dual system estimates other than sampling error which 20 are not captured by the variances. 21 Now, some -- I'm sorry, I do want to try to 22 finish this answer because it is a complicated thought and 23 I'll lose it. 24 The other sources of error are not quantified at 25 all in the variances, but many of them, in my opinion not 2480 1 all, perhaps even not even important ones, but many of them 2 are captured, I think was the term you used, in the total 3 error model and reflected in the loss function analysis. 4 So my position is that the variances measure only 5 sampling error and they understate those. 6 Many, but not all, and not even all important 7 sources of error, are captured in the total error model and 8 reflected in the loss function analysis. 9 Q. I will ask you about variances. You answered 10 that question very early in the questions. 11 A. You asked me about -- 12 THE COURT: Let's not fight with each other. 13 Q. With respect to your estimate of the additional 14 variance, your best estimate of the additional variance is 15 about two times, is that a fair statement? 16 A. As a measure of sampling error, yes. 17 Q. And all the work that you have done and all the 18 opinions you expressed yesterday was really based on an 19 assumption that the variance was about two times more than 20 what it had been estimated it to be, is that right? 21 A. No, I don't think that's right at all. 22 Q. To the extent you were talking about variance 23 yesterday, have you ever calculated a variance that you can 24 justify greater than two times? 25 A. I think so. For example, with that alternative 2481 1 assumption, if you take that, then you get variances for 2 shares which are nearly three times what the Bureau's 3 variances are. 4 Q. Thank you. 5 Have you calculated anything greater than three 6 times? 7 A. I don't believe so. Speaking strictly as a 8 sampling error, the answer to that is no. 9 Q. I would like you to turn, please, to DX 64, 10 again, the loss function paper, to the glossary, way at the 11 back, page 2. 12 Let me ask you just as a predicate whether I was 13 understanding how the various papers fit together. 14 The shorter loss function paper you talked about 15 -- Professor Freedman, are you all right? We can go off 16 the record. 17 A. No, I am fine. My eyes are bad so I can't see 18 both you and the document. 19 Q. Okay, fine. 20 The first paper, the short paper, talks about 21 changing the variables that are used in the smoothing and 22 you used the results of that in the larger smoothing paper 23 and you used the results of that in your loss function 24 paper. Is that generally correct? 25 A. Well, Exhibit 63 is about alternative variability 2482 1 selection procedures and some of those results are used in 2 Exhibit 62 and some of the calculations in Exhibit 62 feed 3 into Exhibit 64. 4 It's a little confusing because the discussion of 5 sampling error in Exhibit 64 is really the front end to 6 Exhibit 62. 7 Q. Okay. 8 A. I apologize for the confusion. 9 Q. Okay. That clarifies it. 10 Now I am on page 2 of the glossary. 11 A. Yes, sir. 12 Q. And you have the equation there for the dual 13 system estimator. 14 A. Yes. 15 Q. The EE term there, what is the EE term? 16 A. EE is the estimated number of erroneous 17 enumerations from the post-enumeration survey. 18 Q. And that is a sample estimate, is that right? 19 A. It is. 20 Q. Are you aware that the Bureau took that EE and 21 modified it before including it in the formula? 22 A. I'm aware that the Bureau uses a different 23 formula for 1990 than this formula. This is the right 24 formula, I think, for 1980, it is not the right formula for 25 1990. 2483 1 I'm not quite sure what you are referring to. 2 Q. This is not the right formula for 1990 is what I 3 am referring to, but I thank you. 4 I would like you to talk to the small paper, the 5 short smoothing paper, DX 63. I would like you to turn to 6 page 18, under the heading, "Smoothing the covariance 7 matrices." 8 Do you have that? 9 A. Yes. 10 Q. Is this the formula that you used to smooth the 11 covariance matrices, or as you were calling it yesterday, 12 presmooth, in this short paper? 13 A. In the short paper, yes, not in the long paper, 14 but in the short paper. 15 Q. And you know that this is an incorrect formula, 16 that is, it is not the formula the Bureau, in fact, used, is 17 that a fair statement? 18 A. I do not accept at all the characterization that 19 it is incorrect. It was chosen as an alternative. It is 20 rather close to the procedure that was described in Cary 21 Isaki's paper of March. 22 It is certainly right to say that it is different 23 from the method the Bureau used for the production dual 24 system estimator. 25 Q. Now, you say that in the larger smoothing paper 2484 1 you changed the formula for presmoothing, is that right? 2 A. In the main paper, when we are replicating the 3 Bureau's model, we used their procedure for presmoothing. 4 When we present results from this paper, of 5 course, the presmoothing is done the way it is done in this 6 paper. 7 Q. But what I want to be clear on is that in your 8 larger smoothing paper, when you do the presmoothing, when 9 you calculate the presmoothing under the Bureau method, you 10 are not using the log equation, is that right, you are using 11 a different equation? 12 A. I'm using the Bureau's equation. 13 Q. You are aware that the Bureau considered the log 14 equation which you use in your first smoothing paper, aren't 15 you? 16 A. Not only am I aware of it, that's why we did it 17 like -- well, that's not quite right. Part of the reason 18 we did it was because it was a suggestion of the Bureau's 19 and partly it's because my proposal for presmoothing drove 20 Richard Cutler crazy, he just wouldn't do it that way. 21 Q. Are you aware that the Bureau rejected this 22 formula for use in presmoothing? 23 A. Obviously, because they presmoothed a different 24 way. 25 Q. Do you agree that the reason they rejected it was 2485 1 because this formula requires more ratio adjustment, more 2 benchmarking than the formula that was, in fact, used? 3 MR. SITCOV: I object. 4 THE COURT: Overruled. 5 A. No, I don't really know why the Bureau chose the 6 method that they chose over this logarithmic equation. 7 I do agree that the benchmarking of the variances 8 is an argument against these methods, but I'm not -- I 9 don't know that that's why they made the choice that they 10 made. 11 Q. What you are testifying to is that you agree by 12 using the method that you used, you have to do more 13 benchmarking than, in fact, the Bureau's method? 14 A. If I said that I was clumsy. What I was trying 15 to say is I agree that the need to benchmark the predicted 16 variances is a flaw in these methods. 17 What I do not know and I will certainly accept 18 your statement of the fact if you are telling me that, is 19 that with this method there is more benchmarking than with 20 the other method. 21 Q. Let me show you PX 145. 22 MR. SOLOMON: May I, your Honor? 23 THE COURT: Yes. 24 (Handing to the witness) 25 (Pause) 2486 1 Q. I would call your attention to the second page, 2 capital B, under modeling the estimated sampling variance. 3 Tell me if I am correct that the log formula 4 which you use is rejected here and the Bureau says of that, 5 "The ratio factor for the nonlog dependent variable was much 6 closer to the one, so we decided to use the one that they, 7 in fact, used as the dependent variable." 8 Do you see that, sir? 9 A. Are you asking me if I see what they have written 10 about that? 11 Q. That's the first question, that's all I am 12 asking. 13 A. Yes, I do see it. 14 Q. Does that mean that the log equation that you 15 used for presmoothing requires more benchmarking than the 16 one that the Bureau, in fact, used? 17 A. No, it does not, and the reason is that the 18 equation we are using is different in detail from the 19 equation that they are using. 20 Their equation and our equation involved logs and 21 they involved regression, but the detailed formulation of 22 the equations is different. 23 Our equation, I think, is explained as you were 24 pointing me to at the bottom of page 18 and their equation 25 is explained in this right-hand column of type in this 2487 1 exhibit you handed me, and at least to a statistician there 2 are clear differences between the two methods. 3 Now, I don't want to go beyond what I know. It 4 could be that with my equation, too, there is more 5 benchmarking than with the Bureau's equation, but I do not 6 know that and it does not follow from what you have showed 7 me. 8 Q. You didn't check to see whether yours, the 9 formula that you were using required more benchmarking than 10 the Bureau's before you used it in this paper, is that 11 correct? 12 A. That I must concede. 13 Q. Did you anyplace in this paper explain that what 14 you were using as the presmoothing was not, in fact, what 15 the Bureau was using? 16 A. May I just take a look, please? 17 Q. Of course. 18 (Pause) 19 A. May I ask you, please, to look at the top of page 20 19, but below the equation there, and it says, "This 21 method," in other words, the method of presmoothing that we 22 have just discussed in the paper, "This method is similar to 23 the one used by the Bureau, but differs in detail." 24 Q. Is that it? 25 A. That's it. 2488 1 Q. One more -- 2 A. Let me look at the other report for a second, 3 please. 4 (Pause) 5 A. No, I think that's all. 6 Q. You agree, don't you, Professor Freedman, that in 7 fitting the smoothing model, the choice of variables is 8 critical? 9 A. Critical, I think, is too strong, but I do think 10 that it matters. 11 Q. It matters a lot, is that a fair statement? 12 A. I think that is fair. 13 Q. And you are aware that the Bureau choose the 14 variables for the smoothing that was done as part of the 15 PES, correct? 16 A. Yes, sir. 17 Q. And the method that was chosen was the best 18 subsets regression, correct? 19 A. Well, not quite. You have to throw in Colin 20 Mallows, and the method is called the capital C sub lower 21 case P method. 22 Q. Is it your testimony that it is unfair to 23 characterize the Bureau's choice of variable method as the 24 best subsets regressions method? 25 A. I think it is. I think you have to throw in this 2489 1 Mallows C sub P business. 2 Q. Okay. 3 And the variables that you use in the paper and 4 is the basis on which you testified to yesterday, those 5 variables are different from the variables the Bureau used, 6 correct? 7 A. Well, I don't think that's fair. 8 What we did is we started with the exact same 9 variables that they started with, in fact, they supplied 10 them to us, and then we used a different variable selection 11 rule. 12 So what is quite fair to say is that our final 13 equation has different variables in it from their equation. 14 Q. Okay. 15 You say that you started with exactly the same 16 variables, but you used a different choice rule. 17 What rule did you use? 18 A. Well, I thought it was explained in the paper, 19 but if I can try to summarize it here, what we did was -- 20 well, sorry, let me back up to the beginning. 21 Q. Why don't I just rephrase the question. 22 I withdraw the question. 23 THE COURT: Yes. 24 Q. I think I can just speed it along. If you think 25 you described it yesterday, that's why I am raising it now. 2490 1 You look for variables with T ratios greater than 2 2, is that correct? 3 A. Yes. 4 Q. And in what you did, you included all of the 5 Bureau's variables with T ratios greater than 2, is that 6 correct? 7 A. Well, that's the part that's not quite right and 8 I was trying to explain that. 9 That is, they have some variables which are built 10 up out of other variables by multiplication, subtraction and 11 so on and so forth, and so we discarded all those composed 12 variables. That left us with, so to speak, their initial 13 stock. 14 Then we applied this variables selection rule 15 that you described, namely, that if the T statistic was 16 bigger than 2 we kept it. 17 I think there was a case on refitting there were 18 insignificant variables and then we made a second pass. 19 So that is our algorithm. 20 Q. So in creating your variables, you took only some 21 of the Bureau's variables that had T ratios greater than 2, 22 correct? 23 A. We took the variables that had a T ratio bigger 24 than 2 on the second pass, so that would exclude some that 25 had a T ratio bigger than 2 on the first pass, that's right. 2491 1 Q. And do you know that you excluded by doing that, 2 I don't know, between three and six in each of the various 3 regions that you calculated variables for? Is that a fair 4 estimate? 5 A. I'm really not with you. 6 Q. You said that your method of selection was to 7 look for variables with T ratios greater than 2, but you 8 didn't take all the variables with T ratios greater than 2, 9 even some variables with T ratios greater than 2 you 10 discarded? 11 A. No, no, we fitted the equation again and if in 12 the second pass the T statistic was smaller than 2, we 13 omitted them. 14 Now, I can't remember in how many regions that 15 happened. I believe it happened in the northeast and we 16 eliminated two variables on that basis. 17 If you want me to look at the paper, we do report 18 all that and I can count them for you. 19 Q. You talked about the cumulative variables, right? 20 The Bureau used those, right? 21 A. Cumulative? 22 Q. It is just easier to show you the document. 23 I would like to show you DX 69. 24 Do you have this in your binder, DX 69? 25 A. Yes. 2492 1 Q. One example will proof the point. 2 Will you turn, please, to page 6. 3 This document, by the way, sets out the variables 4 that the Bureau used, correct? 5 A. Yes. 6 Q. And on page 6 are the variables that the Bureau 7 used for the south region, correct? 8 A. Yes. 9 Q. And when you did your analysis, you discarded 10 F 25, F 107, F 108, F 109, F 113, isn't that right? 11 A. Well, I'm sure we discarded, without looking at 12 anything, I'm sure we discarded F 107, because that's one of 13 those composed variables which is obtained by multiplying 14 the indicator for Houston, Dallas and other large central 15 cities with percent of Hispanic, so that's what I called a 16 composed variables and we for sure eliminated that. 17 We may well have eliminated variables such that 18 the Bureau's T statistic exceeded 2. 19 We were not using their T statistics, we were 20 using our own T statistics, and if I was unclear about that, 21 I'm glad to have a chance to clarify it. 22 Q. Okay. So you discarded the ones that I mentioned 23 and you also performed your own T statistics and then 24 discarded some more, is that right? 25 A. No, no, no. We did not look at their T 2493 1 statistics in choosing our variables. What we did is we 2 started with their list of variables, we discarded what I am 3 calling the composed or derived variables, of which a good 4 example was F 107 in the south or F 108 or F 109 or F 113. 5 We then ran a generalized least squares regression using a 6 variance covariance matrix somewhat different from the 7 Bureau's. 8 If in that regression the T statistic was less 9 than 2, we discarded the corresponding variable and we 10 refitted, and in a couple of cases where the T statistics 11 were less than 2 in the second pass we discarded those. 12 Q. Do you agree, Professor Freedman, that the reason 13 the Bureau sought to smooth and presmooth and benchmark was 14 to reduce the effects of sampling error while preserving to 15 the maximum extent possible the meaningful systematic 16 differences among the adjustment factors? 17 A. Well, that's somebody's version of the events. I 18 am not sure I want to adopt that as my own testimony. 19 I believe they smoothed to reduce sampling error. 20 The presmoothing and the benchmarking I think is about 21 something else. 22 THE COURT: Shall we take a little recess here? 23 MR. SOLOMON: Yes. 24 THE COURT: We will resume at 11. 25 (Recess) 2494 1 THE COURT: Mr. Solomon. 2 MR. SOLOMON: Thank you, sir. 3 BY MR. SOLOMON: 4 Q. Professor Freedman, you told us there was one 5 formula for presmoothing using one of the papers and another 6 in another of the papers. 7 I would like to ask you a couple more questions 8 about the presmoothing formula that are being used here that 9 you testified to yesterday. 10 In terms of the choice of variables for the 11 modeling of the variances for the presmoothing part of this 12 process, the Bureau used what? 13 A. Beats me. As far as I can tell, they just picked 14 those variables. 15 I asked Cary Isaki how he did it and he said, "I 16 just picked them." 17 Q. And what did you do in your papers? 18 A. Well, let's take them separately. 19 In Exhibit 63, which is the alternative models 20 for smoothing adjustment factors, the rule we used was to 21 use exactly the same variables for the presmoothing that we 22 were using in the regression model for smoothing, so we only 23 had one variables selection algorithm. 24 Q. Okay. And in the larger paper? 25 A. In Exhibit 62, we used exactly the variables that 2495 1 the Bureau used. 2 Q. I see. So you knew which variables the Bureau 3 used? 4 A. Let me try and find an exhibit if I may. 5 (Pause) 6 Yes, they are listed in Exhibit 63. 7 Q. So you now know which the Bureau used. 8 Did you use the same variables for the 9 presmoothing as the Bureau did? 10 A. Well, I have to differentiate between the two 11 papers. 12 In the alternative selection rule paper with Dan 13 Costa and Richard Cutler, we followed the rule that I said, 14 namely, for presmoothing we used exactly the same variables 15 that we used for smoothing. 16 In the smoothing paper itself, when I was trying 17 to replicate the Bureau's model and in the simulations and 18 so forth, I used the variables that the Bureau used for 19 presmoothing. 20 And to complete the answer let me just refer to 21 that list of files that we were looking at. 22 (Pause) 23 The handwritten notes in that bundle of papers 24 that is PX 766 gives the layout to the files, and the file 25 names, although the Xeroxing is not so good, are V SMOG 2496 1 X 1.DAT and so forth. V SMOTH X 1.DAT, that's for region 2 one, and there are similar files for two, three, four and 3 five. 4 Q. I wasn't asking you about that, but thank you. 5 Let's talk about outliers. 6 By the way, you referred to California yesterday 7 as an outlier. You surely did not mean that in any 8 statistical sense, correct? 9 A. I don't recall -- well, California is an outlier 10 in many respects. 11 I don't recall referring to it as an outlier. I 12 recall saying that I thought California's estimated risk 13 numbers were so big that they skewed the loss function 14 analysis, but I don't recall referring to it as an outlier. 15 Q. What outlier method did the Bureau use? This is 16 with respect to the modeling of the variances, the 17 presmoothing part of the process. 18 A. I'm not sure they used a method. They selected 19 some of the ones, I think they are the big ones in each 20 region, and those are listed in one of these exhibits, 69. 21 Q. Did you use the same method in your papers? 22 A. Well, I have to answer separately for the two 23 papers. 24 In the paper on alternative selection models, no, 25 we did not, we used our own method, and since we took logs, 2497 1 I don't think we had big outliers, maybe we did, I don't 2 remember that. 3 In the main paper on smoothing, we followed in 4 most of the analysis exactly the same principles that the 5 Bureau did, we used exactly their formulas, as best I could 6 determine it. 7 In the simulations we did not use a rule for -- 8 no, that's going to be wrong. 9 In the simulations we did have outliers in the 10 variance world that we had to deal with and we dealt with 11 them by a different principle. 12 Our problem was not big variances, but due to the 13 way we were simulating them we came up with some negative 14 variances and we truncated those below at a convenient small 15 positive number. 16 Q. By a different method in your last answer, you 17 just mean different from the way the Bureau did it, correct? 18 A. In the simulations, yes. 19 Q. Thank you. 20 With respect to ratio adjustment of the formula 21 for the variances, modeling the variances in the 22 presmoothing, okay, what did the Bureau do with that? 23 The Bureau did ratio adjust, didn't they? 24 A. I believe so. 25 Q. Okay. 2498 1 What did you do? 2 A. In the main part of the paper where we are 3 replicating their model, we did exactly the same thing and 4 we checked that we were able to replicate their presmoothed 5 variances. 6 In the simulations, we ratio adjusted the way the 7 Bureau did, that is, we ratio adjusted the simulated 8 predicted presmoothed variances to have the same sum as the 9 simulated real variances. 10 I don't know if that's such have a clear answer. 11 I can try again if you like. 12 Q. You did exactly what the Bureau did in your 13 simulations? 14 A. As far as the ratio adjustment, yes. 15 As far as the outliers, no. 16 Q. All right. 17 With respect to the factors themselves, the 18 adjustment factors themselves, the Bureau had a rule for 19 treating outliers, correct? 20 A. It did. 21 Q. And you did not follow that rule in your 22 simulations, correct? 23 A. That is right. 24 Q. And in terms of the method of estimation, the 25 estimation method, the Bureau used a maximum likelihood 2499 1 method for the adjustment factors, correct? 2 A. Yes. The Bureau used maximum likelihood. 3 Q. And you used a different one, called method of 4 moments, correct? 5 A. That is certainly true for the main tables in the 6 paper. 7 We did some checks to see whether there was any 8 difference between method of moments and maximum likelihood. 9 That is also reported in the paper. 10 In the simulations we ran we saw little 11 difference, except in one case. In that case I was not sure 12 the code was right for other reasons. 13 Q. And with respect to ratio adjustment of the 14 adjustment factors or benchmarking, the Bureau did ratio 15 adjust and you didn't, correct? 16 A. In the -- on the paper where we are discussing 17 their results, replicating their results, sometimes I 18 benchmarked, sometimes I didn't, and the notes to the tables 19 try to explain which is which. 20 Q. Okay. 21 A. In the simulations we did not -- I did not 22 benchmark and I think that is indicated, too, in the notes 23 to the table. 24 Q. With respect to whether you are going to use 25 sigma squared I or your preference V sub 3, I would like to 2500 1 ask you a couple of questions about that. 2 The one that you have used, V sub 3, has the 3 effect of making the variance covariance matrix of the true 4 exactly proportional to the variance covariance matrix of 5 the raw adjustment factors, correct? 6 A. That's right, although I don't so much like 7 characterizing it as my preference. 8 Q. Okay. The one you talked about yesterday? 9 A. It sure is. 10 Q. All right. 11 And the effect of the one that you talked about 12 yesterday pulls adjustment factors the same distance to the 13 regression model, all of the adjustment factors, correct? 14 A. Well, that would be true if you were using it to 15 adjust the population. However, that is not what I was 16 using that matrix for. 17 What I was using that matrix for was as an 18 alternative basis for computing the standard errors of the 19 Bureau's procedures. In other words, I am following the 20 Bureau in adjusting the population of the United States as 21 they did, but I am running a sensitivity test on their 22 method for computing the standard errors. 23 So that alternative, the scissors and scotch tape 24 alternative is offered not as a method for adjusting the 25 United States, but as an alternative way of computing the 2501 1 standard errors. 2 If we were to adjust the United States that way, 3 I think what you said is quite right, that it would be 4 shrunk down by a constant factor. 5 Q. Okay. 6 Are you aware that the Bureau considered and 7 rejected the V sub 3 or a substantively equivalent 8 adjustment factor? Yes or no? 9 A. I've been told that. 10 Q. And I take it that it is correct that you never 11 talked with Dr. Fay about your proposed adjustment factor 12 prior to the time that the Bureau used sigma squared I, is 13 that correct? 14 A. I think that's right, yes. 15 Q. You also know that Dr. Fay prefers what the 16 Bureau did to what you did? You know that, too, don't you? 17 A. No, I don't, but if you tell it to me, I will 18 accept what you say. 19 Q. I will ask you -- it's transcript page 1938. 20 "In your view, Dr. Fay, is that a preferable 21 alternative to the assumption used by the Bureau in the 1990 22 smoothing? 23 "A. No, it's not preferable." 24 MR. SITCOV: I object. I'm not certain what the 25 purpose is of reading to -- 2502 1 MR. SOLOMON: I haven't asked him a question. 2 MR. SITCOV: He is simply reading to the witness 3 Dr. Fay's testimony and the witness -- 4 THE COURT: I presume he is going to ask him to 5 comment on it. 6 Is that basically it? 7 MR. SOLOMON: Yes. 8 THE COURT: Comment briefly. 9 BY MR. SOLOMON: 10 Q. No. You don't have any basis to disagree with 11 that, do you? 12 A. Well, if that's what Bob Fay things, that's what 13 Bob Fay things and I was happy to take your word for it, but 14 it's not what I think. I mean, it isn't -- . 15 Q. You disagree with the Bureau. All right. 16 A. Oh, I disagree with the Bureau in many 17 particulars, including that one. 18 Q. Let's talk about loss functions if we can for a 19 few minutes. 20 You agree that a loss function is a conventional 21 summary statistic? 22 A. I think that's right. 23 Q. All right. And in your opinion, it would be 24 unreasonable to -- if you are doing a loss function 25 analysis, it would be unreasonable to ignore areas that have 2503 1 a lot to gain and a lot to loss merely because they have a 2 lot to gain and a lot to loss? Do you agree with that 3 statement? 4 A. I'm not sure the amount of gain or loss in terms 5 of population share or dollars or Congressional 6 representation plays a large role in my thinking. The 7 amount of gain or loss in accuracy plays some role. 8 Q. But you agree that if what we are talking about 9 is gain or loss not in the sense of accuracy but as a result 10 of a correction, you agree that it is illegitimate to 11 disregard large gains or large losses merely because they 12 are large gains or large losses, don't you? 13 A. I'm not sure we are agreeing with each other 14 here. 15 Q. Okay. 16 A. That is, I think that it is good to pay attention 17 to gains in accuracy, large or small, and thing about them 18 and thing about whether the loss function is a good summary 19 statistic. 20 I'm not sure that it is such a good idea to think 21 about the actual gains or losses in terms of dollars and 22 Congress and so forth. 23 I'm not sure it is a bad idea to think about 24 that, either. 25 Q. Am I correct that in your opinion, if one could 2504 1 show that adjustment was going to move the shares in the 2 right direction by a substantial amount and only for a few 3 states would it be worse and that the amounts by which they 4 would be made worse is small, that in your opinion, you 5 would favor adjustment? 6 A. That's right. I was adopting Fisher's testimony, 7 Dr. Fisher's testimony. 8 If the moves in the right direction are a lot 9 more numerous and a lot bigger than the moves in the wrong 10 direction, then that's a good basis for adjustment. 11 Q. Even if a few states were made a little bit 12 worse, that wouldn't stand in the way of adjustment, in your 13 view, right? 14 A. That is correct. 15 Q. I would like to show you an exhibit. 16 MR. SOLOMON: May I, your Honor? 17 THE COURT: Yes. 18 (Handing to the witness) 19 (Pause) 20 Q. That for identification is PX 757. 21 You recognize this, don't you, Professor 22 Freedman? 23 A. I surely do. 24 Q. And what this shows is a hypothetical adjustment 25 technique, the original enumeration counts, in essence, a 2505 1 total of five for each of five areas, for the total of five 2 areas, if the adjustment moves us to nine, but the truth is 3 at ten, correct? 4 A. Right. That's for total. 5 Q. Total. 6 And in this example, states or units B, C and D 7 are made worse off, disimproved by the proposed adjustment, 8 correct? 9 A. Well, B, C and D are -- sorry, may I just do the 10 arithmetic again? I know I saw this exhibit before, but I 11 would like to do the arithmetic. 12 Q. Go right ahead. 13 (Pause) 14 A. Okay, now I have done the arithmetic, but I have 15 lost the question. 16 Q. No problem. 17 This shows that as a result of the adjustment, 18 states B, C and D or units B, C and D are disimproved, 19 correct? 20 A. States B, C and D have their -- well, I don't 21 have such a simple answer to that, I'm afraid. 22 This is an interesting example, because states B, 23 C and D have the accuracy of their shares made slightly 24 worse or somewhat worse, however, they actually gain from 25 adjustment. That's one interesting feature of this example 2506 1 which was presented to me earlier. 2 Q. Right. You say they gain because in terms of 3 strict numeric accuracy they are better numbers by the 4 adjustment, correct? 5 A. No, actually, I was looking at shares. 6 Q. Okay. 7 A. And I observe that the shares for B, C and D are 8 made slightly worse in terms of accuracy, that is, the 9 census or original enumeration gave them each 20 percent, 10 the truth is 20 percent, the adjustment moves them up to 22 11 percent, so that introduces a little or, I don't know if I 12 should say little, that introduces an inaccuracy, but 13 interestingly, it is an inaccuracy which actually works to 14 the benefit of B, C and D because they gain population 15 share. 16 Q. What I am saying is, just so the court 17 understands it, as a result of this hypothetical, states B, 18 C and D which were each at 20 percent before the adjustment, 19 and we know the truth is 20 percent for each, each of those 20 move away from 20 percent and, therefore, are made more so, 21 is that correct, in terms of the accuracy of the shares? 22 A. Right. In terms of the accuracy of the shares, 23 states B, C and D are made worse. 24 Q. And state E is very substantially improved, 25 though it doesn't go all the way to truth, correct? 2507 1 A. Yes, states A and E are improved in terms of the 2 accuracy of their shares, B, C and D are made worse. 3 Q. And in this hypothetical, you would agree to 4 adjust, correct? 5 A. Yes, I would. The improvement for A and E is 6 substantial, the disimprovement, if I can adopt that word 7 for the moment, for B, C and D seems relatively small. 8 B, C and D seem to suffer no practical harm, 9 indeed, some practical gain from adjustment and, 10 furthermore, and this is a feature which I think is really 11 important and is not captured in the Bureau's loss function 12 analysis, the population ratios, that is, A to B, B to C, 13 A to D and so forth, are made better. 14 The example is a little unrealistic, of course, 15 because, well, there are several features, including -- 16 Q. I am not asking you that, sir. You can be asked 17 that on redirect. 18 In your opinion, if someone looking just at this 19 hypothetical were to thing that one shouldn't adjust, you 20 feel comfortable saying he or she was borderline 21 unreasonable, though you are not going to tell us which side 22 of the border, is that a fair statement? 23 A. That's fair. 24 Q. Okay. 25 I would like to have you look at page 6, figure 1 2508 1 of your loss function paper. 2 A. Do you mean figure 1 on page 6? 3 Q. Yes. Page 6, figure 1. 4 Do you have it? 5 A. Yes. 6 Q. Okay. 7 What this shows is that according to the Bureau, 8 there are some states for which we are highly confident that 9 an adjustment would improve them and a large number of 10 states where we cannot tell, is that correct? 11 A. I'm not sure that's quite right. It's close. 12 What we see here is there are a lot of states 13 where you can't tell whether the proposed adjustment is just 14 reflecting sampling error or is reflecting real differential 15 overcounts and undercounts. 16 There are a number of states, however, where 17 there is clear statistical evidence that the proposed 18 adjustment is not just reflective of sampling error. 19 Whether adjustment would be improving the shares is yet 20 another question. 21 Q. Okay. 22 It would be incorrect to state that with respect 23 to the lines in the middle that cross zero, improvement will 24 make the shares worse, isn't that right? 25 A. I think you mean adjustment would make the shares 2509 1 worse rather than improvement. 2 Q. I do, yes. 3 That would be wrong to say, correct? 4 A. Well, we don't know for sure. I think for 5 confidence intervals that cross zero, I would have very 6 little personal belief in the efficacy of adjustment, 7 because we haven't even demonstrated that the census is 8 wrong. 9 Q. So you don't have any basis to believe that they 10 are, in fact, made worse, you are uncertain, is that right? 11 A. Well, we are to some extent uncertain even about 12 states where the confidence interval fails to cross zero, 13 but we are, we are uncertain about all of these states. The 14 uncertainty is if the improvement from adjustment seems to 15 me to be very large for those states where the confidence 16 interval crosses zero. 17 Q. If on the basis of this data we didn't adjust, I 18 take it what we are saying is that we have no more 19 confidence about California and Texas and New Mexico and 20 Arizona and then the ones on the bottom than we do about all 21 the other ones, is that a fair statement? 22 A. No, I did not mean to say that, I don't think I 23 did say that. 24 I think what this chart shows is that for a large 25 majority of the states we don't even have any convincing 2510 1 evidence on the table that the census is wrong, never mind 2 that adjustment will make it better. 3 For some substantial states but still a minority, 4 taking the theory of adjustment at its face value, there is 5 quite reasonable evidence that the census was wrong. 6 Q. But was it on the basis of these numbers that the 7 Bureau calculated its loss function analyses? 8 A. I don't think that is quite right. What is true, 9 I think, is that some of the raw material for figure 1 on 10 page 6 feeds into the loss function analysis. However, you 11 need additional information, namely, information about the 12 biases in the dual system estimator which is not reflected 13 on this page at all. 14 Q. And there is one other difference, and that is 15 that you have changed the standard errors in these numbers 16 from what the Bureau calculated, is that correct? 17 A. Look, the Bureau in its press release of June 13 18 got the formula for the standard errors wrong. When they 19 went and did the loss function analysis, I believe they got 20 the formula right. 21 Q. All of the Bureau's loss function analyses 22 favored adjustment, isn't that correct? 23 A. On their face I think that's right. 24 Q. Okay. 25 Lit me ask you a couple of questions. 2511 1 A. If I may, I would like to amend a previous 2 answer. 3 It's not quite right to say that the Bureau made 4 a mistake in its press release of June 13, they were in a 5 big rush to get it out as far as I understand and they 6 deliberately took a shortcut. They knew what they were 7 doing, they knew they were taking a shortcut. 8 Q. I would like to show you what Mr. Sitcov 9 yesterday showed you as PX 686. 10 Do you have that up there? 11 A. I have it. 12 MR. SOLOMON: Your Honor, do you have it? 13 THE COURT: No. 14 MR. SOLOMON: You can have this one. 15 (Handing to the court) 16 (Pause) 17 BY MR. SOLOMON: 18 Q. You recall talking to Mr. Sitcov about this 19 yesterday? 20 A. I recall testifying about it. 21 Q. Okay. And this is the chart that Dr. Fisher 22 talked about, correct? 23 A. Right. 24 Q. And what you said was in trying to explain in a 25 rough and ready way was that if you assumed that the census 2512 1 was perfect, which you took just as a hypothetical, 2 according to this chart, an adjustment would be making all 3 of the states' shares -- and you assume -- let me start 4 again. 5 If you assume that the census was perfect so that 6 adjustment by definition is going to make all the states' 7 shares worse, you testified that a procedure that was 8 reflected in this chart would suggest that approximately 9 half of the states would be expected to improve. 10 Do you recall that generally? 11 A. I do. 12 Q. And you said that the reason why you thought what 13 you believed to be an error was made was that the people 14 calculating this ignored the variability in the dual system 15 estimator; correct? 16 A. That's right. 17 Q. Okay. 18 Now, you testified that you heard or you read 19 Professor Fisher's testimony. 20 Do you recall that he said that he based this 21 chart on a memorandum by the Bureau, the Spencer to Mulry 22 memorandum? Do you recall that? 23 A. Yes. 24 Q. Did you review that memorandum? 25 A. No. 2513 1 Q. You just reached a conclusion that they must have 2 ignored variability in the dual system estimator without 3 reading the memorandum that Professor Fisher said he relied 4 on, is that correct? 5 A. That's correct. I heard his testimony and I 6 relied on his testimony. 7 Q. Okay. 8 I am going to show you the memorandum. 9 MR. SOLOMON: May I? 10 THE COURT: Yes. 11 (Handing to the witness) 12 (Pause) 13 Q. Let me ask you to look at page B 7166. 14 A. Yes. 15 Q. And you see in the second to the last equation on 16 the page, it's the absolute value minus U is less than DS, 17 subscript minus T is less than the absolute value of U? Do 18 you see that? 19 A. Yes. 20 Q. I want you to assume as Professor Fisher said in 21 his testimony this is the memorandum that he was relying on, 22 you can tell, can't you, that if the census was perfect as 23 you suggested in your hypothetical, that that equation would 24 not improve any state, all you would have would be zeros? 25 A. I'm having a little trouble following the 2514 1 notation. 2 I think what you are saying is right. 3 Q. Okay. What the absolute value is less than 4 something is, again, less than the positive absolute value, 5 so it's the same absolute value, so if the census is 6 perfect, then the two hands clapping are zero and nothing 7 fits in between, right? 8 THE COURT: You did say the two hands clapping? 9 MR. SOLOMON: The two hands, the two Us on both 10 sides of this inequality. This is inequality A is less than 11 B is less than C and A and C are the minus whatever the 12 undercount is and plus whatever the undercount is and the 13 purpose is to see whether the estimate fits in between, and 14 as Professor Freedman said yesterday if you assume the 15 census is perfect, then, of course, there is zero undercount 16 and zero less than something is less than zero and you never 17 get anything. 18 I just want to confirm that that is basically 19 right. 20 A. Well, I'm not ready to commit, but I think it's 21 right. 22 Q. Okay. 23 Look at the next formula underneath that, in particular, the reference to V as in victory. 24 Do you see that? A. I do. 25 (Continued on the next page) 2515 1 Q. That notation that is being used here is a 2 reflection of the variability in the dual-system estimator, 3 isn't it? 4 A. Come on. The fact that it is in the formula 5 doesn't mean that you are properly taking it into account. 6 Q. Did you want to answer my question? Then you can 7 ask yourself questions on redirect. 8 It is in the formula, isn't it? 9 A. Yes. The standard deviation of the DSE is in the 10 formula. 11 Q. So if, as Professor Fisher said, he relied on 12 this, then it is incorrect to state that he didn't take the 13 variability of the dual-system estimator into account, is 14 that fair? 15 A. I don't think that's right. 16 Q. Let me ask you next to look at PX-698, which was 17 the other chart that Mr. Sitcov talked to you about. Do you 18 have that? With respect to this one, I believe you 19 testified to Mr. Sitcov that -- 20 MR. SITCOV: Your Honor, I think I am going to 21 have to move to have some of this stricken. Mr. Solomon has 22 been representing that Dr. Fisher testified about 23 Plaintiffs' Exhibit 711. It is our understanding that 711 24 has never been before the Court before. 25 MR. SOLOMON: 711? 2516 1 MR. SITCOV: Yes, 711. 2 MR. SOLOMON: I don't know what you are talking 3 about. What is 711? 4 MR. SITCOV: 711 is what you just handed us. 5 MR. SOLOMON: No, no. What I said, your Honor, 6 was, I asked this witness whether he had heard the 7 testimony. The witness was asked, what was the basis on 8 which he created this chart, PX-686. Professor Fisher at 9 transcript 1125, "It was a memorandum, Bruce Spencer to Mary 10 Mulry, generated sometime around the beginning of this year 11 or the end of last year, describes how that was done." 12 All I asked the witness to do was to assume that 13 that was the memorandum. I haven't offered it. 14 MR. SITCOV: I did not hear him to assume that 15 was the memorandum. I heard him to assume some things about 16 the memorandum. I certainly think that that sort of 17 questioning is quite unfair of this witness. Are they 18 offering this as the memorandum that Dr. Fisher relied on or 19 not? I think that the witness is entitled to know that, as 20 are we. 21 MR. SOLOMON: I am entitled on cross-examination, 22 I believe, your Honor, to ask the witness hypotheticals. I 23 can ask him to assume this. In the transcript Professor 24 Fisher says it's a memorandum, Spencer to Mulry, dated 25 sometime around the beginning of this year or the end of 2517 1 last year. And who knows, we may have a rebuttal case, and 2 who knows, Professor Fisher may in fact be able to connect 3 it up. Even if he can't, I am entitled to ask. 4 THE COURT: Dr. Freedman is blithely doing his 5 equations there, having fun. 6 Are you aware that we have no proof that PX-711 7 is, in fact, the document that supplied the data that led to 8 686? So your last set of questions was based on the 9 assumption that 711 contained the data. Are you aware of 10 that? 11 THE WITNESS: I am now. 12 THE COURT: Do you want to change anything you 13 said based on that? 14 THE WITNESS: I do not. 15 BY MR. SOLOMON: 16 Q. Will you look at PX-698. 17 THE COURT: "Rejected"? 18 MR. SOLOMON: Right. 19 Q. As to this, you told Mr. Sitcov that you believe 20 that Professor Fisher made a similar mistake in that he 21 didn't estimate the variability of the dual-system 22 estimator; is that basically what you said? 23 A. No. I think what I said, and I think I'm right, 24 and I will say it again today, what I think he did -- he 25 didn't do it. Whoever was computing these numbers for him, 2518 1 I think, failed to take into account the variability in the 2 dual-system estimator but, rather, took into account the 3 variability in the targets. 4 Q. You testified yesterday that you, on the basis of 5 two slightly different methods, succeeded in following his 6 recipe in getting the number that he came out with, correct? 7 A. I did. 8 Q. Tell us as precisely as you can recall what those 9 slightly different methods were. 10 A. Sure. I listened to his description of the 11 method, the critical ingredient being that you get some list 12 of numbers which he described and which I suppose I can 13 reconstruct, and you take the standard deviation of that 14 list. I asked Bob Fay to do that on the data files that are 15 the basis for this chart. He did, and he came up with a 16 number of something like 5.65, if I remember properly. It 17 was immaterially different from 5.68. 18 I also asked Bob to provide for me certain 19 summary data on these files, more specifically the 20 variance/covariance matrix for the targets, for the thousand 21 targets that are the basis for this calculation. 22 Then I computed by a theoretical method what the 23 results of computing the standard deviation on the list of a 24 thousand numbers was. I came up with the number which I 25 think was just under 560, in part, I think, because I am 2519 1 using a slightly different version of the data than Dr. 2 Fisher is. Or it may just be that when you transfer these 3 numbers from one machine to another they lose a little 4 accuracy. 5 Q. Did you produce to us the results of what Dr. Fay 6 did? 7 A. The only result I know of is the calculation of 8 the standard deviation. I am not sure I remember what it 9 is. 10 Q. Did you give that to us in the disk that you gave 11 to us? 12 A. No. 13 Q. You then said that you did your own calculation. 14 Did you give that to us? 15 A. I don't think so, no. 16 MR. SOLOMON: Your Honor, we would like both of 17 those produced under 705, asit formed the basis of an 18 opinion that he offered yesterday. We cannot check it. I 19 have not been hounding you much today on that subject. 20 Much. 21 THE COURT: It is never off my mind. 22 Mr. Sitcov, can you accommodate him? 23 MR. SITCOV: Well, that is certainly just 24 rebuttal testimony that he offered to Dr. Fisher. I think 25 we can accommodate him in any event. 2520 1 THE COURT: Thank you. 2 Q. On this chart that you are looking at, PX-698, 3 the source that is given for it is the final loss function 4 analysis. Do you see that, sir? 5 A. I do. 6 Q. I ask you to assume that that is correct. Is it 7 correct that the final loss function analysis did estimate 8 the variability in the dual-system estimator? 9 A. No, no. I think it is fairer to say that the 10 final loss function analysis took into account estimated 11 variability in the dual-system estimator. 12 Q. That is fair to say? 13 A. Oh, I think so, yes. 14 Q. That is what you believe based on the checking 15 that you have done that Dr. Fisher did not do, is that 16 correct? 17 A. No. It seems to me he is talking about here the 18 variability in the estimated risk difference. The estimated 19 risk difference has two major sources of variability in it, 20 one from variability in the dual-system estimator and one 21 from variability in the targets. 22 He took the variability of the targets into 23 account but not the variability in the dual-system 24 estimator. And he picked much the smaller of the two 25 sources of variance. That is my testimony. 2521 1 Q. Right. But the Bureau, in the final loss 2 function analysis, did it correctly, right? 3 A. That is a much closer call. My testimony 4 yesterday was that I thought they had made a small and not 5 material error in their calculation of the loss from 6 adjustment. 7 A much more serious question is whether they got 8 the variance/covariance matrix for the smoothed adjustment 9 factors right, and that is a question of a factor of 2 10 perhaps. And there is a further question as to whether they 11 got those state level biases right. Those I think are quite 12 serious questions. 13 Q. Thank you. But all I was asking was in the final 14 loss function analysis, the Bureau didn't make the mistake 15 that you believe Dr. Fisher made, correct? 16 A. Right. That's fair. 17 Q. If you are right, the source on this page is 18 wrong? 19 A. No, not at all. That is not my testimony at all. 20 My testimony is whoever made this chart up made a mistake in 21 using those numbers. They used somehow the wrong idea to 22 get started on the calculation. 23 The variance you want is in there, but the person 24 who did this didn't get at it for the purpose of computing 25 the standard deviation of the estimated risk difference. 2522 1 That is what I am trying to say. 2 Q. Thank you very much. 3 I want to turn to page 35 of your loss function 4 report. You testified yesterday -- 5 A. Okay. I am on page 35. 6 Q. You testified yesterday that you performed tests 7 of significance in lieu of a loss function analysis, 8 correct? 9 A. I don't know in lieu of. I think in addition to. 10 Q. All right. You talked with Mr. Sitcov about page 11 39 of this document, correct? 12 A. Yes. 13 Q. And the formula for what you have calculated on 14 page 39 appears on page 35, correct? 15 A. In part. There is a chi-square statistic there. 16 Then you have to do some minimization, which is discussed on 17 page 36 in order to get the numbers which are on page 39. 18 Q. The formula to calculate the chi-squared 19 statistic is on page 35, correct? 20 A. Right. 21 Q. I want to spend a minute to see whether we can 22 understand what you have done here. The X in the formula in 23 number 1, formula number 1, that is the dual-system 24 estimator of the undercount, correct? 25 A. Well, no, not exactly. What it is is it is the 2523 1 dual-system estimator for the share or the change in shares 2 for the states. 3 Q. The U with a little line down the left, the mu, 4 that is the assumed real undercount, correct? 5 A. No. It is the assumed real share or share 6 change. If you want me to sort of sharply distinguish 7 between shares and share changes, I will have to read the 8 paper and try to remember. 9 Q. No. In any event, it is the assumed real share 10 of the change in the undercount and the X is the estimate of 11 the share change in the undercount, correct? 12 A. X is the estimate and mu is the truth. It is 13 either a share or a share change. 14 Q. What you are telling here on page 35 is in the 15 first instance just assume that mu is equal to zero, that 16 is, that the census got all 51 shares correct, right? 17 A. Right. 18 Q. And on the basis of that you calculate a 19 chi-square statistic of 338, right? 20 A. Right. 21 Q. You then go on and you say that you are going to 22 change the formula. You say the census is better than 23 adjustment for area I when, and you then have a formula. 24 You then have a fortiori when, and you have another formula, 25 right? 2524 1 A. Right. 2 Q. The second formula says, in words, and correct me 3 if I am wrong -- you will -- the absolute value of mu is 4 less than the absolute value of X over 2, is that right? 5 A. Right. 6 Q. That, you think, always follows from equation 7 number 1, the absolute value of mu is less than the absolute 8 value of X minus mu, right? 9 A. No. Perhaps my Latin is terrible, but I thought 10 it was the other way around. That is, I thought that when 11 the absolute value of mu was less than the absolute value of 12 X over 2, the first equation followed, and then the census 13 was better. 14 Q. So the second equation does not always hold; you 15 agree with that, correct? 16 A. Oh, sure. 17 Q. Do you use the second equation in calculating the 18 chi-square statistics that you report four pages later? 19 A. I don't know if that is quite fair. I used the 20 logic of that sentence. Then to test the hypotheses to see 21 whether these things are holding or not holding, I make the 22 substitution discussed in the next line, where it says put 23 mu equals X over 2 in 1. 24 Q. When you put mu to equal X over 2, mu, we said 25 before, was the truth, right? That is a real nonvariable 2525 1 number, correct? 2 A. Right. 3 Q. X over 2 is a variable number, right? 4 A. That is true. 5 Q. What you are asking here is whether one variable 6 number is greater than another variable number over 2, is 7 that right? 8 A. No. What we are asking for is whether there is a 9 certain relationship holding between the parameters in the 10 problem, which is the mus, and the X in the problem, which 11 is the random data. And I think I even tried to write that 12 out algebraically on page 40. That is why technically 13 speaking this is a simultaneous confidence interval rather 14 than strictly speaking a hypothesis test. 15 Q. I would like to take an example where mu is equal 16 to X over 2. 17 MR. SOLOMON: Sorry. 18 Q. See if we can work through the formula. We can 19 do it quickly. Let me get something to write on. 20 I don't suppose you are going to offer to do 21 this, are you, Professor? 22 A. Sure. Just tell me what you want done, and I 23 will do it. 24 Q. That would be great. I would like to work 25 through one example of your mu equals X over 2. Just so the 2526 1 judge is clear, when you put mu equals X over 2, that is 2 what you are telling the Court to do in order to test for a 3 51-state analysis whether the adjusted is better than the 4 unadjusted, it is your test of significance, correct? 5 A. No, I don't think that's right. It is what I am 6 suggesting we do plus some additional minimization and 7 maximizeation, I forget how I set it up, to get the chi- 8 square tests listed on page 39. 9 Q. Okay. 10 A. Incidentally, I do think that X and mu are share 11 changes rather than shares. I believe it says that on page 12 29. 13 Q. What you are telling us is that what you came up 14 with on page 39 is derived from putting mu to equal X over 15 2? 16 A. True. 17 Q. I would like to go through an example of putting 18 mu to equal X over 2. 19 Equation number 1 can be written -- which is X 20 minus mu to the slash G to the minus 1, that is the same as 21 putting G underneath in the denominator, right? 22 A. Sort of. These are matrices, not numbers. So 23 you can't really put it in a denominator. But it is sort of 24 right. 25 Q. Can I put it in the denominator or not? 2527 1 A. No. 2 THE COURT: So you are going to do it anyway. 3 MR. SOLOMON: I want to know whether it is right. 4 Q. That is what we are doing. We are setting mu to 5 X-2. Where as in your first example you set mu to X-zero. 6 That is all we have done so far. 7 A. Right. 8 Q. We haven't done anything so far. 9 I want to replace each of the places are mu shows 10 up, I want to replace that with X over 2, right? I just 11 replace the mu with X over 2. Is that what I should do? 12 A. Hang on a second. I misspoke earlier. For page 13 39 you replace many of the mus about X over 2 and then do 14 some minimization over the rest. 15 If what you are talking about is the test that is 16 described on page 35, that is, where improvement occurs for 17 all 51 areas, then e yes, you put mu for X over 2 for all 51 18 areas. 19 Q. You are saying that this is used, at least in 20 part, in deriving what you did on page 39, which you showed 21 the judge yesterday, right? 22 A. Yes. It is the second line in the table. 23 Q. Just for a one-state example, if I can figure out 24 what this location should be, every time I see the mu, I am 25 going to replace it with X over 2. So I go mu equals X 2528 1 minus X over 2, X minus X over 2 over G, for a one-state 2 example. Is it accurate when I have put mu to equal X over 3 2 to rewrite your equation that way? 4 A. Yes. 5 Q. This is really just saying X minus a half of X, 6 right? 7 A. Right. 8 Q. That is a half X? 9 A. Right. 10 THE COURT: It's not the only one. 11 Q. This is also fairly rewritten half X? 12 A. So far we are in sync. I don't know what is 13 going to happen next, but so far it is fine. 14 Q. You don't have a problem rewriting that as X over 15 2 squared over G? I haven't done anything. All I have done 16 is grouped these together and squared them? 17 A. Right. 18 Q. Because the two of these I have multiplied 19 together and they are both the staple. 20 A. Right. 21 Q. I would fail your statistics course, wouldn't I? 22 A. No. You are doing great. 23 Q. We can rewrite this too, without, I think, 24 changing anything at all. Correct me if I am wrong, as X 25 squared over 4 over G? 2529 1 A. Yes, sure. 2 Q. That is equal to X squared over 4 G? 3 A. Right. 4 Q. So far? 5 A. Sure. 6 Q. I haven't done anything that is not in your 7 formula, correct? 8 A. Right. That is why you are dividing by 4. Do 9 you see the 338 over 4? That is where the 4 comes from. 10 Q. Absolute right. When you go from 338 -- you say 11 338 divided by 4, and that is 85? 12 A. Correct. 13 Q. What is G? 14 A. G is the variance/covariance matrix for the 15 estimated change in shares from the production deal system 16 estimator. 17 Q. G is the variance/covariance -- does anybody use 18 that notation? I just want to use that for variance/ 19 covariance. 20 A. It's fine for the moment. 21 Q. That is the variance/covariance matrix of X, 22 correct? 23 A. Right. 24 Q. What is the variance/covariance matrix of X over 25 2? 2530 1 A. It is one-quarter G. 2 Q. So the variance/covariance matrix of X over 2 is 3 one-quarter G. The reason for that is when I cut X in half, 4 the variance is reduced by a full quarter, correct? 5 A. Right. 6 Q. In your equation here, when I change mu to X over 7 2, you used the variance/covariance matrix of X, not the 8 variance/covariance matrix of X over 2, correct? 9 A. That's right, and I am not inclined to change. 10 Q. If we were to take the variance/covariance matrix 11 of X over 2, which is what you told me to put into the top 12 formula, instead of the variance/covariance matrix of X, 13 then what we got over here would be X squared over 4 G over 14 4, correct? 15 A. Well, it seems to me you have now made sort of 16 two substitutions of a factor of 2. If you want to do that, 17 yes, you are going to get X squared over -- 18 Q. I want you to agree with me. If I use the 19 variance/covariance matrix of X over 2 rather than the 20 variance/covariance matrix of X, then this is a proper way 21 to write the equation? 22 A. That is exactly what I am not prepared to give 23 you. It seems to me that the procedure I am following here 24 is the standard procedure for getting multiple comparison 25 intervals of this kind. In fact, there is a little 2531 1 algebraic demonstration on page 40 of the narrative of what 2 I am doing. What you are doing, I think, is you are 3 dividing by the 2 twice, if I am following you. 4 Q. Agree with me this far, that what I am doing is 5 replacing the variance/covariance matrix for X with what you 6 have told me is the variance/covariance matrix for half X. 7 A. I will certainly give you that the 8 variance/covariance matrix for X over 2 is G over 4. If you 9 want to substitute X over 2 for mu in the equation and G 10 over 4 for G in the equation, if you want to do that, you 11 will get G over 16, yes. 12 Q. This? 13 A. X prime G X over 16. 14 Q. I don't presume to correct you. I have written 15 this correctly, right? The G was four times too large if 16 you are talking about the G for X over 2 rather than the G 17 for X, right? So I need to divide this G by 4? 18 A. All I am prepared to give you at the moment is 19 that if you want the variance/covariance matrix for X over 20 2, it is G over 4. 21 Q. All right. 22 A. I am not yet prepared to go further than that. 23 Q. All right. So far so good. If we use G over 4 24 rather than G, then it is fair to take this equation, this 25 number, which we came up with before, X squared over 4 G and 2532 1 just divide it by 4, right? 2 A. I am prepared to make another concession, which 3 is that if you take my chi-square statistic and you, A, 4 replace mu by X over 2. 5 Q. Which is what you do. 6 A. Let me finish the concession, then you ask 7 another question. 8 Q. Okay. 9 A. What I am trying to concede is that if you will 10 replace mu by X over 2 in the chi-square statistic and then 11 replace G by the variance/covariance matrix for X over 2, 12 you will get a division by 16 instead of a division by 4. I 13 am even giving it to you in a multidimensional case. I am 14 not going an inch further than what I just conceded yet. 15 Q. I think it may be enough. For my purposes, it is 16 accurate simply to rewrite this by crossing these two things 17 out, isn't it? 18 A. Maybe I am making a terrible mistake in my 19 concessions. I thought you would get X squared over 16 G. 20 Q. Can we just agree that it is simply 21 mathematically, without attributing anything else to this, I 22 can rewrite this triple decker fraction as X squared over G? 23 A. Actually, you are right and I'm wrong. Let me 24 modify my concession. I'm terrible at doing algebra, 25 especially on the stand. 2533 1 If you replace the X by the X over 2, you will 2 get a factor of a quarter. Then if you replace the G by the 3 variance/covariance matrix for X over 2, G goes down by a 4 factor of 4, so G goes up by a factor of 4, and so it 5 cancels. You saw that and I didn't, and I apologize. 6 Q. That's right all right. We are now in agreement 7 that this X squared over G is the way that we would write it 8 if we replaced mu with X over 2 and correspondingly we took 9 the G not for X but the G for X over 2? Do you agree with 10 that? 11 A. I agree that if you made those two modifications 12 in the formula for chi-squared -- I am just following like a 13 little mechanical person here -- if you make those two 14 changes in the formula, you will get the answer that you 15 have written down. 16 Q. If we make the replacement of the G rather than 17 using it for X, using it for X over 2, using the G that 18 corresponds to X over 2, which is a quarter G you told us -- 19 if we do that, then am I correct that the chi-square 20 statistics that you calculate and use on page 39 are low by 21 a factor of 4? 22 A. What I am prepared to concede is that if you make 23 the changes that you seek to make in the formula, you will 24 multiply the chi-squares -- it may not be exactly 4, but 25 close enough. 2534 1 Q. The only change that I am making that you and I 2 may have a dispute about is that I am using the variance/ 3 covariance matrix associated with X over 2. We don't have 4 any other dispute, correct? 5 A. It seems to me we have a major dispute about how 6 to compute confidence intervals. Now that you have 7 corrected my mistake in arithmetic here, we do not have a 8 dispute about arithmetic. We have a dispute about how to 9 compute confidence intervals. 10 Q. You were mentioning before that if you agreed 11 with the substitutions that I suggested. I am only talking 12 about one change, not two. That is what I want to be clear 13 on with you. You made the first substitution because you 14 wanted mu to equal X over 2, correct? 15 A. No, I don't want mu to be equal to X over 2 16 particularly. What I want to do is test the hypotheses 17 which are listed on page 39. In my opinion, the appropriate 18 way to do that is to stick with the original G and put mu 19 equals X over 2 for some or many areas. 20 The demonstration of that, not that it needs much 21 demonstration, I think, is on page 4. 22 Q. If I am correct that by replacing X with X over 2 23 you should have replaced the G with the variance/covariance 24 matrix associated with X over 2 rather than X, I ask you 25 simply to assume that, then you will agree with me that the 2535 1 chi-square statistics on page 39 are all too low by a factor 2 of 4? 3 A. I am not sure that 4 is the right number. But 4 certainly it is close to 4. So let me try and say it again. 5 If I got the method for calculating confidence 6 intervals wrong, then all my chi-square statistics are 7 wrong. I just don't happen to think that. 8 Q. If they are wrong in the way that I say, then 9 they are all too low by a factor of 4 or close to 4, 10 correct? 11 A. That I will gladly concede. 12 Q. If the chi-square statistics on page 39 which you 13 went through with the judge yesterday are all too low by a 14 factor of 4 or roughly 4, then on the basis of what you tell 15 us on page 38, am I correct that each of the conclusions 16 that you draw is reversed? 17 A. Absolutely. If you should multiply all those 18 chi-square statistics by 4, which is a hypothetical you 19 require me to adopt, then, sure, all this evidence flips 20 over. 21 Q. It is your testimony that you believe it is 22 appropriate to use the variance/covariance matrix for X even 23 though what you are testing is X over 2, correct? 24 A. No. I think that is the fundamental problem. 25 What I am testing are the hypotheses listed on the lefthand 2536 1 side of the chart Table 17 on page 39. Substituting mu 2 equals X over 2 for some or many statesties only a step on 3 the road to making that test. 4 Q. In the test that we are talking about, even 5 though you have taken X, which is the estimate of the 6 undercount, and you have cut it in half, you still believe 7 it is appropriate to take the variance/covariance matrix for 8 the full X, not the X over 2; am I understanding you 9 correctly? 10 A. It is my opinion -- and of course everybody can 11 make mistakes, starting with me, but it is my opinion, 12 despite this cross-examination, that I have got the method 13 for computing those confidence intervals right. So I am 14 insisting that you should keep the original G and it is 15 appropriate to substitute mu equals X over 2 in that formula 16 for the purpose of testing the hypotheses in the lefthand 17 column of Table 17. That is my testimony. 18 Q. Thank you. Let me ask you one final series. It 19 goes something like this. 20 If X is an estimate of an undercount and I have a 21 variance estimate for that X of A, would you agree that if I 22 instead changed the X to X over 2, then mathematically you 23 can tell me that the variance for X over 2, rather than 24 being A, is going to be A over 4? 25 A. Sure. I gave you that several questions ago, and 2537 1 I will give it to you again. That is quite right. 2 MR. SOLOMON: I have nothing further. 3 THE COURT: Thank you. 4 MR. SOLOMON: I neglected to mark this. We would 5 offer PX-757, which is the one-page chart with the shares 6 that three states disimprove. We would offer it for 7 illustrative purposes only. 8 THE COURT: No problem? 757 is admitted for 9 illustrative purposes only. 10 (Plaintiffs' Exhibit 757 for identification was 11 received in evidence) 12 THE COURT: Any redirect? 13 MR. SITCOV: Could we take a short break? 14 THE COURT: Thank you. Let's do it. Five 15 minutes. 16 (Recess) 17 THE COURT: Mr. Sitcov? 18 MR. SITCOV: Thank you, your Honor. 19 REDIRECT EXAMINATION 20 BY MR. SITCOV: 21 Q. Do you see Exhibit 686 in front of you there? 22 A. Yes. 23 Q. Do you recall that Mr. Solomon asked you a couple 24 of questions about this chart? 25 A. Yes. 2538 1 Q. You testified that you think Dr. Fisher's mistake 2 was not taking the variability in the dual-system estimate 3 into account. Do you remember that? 4 A. Yes. 5 Q. If the variance from the dual-system estimate is 6 in the formula, how can you say that Dr. Fisher didn't take 7 variability in the due system estimate into account? 8 A. The formula that Mr. Solomon showed me is in 9 Plaintiffs' Exhibit 711 on page 7266. At the bottom of the 10 page, I can't quite read the subscripts, but there is an F 11 paren UV and V. E is, if I am following the notation, the 12 standard deviation of the dual-system estimator. That is in 13 the formula. But now we do not know U. U is the bias in 14 the census estimate. So knowing U is equivalent to knowing 15 truth. This is exactly what we do not know. 16 My understanding of Dr. Fisher's testimony is 17 that he simply substituted for U an estimate for U based on 18 the dual-system estimate, and he did not take into account 19 the variability in U and the imprecision in his estimate 20 based on the fact that the dual-system estimate for U is 21 only an estimate. 22 In fact, this paper talks about a bootstrap 23 adjustment to take care of that problem, although in my 24 opinion the bootstrap adjustment will be probably 25 unsuccessful. I have written on this topic, and the 2539 1 bootstrap is not a good way to adjust these kinds of biases. 2 Q. Do you have Plaintiffs' Exhibit 757 in front of 3 you? It was the one that Mr. Solomon handed you, Dr. 4 Freedman. 5 THE COURT: 757? 6 MR. SITCOV: 757. It is a single page. 7 THE COURT: The one you did the arithmetic on. 8 A. Got it. 9 Q. In your opinion, would you favor adjustment on 10 the basis of the at that time a in that table? 11 MR. SOLOMON: Objection. Asked and answered. 12 THE COURT: Overruled. 13 A. Yes, I would. 14 Q. Why? 15 A. We are making two states better, A and E, by 16 quite a margin. We are making B, C, and D a little bit 17 worse. However, the states that we are making worse in 18 terms of accuracy, namely, B, C, and D, we are giving them a 19 little more population, which is presumably in their 20 interest. 21 Finally, a feature that weighs heavily in my 22 mind, although it does not appear in any of the loss 23 function analyses that I have seen, is that the population 24 ratios, for example, the population of A to B -- let me just 25 do that one. In the original enumeration, the ratio of A to 2540 1 B in terms of population is two to one. Adjustment makes it 2 perfect. The most dramatic example, of course, is A to E. 3 In the original enumeration, which is like the census, the 4 ratio is infinite, 2 divided by zero. I have never before 5 testified about infinity. 6 The adjustment moves from a ratio of 2 to zero to 7 a ratio of 2 to 1, which is a lot closer to truth, which is 8 2 to 2. 9 So to me these data would favor adjustment. I 10 don't remember this hypothetical as particularly realistic. 11 We do know truth here, and we are adjusting an area which 12 has zero population. So at least in these two respects I 13 think this hypothetical is unrealistic. But if these were 14 the facts, I would favor adjustment on that basis. 15 Q. When you say "unrealistic," what do you mean 16 unrealistic in relation to? 17 A. They are different. As I see it, it is the 18 problem of trying to decide whether or not to adjust the 19 census of 1990 using the production dual-system estimates. 20 Q. Dr. Freedman, Mr. Solomon asked you some 21 questions about the formula that appears, I believe, on page 22 35 of Defendants' Exhibit 64. 23 A. Right. 24 Q. He took you through a number of permutations and 25 asked you as part of his questions to consider dividing G. 2541 1 Do you recall that? 2 A. Yes. 3 Q. You testified that you were unwilling to consider 4 dividing G? 5 A. I was willing to consider it, but I wasn't 6 willing to change my formula. 7 Q. Why are you not willing to divide G in your 8 formula on page 35? 9 A. What I am trying to get at by this technique is 10 the relationship between truth, represented by mu, and the 11 production dual-system estimator, represented by X. I am 12 trying to describe the joint properties of truth and the 13 estimate for it. 14 What Mr. Solomon seems to want to do is to try 15 and consider the relationship between truth or half of truth 16 and half the dual-system estimator. I am not interested in 17 half the dual-system estimator. I am interested in the 18 dual-system estimator itself. 19 MR. SITCOV: No further questions, your Honor. 20 THE COURT: Thank you. Can we excuse Dr. 21 Freedman? 22 MR. SOLOMON: Your Honor, we can. I want the 23 documents that we didn't have that inhibited 24 cross-examination. I don't think there is going to be a 25 need to see Dr. Freedman in this context anymore. Subject 2542 1 to that, I have nothing further. 2 THE COURT: Thank you very much, Dr. Freedman. 3 Most illuminating. 4 (Witness excused) 5 MR. SITCOV: Your Honor, at this time I would 6 like to move into evidence Defendants' Exhibit 67. It was 7 something that was discussed by Dr. Ericksen during his 8 cross-examination. It is in the binder of defendants' 9 exhibits. 10 THE COURT: 67 is entitled "Regression Models for 11 Adjusting the 1980 Census" by Freedman and Navidi. 12 MR. SITCOV: Yes. 13 MR. SOLOMON: We object. In relation to the 1980 14 census, Professor Freedman was on the stand. He could have 15 asked him any questions he wanted to about that. It is 16 hearsay. It is completely irrelevant. 17 MR. SITCOV: Dr. Ericksen testified rather 18 extensively about it on cross-examination, your Honor, and 19 identified, for example -- 20 THE COURT: I don't recall this. Why did you 21 wait till now to offer it? 22 MR. SITCOV: I think that I probably just forgot 23 to or it at the time, your Honor. 24 THE COURT: Ah. And it was brought out during 25 Dr. Ericksen's -- 2543 1 MR. SITCOV: Yes, it was. 2 THE COURT: He was the first witness. 3 MR. SITCOV: Yes, he was. I apologize for my 4 mistake. 5 MR. SOLOMON: Your Honor, it was brought out on 6 cross. Mr. Sitcov got whatever he wanted out of the cross- 7 examination. We have this document marked for 8 identification. There is not going to be any question about 9 what it is. But, respectfully, it should not be evidence. 10 THE COURT: I will admit it. Defendants' Exhibit 11 67 is admitted. 12 (Defendants' Exhibit 67 for identification was 13 received in evidence) 14 MR. SOLOMON: Just a point of clarification. It 15 is that document that includes comments, including that of 16 Professor Madansky? 17 MR. SITCOV: Yes. 18 MR. SOLOMON: You are offering the whole 19 document? 20 MR. SITCOV: Yes. 21 MR. SOLOMON: Thank you. 22 MR. SITCOV: Also, at this time, your Honor, 23 defendants wish to renew their motion to exclude certain 24 charts that were introduced through the testimony of 25 Franklin Fisher and to exclude his testimony with respect to 2544 1 those charts. I know your Honor has the dueling 2 declarations that refer to that. In our opinion, there 3 isn't any doubt but that plaintiffs had had everything they 4 needed in order to compute the charts as of the end of 5 February of this year. 6 I think perhaps one of the more striking 7 demonstrations of that, as we pointed out in our 8 declarations, is that we have received two different 9 versions of these charts from the plaintiffs. One we 10 received on the eve of trial. One we received at trial. 11 And there are certain rather significant differences in the 12 exhibits. 13 For example if you will look at 684, 685, 686, 14 687'. I'm sorry. I think your clerk has a copy of the 15 papers that we have submitted. 16 If your Honor would take a look at attachment A, 17 you will see Plaintiffs' Exhibit 684 and 686. If you look 18 at the bottom of those two documents, there is, I guess, 19 some sort of citation that simply says "PRODSE." 20 If you look at attachment B, which reproduces the 21 same -- well, those are supposed to be the same exhibits. 22 Now they have, under PRODSE, a number of data files 23 represented. You will find the same thing if you compare 24 the version of 686, that is at attachment A, with the 25 version of 686 in attachment B. 2545 1 Another perhaps even more interesting difference 2 would be comparing I think it is 696, the two versions of 3 696. If you compare the 696 that appears as attachment A 4 with the 696 that appears at attachment B, in the bold-face 5 type at the top of the large squared area, it says number of 6 standard deviations from zero, 5.89. That is the 696 that 7 we got on the eve of trial. If you look at the version of 8 696 that the plaintiffs submitted in court, you will see it 9 gives a rather different number. That number is 584. 10 THE COURT: 524? 11 MR. SITCOV: I'm sorry. 524. I take that back. 12 Again, if you look at the sourcing materials, the 13 698 that is at Attachment A simply says PRODSE final loss 14 function analyses. 696 that is in Attachment B, it lists I 15 think the same data files that are listed for the other 16 things in Attachment B. 17 We have seen two versions of these documents with 18 no explanation as to why we have seen two versions and 19 certainly no notice to us that we had two versions. Had 20 someone not actually gone through and pulled them out, we 21 would never have found that out. 22 The other thing, and I think it is certainly 23 significant, the 696 document that I was just referring to 24 in both versions it has the number 524 is a certain type of 25 statistic that is suggestive of a strongly estimated number. 2546 1 During Dr. Fisher's deposition, he testified about that 2 number and that he knew what that number was. So clearly 3 the plaintiffs had in their possession at this time, at the 4 time of his deposition, which was April 15th, 16th, and 5 17th, the data necessary to derive this number. 6 I think it is clear from the Kortakowsky and the 7 Lyon declarations and even, I would say, from the 8 declarations that were submitted by the plaintiffs, that all 9 the material that they needed to compute the data that 10 appears on these documents was made available to them before 11 the end of February of this year. 12 There is absolutely no reason why they can't have 13 done this in time enough for their witnesses to know about 14 it in depositions and for us to be able to inquire of them 15 during depositions on these topics. In fact, as I said, Dr. 16 Fisher apparently knew about what that T statistic on 696 17 was going to be, because he testified that the number was 18 about 5. 19 So it is simply incredible to me that the 20 plaintiffs might say that they didn't have the information 21 necessary to make these charts up. When you take that, your 22 Honor, in connection with the differences in the two 23 versions of these exhibits with no explanation as to the 24 difference, we find that quite problematic and certainly 25 prejudicial. 2547 1 THE COURT: Mr. Rifkind? 2 MR. RIFKIND: First let me say about the dueling 3 affidavits. They pass each other in the night. There is a 4 long discussion in the affidavit. But it doesn't address 5 the proposition asserted in the affidavit submitted by Brian 6 Palmer, which is attached to Mr. Bowker's affidavit, that 7 CRA was not able to replicate the final loss function 8 analysis as of the middle of March because we possessed 9 neither the programs used by the Bureau to perform the final 10 loss function analysis nor the algorithm employed by the 11 Bureau for that purpose. One or the other of those was 12 necessary to replicate the final loss function analysis. 13 Then Mr. Palmer goes on to describe how, by a 14 process of detective work, in April they found a document 15 which suggested a method by which they might replicate it, 16 cracked the code, developed their own version of it, which 17 later turned out to be identical to the Bureau's or 18 substantially identical to the Bureau's, and that therefore 19 the analysis of the loss function data could not have been 20 performed until sometime after, I believe it was, April 21 23rd, as stated in -- 22 THE COURT: I don't think I need much more 23 guidance on that point. What about the fact that you have 24 two sets of books here? 25 MR. RIFKIND: The charts, there are two types of 2548 1 changes. In the version we gave them on the 1st of May, in 2 the process of preparing these in white heat, having only 3 received the calculations at the very end of April, we 4 omitted to put on the chart, as we had intended to do, the 5 data sources, the data files, the Bureau data files from 6 which the calculations are derived. So for most of these 7 charts the difference is that we added one line there 8 identifying the file source. 9 That information was available to them on May 1. 10 Indeed, we asked them on May 1, is there anything else you 11 want to know? But we thought for clarity of preparation it 12 ought to be on the physical chart when we presented it in 13 court, and we added that. And we called that to their 14 attention some days before Dr. Fisher testified. 15 As Dr. Fisher also testified on the stand, on 16 direct, he found that someone had miscounted the number of 17 zeros in one of these calculations, missed a zero, and that 18 changed in a modest way the number of standard deviations 19 calculation and a few of the other calculations. The 20 difference between the chances of something being true are 21 one in 80 million or one in 3 trillion doesn't seem to me 22 exactly earthshaking. But on direct examination Dr. Fisher 23 pointed that out. They were free to cross-examine him at 24 the time. 25 THE COURT: Why do you need both sets of 2549 1 documents? 2 MR. RIFKIND: No. We only want the ones we 3 offered in evidence and were received in evidence, and those 4 were the corrected ones. They would have corrected it by 5 pen from the stand if we hadn't had it printed up in 6 advance. 7 THE COURT: I misunderstand what Mr. Sitcov was 8 saying. I thought he was saying both were being offered. 9 MR. RIFKIND: No. 10 MR. SITCOV: No. We got both. 11 MR. RIFKIND: We gave them one on May 1. Then 12 about three days before Dr. Fisher testified, we found the 13 error and SENT them that I think three days before. I think 14 Mr. Sitcov's indication of what Dr. Fisher testified to at 15 his deposition is not borne out by the deposition testimony. 16 Whether he had some expectation of what the values would be 17 or not, perhaps the deposition indicates. But the fact is 18 he hadn't done the calculation. That is only done following 19 the 23rd of April, and he couldn't have done it before that. 20 THE COURT: I think I am ready to rule. 21 MR. RIFKIND: I might add one last point on this. 22 The dramatic difference between the situation faced with 23 respect to Freedman and the situation faced with respect to 24 Dr. Fisher is quite clear. We still are waiting for the 25 data with respect to Dr. Freedman. 2550 1 MR. SITCOV: May I make just one more comment? 2 THE COURT: Yes. 3 MR. SITCOV: Dr. Fisher testified, and I would be 4 happy to go across the hall and get his deposition and read 5 it, Dr. Fisher testified that someone working on plaintiffs' 6 behalf had already as of April 15th of this year computed 7 the T statistic that appears in Plaintiffs' Exhibit 696. 8 That is simply impossible if they didn't have the data that 9 were required to produce those charts. 10 THE COURT: Thank you. 11 The plaintiffs have offered Exhibits 684, 686, 12 and 696 through and including 701. Defendants object to the 13 admission of all of the defendants. 14 There are basically two grounds. First, some of 15 the documents appear in duplicate and they are not the same. 16 That has been resolved by the plaintiffs offering only the 17 later of the two documents. So my ruling pertains only to 18 the admissibility of the later of the documents. 19 The other ground is that the plaintiffs and 20 defendants have a serious factual dispute as to whether the 21 calculations represented in these documents could have been 22 made before April 23. The defendants say yes, the 23 plaintiffs say no. Each has put in affidavits supporting 24 its side of the case. 25 I am obviously not in a good position to decide 2551 1 that issue of fact. Accordingly, I ought to err, if I am 2 going to, on the side of admissibility. I will therefore 3 admit all of those documents, giving appropriate 4 consideration to the arguments raised by the defendents that 5 the documents might have been compiled before April 23. 6 (Plaintiffs' Exhibits 684, 686, and 696 through 7 701 for identification were received in evidence) 8 THE COURT: Anything else? 9 MR. SOLOMON: Just one thing, your Honor. I 10 would like to mark that chart for identification purposes as 11 the next PX number and offer it merely as what we used 12 during the cross-examination. 13 THE COURT: I will admit it for that purpose. If 14 you can give us a number. 15 MR. RIFKIND: 767. 16 THE COURT: 767 is admitted for illustration 17 purposes as an adjunct to the cross-examination of Dr. 18 Freedman. 19 (Plaintiffs' Exhibit 767 for identification was 20 received in evidence) 21 MR. RIFKIND: I have a further housekeeping 22 matter. We have discussed putting in the record various 23 dispositions. I wonder whether this isn't the appropriate 24 time to do that. 25 THE COURT: Yes, I remember Mr. Cohen. I see him 2552 1 coming at me. Even if your name is Quinn. 2 MR. COHEN: I was suggesting to somebody this 3 morning, your Honor, that if this were the Army, I would be 4 the person handed the toothbrush to clean the floors. 5 MR. MILLET: Your Honor, I had understood from 6 our earlier side bar that this was going to be done after 7 the formal close of the proceedings. They have designated 8 additional transcript pages outside the schedule that we 9 have done pretrial. I haven't had a chance to look them up. 10 MR. COHEN: I don't plan to belabor this. I just 11 want to tell the Court what it is that we are submitting. 12 Obviously, defendants will have the time to -- 13 THE COURT: I wonder if it isn't more efficient 14 to do the whole thing on paper. 15 MR. MILLET: That is our suggestion. 16 MR. COHEN: It is essentially paper which I am 17 going to hand up to you, your Honor. 18 THE COURT: They haven't looked at everything. 19 Why don't we wait until they do and then we will get a 20 complete set of papers from each side, each of you with your 21 designations and cross-designations and objections. I will 22 look at them and issue an order telling which are in and 23 which are out. 24 MR. COHEN: That is fine. There is one other 25 housekeeping matter, your Honor. There has been a pending 2553 1 motion by the State of Oklahoma to bring witnesses. We 2 managed to resolve that. Oklahoma has provided us and 3 defendants affidavits. Both sides have agreed that were the 4 witnesses to appear, the affidavits represent what the 5 testimony would be. We don't, obviously, stipulate that the 6 facts are the truth. 7 I am authorized by counsel for Oklahoma to state 8 that they have withdrawn their motion seeking to present 9 live testimony. 10 THE COURT: I will accept the affidavits that you 11 have just handed up. 12 MR. RIFKIND: There is one further matter that 13 might be raised at this time. Mr. Segen for Hudson County 14 wishes to be heard. Perhaps we could deal with that next. 15 THE COURT: Mr. Segen. 16 MR. SEGEN: Good afternoon, Judge McLaughlin. I 17 would like to say that I find it somewhat apt that I should 18 close the proceedings this morning, but I am not sure that 19 that is the case. 20 When I was last here, the issue was the voting 21 rights act and which of the possibility of 175 different 22 methods that were brooded about to fill the gap in regard to 23 the evidence on the Voting Rights Act case would ultimately 24 be employed. I am glad to be able to tell you, your Honor, 25 that as a result of help from your law clerk Sean McGrath 2554 1 and Dave Goldii of the plaintiffs' committee and Mr. Millet 2 representing the government, that we have, in fact, opted 3 for a solution, albeit one that I would not have thought was 4 going to be the one to be presented. 5 It proceeds in a two-step approach, and it 6 proceeds from the basic fact that it is the government's 7 position that the Voting Rights Act does not, in fact, apply 8 to the federal government. The approach that the government 9 has accepted, defendants have accepted, that in the first 10 instance, if it is acceptable to the Court, the Court will 11 evaluate the validity of Hudson County's claim, legal claim, 12 that in fact the Voting Rights Act does apply to the 13 government. 14 If the Court decides that it does, Hudson County 15 will then proceed with affidavits from the appropriate 16 expert to fill in the gap in regard to the impact of using 17 adjusted statistics on minority districts. It is my 18 understanding that defendants would have full discovery 19 available to them in that regard. I believe they will be 20 making that application after I finish. 21 THE COURT: Discovery as to what? 22 MR. SEGEN: In regard to that affidavit, in 23 regard to that witness. 24 THE COURT: I don't understand what that means. 25 MR. MILLET: Your Honor, I think what we are 2555 1 basically talking about is something akin to summary 2 judgment motions. They will brief the legal issue. In they 3 think they need some facts to support it, they will put it 4 in an affidavit. We will oppose it if we think it is a 5 legal issue. If we want to oppose it about an opposing 6 affidavit, we will do it. If we think we need to take 7 discovery, we will do that. 8 THE COURT: I understand. All right. 9 MR. SEGEN: Your Honor, I just would add that at 10 this point Hudson County, in lieu of the affidavits, would 11 provide the Court with an offer of proof as to what in 12 general the affidavit would contain. If Hudson County were 13 to put on proof that would establish that the use of the 14 uncorrected figures rather than adjusted statistics results 15 in violations of the Voting Rights Act on the federal, 16 state, and municipal levels, and that the undercount of 17 minorities that results from this determination deprives a 18 geographically compact cohesive minority group that votes as 19 a block in a number of minority districts -- no, excuse me 20 -- that votes as a block: One, that there are violations in 21 a number of minority districts that would result in adding 22 minority districts; and, 2, also results in packing that 23 requires reconfiguration of a number of minority districts, 24 thereby releasing members of that compact cohesive minority 25 group from a packed minority district into another district. 2556 1 THE COURT: Thank you. 2 MR. SEGEN: That would occur in a number of 3 states. Thank you, your Honor. 4 THE COURT: Thank you, Mr. Segen. 5 MR. SITCOV: I do have one other thing, your 6 Honor. Since Mr. Solomon questioned Dr. Freedman upon 7 Defendants' Exhibit 63, we would like that to be admitted 8 into evidence. 9 MR. SOLOMON: What is 63? 10 THE COURT: 63 or 163? 11 MR. SITCOV: 63. 12 MR. SOLOMON: Your Honor, that is the same 13 document that you already ruled on. I questioned the 14 witness on cross-examination for the basis of the opinions 15 that he expressed concerning it. I didn't offer it. We 16 object and continue to object on all grounds. 17 THE COURT: I will exclude it. 18 MR. MILLET: If everybody else is done, we rest. 19 THE COURT: Very good. 20 MR. RIFKIND: Your Honor, if it is an appropriate 21 time to break for lunch, we would like to consider over 22 lunch what, if anything, we would propose to put on in the 23 way of rebuttal. 24 THE COURT: This afternoon? 25 MR. RIFKIND: This afternoon. 2557 1 THE COURT: Do you want to come back at two? 2 MR. RIFKIND: Could we have maybe just a little 3 bit longer? 4 THE COURT: Yes. 2:30, 4:30, pick it. 5 MR. RIFKIND: 2:30 will be terrific. 6 THE COURT: We will resume at 2:30. 7 (Luncheon recess) 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 2558 1 A F T E R N O O N S E S S I O N. 2 2:30 o'clock p.m. 3 MR. RIFKIND: Your Honor, we propose, if this is 4 satisfactory to your Honor, to call a witness this afternoon 5 in rebuttal, Dr. Rolph. 6 I did not, when we made our plans, appreciate the 7 court was going to be available tomorrow and I understand -- 8 THE COURT: I didn't plan to be, I'm just trying 9 to be accommodating. 10 MR. RIFKIND: I appreciate the accommodation. 11 Unfortunately, I told my experts that they could go away for 12 the weekend about three weeks ago and some of them did. 13 On the other hand, I'm hopeful to have a witness 14 here tomorrow, in fact, I'm hopeful to have Dr. Fisher back, 15 and what I advised Mr. Sitcov and your clerk is that I would 16 advise both of them tomorrow morning if we have succeeded in 17 getting Dr. Fisher back, and in that case we would propose 18 to proceed as soon as you are available tomorrow afternoon. 19 THE COURT: All right. 20 THE COURT: Is that all for today? 21 MR. RIFKIND: Dr. Rolph will be examined by Mr. 22 Sherman. 23 MR. SITCOV: I wonder if we can't find out today 24 if Dr. Fisher is available tomorrow? 25 THE COURT: When will you be able to determine 2559 1 today whether Dr. Fisher will be available tomorrow? 2 MR. RIFKIND: There are really two questions 3 related, one, is he available and can I get him down here, 4 and I hope to know that answer this evening. 5 The second question is whether after we review 6 the transcript of the testimony of yesterday afternoon and 7 this morning it is worth your Honor's while and ours to put 8 him on. 9 THE COURT: And that, I presume, will take until 10 at the earliest sometime tonight? 11 MR. RIFKIND: That will take some time tonight 12 and we get the transcript in the early evening and it will 13 NOT be UNTIL tomorrow morning. 14 I am not going to do it if it is A cumulative 15 things, your Honor heard plenty. 16 MR. SITCOV: We would like to know. 17 THE COURT: As soon as they decide, give them a 18 telephone number and they will call you. 19 MR. SITCOV: Great. 20 THE COURT: Dr. Rolph, come on up here. 21 JOHN ROLPH, 22 called as a witness in rebuttal by the plaintiff, 23 was examined and testified as follows: 24 THE COURT: I won't bother to swear you again, I 25 think the original oath still holds. 2560 1 THE WITNESS: Okay. 2 Mr. Sherman. 3 MR. SHERMAN: Good afternoon, your Honor. 4 DIRECT EXAMINATION 5 BY MR. SHERMAN: 6 Q. Dr. Rolph, were you in court yesterday and today 7 for Dr. Freedman's testimony? 8 A. Yes, I was. 9 Q. And have you read his two papers on smoothing? 10 A. Yes, I have. 11 Q. Did you hear Dr. Freedman testifying that 12 modeling of variance is not a standard procedure? 13 A. Yes, I did. 14 Q. Do you know if the Bureau of the Census models 15 variances in other areas? 16 A. Yes, I believe they do. 17 Q. Did you hear Dr. Freedman testifying about 18 benchmarking or ratio adjustment? 19 A. Yes, I did. 20 Q. Do you know if the Bureau of the Census uses 21 ratio adjustment or benchmarking in other contexts? 22 A. Yes, they do. Like modeling variances, I think 23 it is routinely used, for example, in the current population 24 survey. 25 MR. SHERMAN: May I approach the witness, your 2561 1 Honor? 2 THE COURT: Yes. 3 Q. Dr. Rolph, I have handed you what I have marked 4 as Plaintiff's Exhibit 763. 5 Can you tell the court what this is? 6 A. Yes. This is one of the reports from the current 7 population survey, one of the recent ones, actually. It's 8 entitled, "Poverty in the United States: 1990," series P 60 9 No. 175. 10 Q. Is this a document from the Census Bureau? 11 A. Yes, it is. 12 Q. And does this document reflect modeling of 13 variances? 14 A. Yes, it does. It has a section later on in the 15 document I believe on what looks to be page 208, which gives 16 a formula which indicates that there is what amounts to a 17 linear combination of two terms used in modeling variances. 18 Q. Does this document indicate that the Bureau did 19 ratio adjustment in conjunction with the modeling of 20 variances? 21 A. Yes, it also indicates that. 22 MR. SHERMAN: Your Honor, we would move 23 Plaintiff's Exhibit 763. 24 MR. SITCOV: No objection. 25 THE COURT: 763 is admitted. 2562 1 (Plaintiff's Exhibit 763 marked for 2 identification was received in evidence.) 3 BY MR. SHERMAN: 4 Q. I'm sorry, Dr. Rolph, is there a difference 5 between 763 and 764? 6 A. Yes. 764 is another example of the use of 7 modeling variances and ratio adjustment for the current 8 population survey. It's titled, "Household and family 9 characteristics: March 1991." 10 It also, as did the previous document, indicates 11 that variances are modeled and ratio adjustment is used. 12 Q. And is it a Bureau of the Census document? 13 A. Yes, it is put out by the Bureau of the Census. 14 MR. SHERMAN: Your Honor, we would move 764. 15 MR. SITCOV: No objection. 16 THE COURT: 764 is admitted. 17 (Plaintiff's Exhibit 764 marked for 18 identification was received in evidence.) 19 BY MR. SHERMAN: 20 Q. Have you had a chance to look at that, Dr. Rolph? 21 A. Yes, I have. 22 Q. I have handed you what has been marked as 23 Plaintiff's Exhibit 768. 24 Will you tell the court what that is? 25 A. Yes. This is the underlying, if you will, 2563 1 technical -- I don't want to call it an appendix, but a 2 description of how to do ratio adjustment. 3 As you can see from the front of it, it was put 4 out some time ago in 1963. Its technical paper number 7 5 entitled, "The current population survey, a report on 6 methodology," and the portion that is in the exhibit is part 7 7 entitled, "Preparation of estimates," and it describes in 8 some detail how to do ratio estimation and it's the method 9 that is used for the current population survey. 10 Q. Does it also involve modeling of variances? 11 A. This particular exhibit doesn't. I believe there 12 is another chapter in the same report that does modeling of 13 variances. 14 MR. SHERMAN: Your Honor, we would move 768. 15 MR. SITCOV: No objection. 16 THE COURT: 768 is admitted. 17 (Plaintiff's Exhibit 768 marked for 18 identification was received in evidence.) 19 BY MR. SHERMAN: 20 Q. Dr. Rolph, I have handed you what are marked as 21 Plaintiff's Exhibits 769 and 770. 22 Will you tell the court what these are? 23 A. Yes. These are two Bureau of the Census 24 publications, or I beg your pardon, these are two chapters 25 in the Bureau of The census publication entitled, "The 2564 1 current population survey, design and methodology." 2 769 is chapter 8, 770 is chapter 5. 3 Chapter 8, which is 769, describes modeling of 4 the variances in the presence of a replication method that 5 is very much like the jackknife that we talked about 6 earlier. 7 770 lays out in more detail the use of ratio 8 estimates in the current population survey and is basically 9 the methodological description of how it is done. 10 Q. Dr. Rolph, we have used the terms ratio 11 adjustment and benchmarking interchangeably. 12 Are they the same thing? 13 A. As I understand the way Dr. Freedman used them in 14 his testimony, he used benchmarking as a synonym for ratio 15 adjustment, he also used the term, I believe, scaling 16 variances as a synonym for ratio adjustment of the 17 variances. 18 Q. Do the terms presmoothing and modeling variances, 19 to your knowledge, refer to the same thing? 20 A. Yes. Modeling variances would be the standard 21 usage in the profession. I think presmoothing has become to 22 some use standard usage in this case. 23 MR. SHERMAN: Your Honor, at this time plaintiffs 24 would move 769 and 770. 25 MR. SITCOV: No objection. 2565 1 THE COURT: Both are admitted. 2 (Plaintiff's Exhibits 769 and 770, respectively, 3 marked for identification were received in evidence.) 4 BY MR. SHERMAN: 5 Q. Dr. Rolph, did you hear Dr. Freedman testify that 6 the assumptions underlying the smoothing model do not hold 7 precisely? 8 A. Yes, I did. 9 Q. I would like to go through the assumptions that 10 he talked about in his testimony. 11 The first one was about bias in the dual system 12 estimation, correct? 13 A. Yes, that was one of his assumptions. 14 Q. And what did Dr. Freedman say about the 15 assumption about -- first of all, what is the Bureau's 16 assumption underlying the smoothing model about bias in dual 17 system estimation? 18 A. The literal assumption is that there is no bias 19 in the dual system estimate. 20 Q. And what did Dr. Freedman testify to about that 21 assumption? 22 A. He indicated, as I recall, that he believed that 23 it was incorrect. 24 Q. And, Dr. Rolph, is there bias in the dual system 25 estimation? 2566 1 A. Yes, I believe there is. 2 Q. How do you know that? 3 A. Well, the Census Bureau studied this problem and, 4 indeed, I believe it's P16, if I remember the numbering 5 correctly, and they did a study in their total error model 6 looking at, among other things, the bias of the dual system 7 estimate of the raw dual system estimate as well as the 8 smoothed dual system estimate. 9 Q. Were the results of that study available to the 10 undercount steering committee last June? 11 A. Yes, my understanding is that they were available 12 and they were discussed. 13 Q. And so did the undercount steering committee last 14 June know that there was some departure from that 15 assumption? 16 MR. SITCOV: I object. 17 THE COURT: Overruled. 18 A. Yes, according to my reading of the 19 documentation, they did. 20 Q. And whatever Dr. Freedman's opinion about the 21 effect of bias in that assumption, the undercount steering 22 committee did not reject the smoothing model because of it, 23 did they? 24 A. No. Indeed, if I am recalling correctly, the 25 estimates from the total error model of bias in the dual 2567 1 system estimate I believe on net was slightly smaller when 2 it was smoothed as opposed to when it was not smoothed, but 3 there was some variation, depending on which evaluation 4 post-strata you looked at. 5 Q. Did Dr. Freedman's testimony provide any new 6 evidence about bias in the dual system estimate? 7 A. Not as I heard it, no. 8 Q. Did his papers provide any new evidence? 9 A. Not as I read them, no. 10 Q. Do you remember the second assumption that Dr. 11 Freedman talked about with regard to the smoothing model? 12 A. The famous sigma squared I and V 3? Yes, I do. 13 Q. Famous might not be the word used in some 14 quarters. 15 A. Infamous? 16 Q. Is it true, Dr. Rolph, that that equation meant 17 that the Bureau's smoothing model took no account of 18 correlations between post-strata? 19 A. Not exactly. It was an assumption about the 20 relationship of the true underlying adjustment factor to 21 this regression. The Bureau's model also included sampling 22 error and in the sampling error they allowed for correlation 23 across post-strata. 24 Q. So another point in the process they did take 25 into account correlations between post-strata? 2568 1 A. That's correct, in the sampling error. 2 Q. So it was just in this particular point in the 3 process, the equation which represents the variance of truth 4 around regression surface, that the Bureau chose not to 5 build in correlations between post-strata? 6 A. I think that's an accurate statement, yes. 7 Q. Are you familiar with what you call Dr. 8 Freedman's famous alternative? 9 A. Yes, I believe I was asked about it at my 10 deposition. 11 Q. Do you know why the Bureau chose to use sigma 12 squared I? 13 A. Well, my understanding is that they did several 14 studies looking at departures from the sigma squared I 15 assumption both in analyzing the 1988 dress rehearsal data 16 as well as in the 1990 census. 17 Q. Is it your view that sigma squared I is a more 18 appropriate term to use than Dr. Freedman's alternative? 19 A. Yes. I base that on really two pieces of 20 information: 21 The first one is that the Bureau did several 22 studies looking at sigma squared I versus other assumptions, 23 some of which actually were fairly close to the V 3 24 assumption, and, secondly, I find the V 3 assumption on its 25 face somewhat unreasonable for the following reason: 2569 1 Basically, the idea behind smoothing is to try to 2 reduce the effect of sampling error. The V 3 assumption 3 will take a post-strata which had a very small sampling 4 error and move it towards the regression surface by the same 5 amount as a post-strata that has an enormous sampling error, 6 and that strikes me as not being a terribly reasonable thing 7 to do statistically. 8 Q. Did you hear Dr. Freedman testify that the Bureau 9 did not consider the extent to which the assumptions of that 10 equation held? 11 A. I recollect him testifying along those lines, 12 yes. 13 Q. Is it correct in your view that the Bureau did 14 not consider the extent to which that assumption held? 15 A. No. As I indicated, I think they did a number of 16 studies looking specifically at that assumption. 17 MR. SHERMAN: May I approach the witness, your 18 Honor? 19 THE COURT: Sure. 20 (Handing to the witness) 21 (Pause) 22 BY MR. SHERMAN: 23 Q. Are you familiar with this document, Dr. Rolph? 24 A. Yes, I am. 25 Q. Can you describe to the court what it is? 2570 1 A. Yes. It's entitled, "Memorandum for the 2 technical design and estimation committee," dated January 2, 3 1990 from Greg Diffendal, and this memorandum is a brief 4 description of some work done by Cary Isaki and done by Bill 5 Bell comparing the sigma squared I assumption to two other 6 assumptions. 7 Q. Without getting too technical, can you tell us 8 what the other assumptions are? 9 A. As I understand them from this document and 10 others, one of them was using rather than the identity 11 matrix using the correlation, the sampling correlation 12 matrix, and the other one was using instead of V 3 what is 13 demoted V zero, which is I guess I would call it a slight 14 variant of V 3 is probably the simplest way to say it. 15 Q. Does this document represent a test of the 16 assumption represented by sigma squared I? 17 A. That's my understanding of the work that was done 18 there. 19 MR. SHERMAN: Your Honor, we would move 20 Plaintiff's Exhibit 70. 21 MR. SITCOV: I object. I move to strike. This 22 is all hearsay. We can't rely on the information in here. 23 He has no knowledge whatsoever of these things. He hasn't 24 testified that he knows about these things, he is just 25 telling what he thinks based on reading this. 2571 1 This is not his testimony about what the Bureau 2 did. 3 THE COURT: Mr. Sherman? 4 MR. SHERMAN: I am unclear whether Mr. Sitcov is 5 objecting to the document, which has a Bates stamp which 6 appears to indicate that it is in the administrative record. 7 Regardless of that, we would say it should be 8 admitted for the Bureau's state of mind. 9 It contradicts the testimony of one of 10 defendants' experts. 11 MR. SITCOV: The Bureau certainly doesn't have a 12 state of mind. 13 THE COURT: That is very clear. 14 MR. SITCOV: If the document is in the 15 administrative record that is one thing, but -- 16 THE COURT: Is it? Is that -- 17 MR. SITCOV: I can't tell if it's ours or not. 18 There are several different Bates stampes I have seen 19 bandied about, and I can't tell you whether this particular 20 one is ours. 21 THE COURT: I assume it is. I will take it in on 22 that basis. If it is wrong, you let my know later on. 23 MR. SITCOV: I would still like to strike the 24 testimony of this witness as to what the Bureau is doing. 25 THE COURT: The testimony can stand as he simply 2572 1 interpreted this document. Without the documents it can't 2 come in, but with the document it's all right. 3 (Plaintiff's Exhibit 70 marked for identification 4 was received in evidence.) 5 BY MR. SHERMAN: 6 Q. Dr. Rolph, I have handed you PX 87. 7 Would you describe to the court what this is? 8 A. Yes. It's entitled at the top, "A draft, C.T. 9 Isaki, 11/15/90." The title of the document is, "Smoothed 10 factors using sigma squared V sub zero." 11 THE COURT: Zero? 12 THE WITNESS: Zero. That's my interpretation of 13 the blob on my Xerox. 14 Q. What does this document represent? 15 A. This represents, again, testing a departure from 16 the sigma squared I assumption and replacing it with sigma 17 squared V sub zero, and it, again, is another example, but 18 it's somewhat greater length of the document we were just 19 discussing, in the Bureau's analysis of alternative 20 assumptions. 21 Q. Is V sub zero similar to V sub 3? 22 A. Yes. They define it toward the end of the first 23 paragraph. It states here, "V sub zero is V sub 3, but 24 absent the P-sample component of V sub 3." 25 Q. In your view, does this document show that the 2573 1 Bureau considered alternatives to sigma squared I? 2 A. Yes, that's exactly what it shows. 3 MR. SHERMAN: Your Honor, we would move PX 87 at 4 this point. 5 MR. SITCOV: I will object. It says draft on it. 6 It doesn't have a Bates stamp number. 7 THE COURT: Isaki is who? 8 MR. SITCOV: Isaki is an employee of the Census 9 Bureau. 10 MR. SHERMAN: In charge of the smoothing 11 operation, as I understand it, one of the individuals in 12 charge. 13 MR. SITCOV: I don't know if that's true, either, 14 your Honor. 15 THE COURT: Pardon me? 16 MR. SITCOV: I say I don't know that that is 17 true, that he is one of the individuals in charge of the 18 smoothing operation. It says a draft. 19 MR. SHERMAN: It is a draft, but, again, it goes 20 to the proposition of alternatives of sigma squared I at the 21 Census Bureau. 22 THE COURT: I will accept that on that basis. 87 23 is admitted. 24 (Plaintiff's Exhibit 87 marked for identification 25 was received in evidence.) 2574 1 BY MR. SHERMAN: 2 Q. I have handed you what is marked as PX 92, Dr. 3 Rolph. 4 Can you describe to the court what this is? 5 A. Yes. This is another memorandum dated December 6 4, 1990. It's to Cary Isaki from Bill Bell entitled, "Error 7 model selection for regression smoothing," and as I 8 understand this document -- 9 THE COURT: Who is Bill Bell? 10 THE WITNESS: Bill Bell is a statistician who is 11 employed by the Census Bureau and was involved in a number 12 of these statistical analyses, as I understand it. 13 I know he is employed by the Census Bureau, I 14 understand from the document. 15 Q. In your view, does this document represent the 16 Census Bureau's test of the assumption represented by sigma 17 squared I? 18 A. Yes. As I read the document, this is a case 19 where they looked at several different alternatives to sigma 20 squared I, and there is a discussion at the very end, I call 21 your attention to the last, next to last sentence -- or the 22 first sentence of the last paragraph, which says, "Based on 23 your work to date, I would conclude that S equals V 3 and T 24 equals sigma squared I are the best choice you got right 25 now." 2575 1 Q. And what does that indicate to you? 2 A. That indicates to me that they have looked at 3 several alternatives to sigma squared I and after doing some 4 analyses have concluded that it is the best assumption in 5 terms of producing the best estimates. 6 MR. SHERMAN: Your Honor, we would move PX 92. 7 MR. SITCOV: I would object. 8 THE COURT: 92 is admitted. 9 (Plaintiff's Exhibit 92 marked for identification 10 was received in evidence.) 11 BY MR. SHERMAN: 12 Q. Dr. Rolph, did you hear Dr. Freedman's testimony 13 about the work in his smoothing papers? 14 A. Yes, I did. 15 Q. Did it change your opinion about the results of 16 the Bureau's smoothing operation? 17 A. No, it did not. 18 Q. Why not? 19 A. Well, basically, three reasons, really: 20 The first one is that, as I listened to Dr. 21 Freedman and read his papers, it was essentially an argument 22 by analogy. By that I mean he created his own models which 23 represented some changes from the Bureau's models and then 24 within those models tried a number of different things and 25 then concluded because there were differences in those 2576 1 models those differences would carry over in the real world, 2 the Bureau real world, and just certainly based on his 3 testimony and reading his papers I don't find that analogy 4 sufficiently persuasive to persuade me that those 5 relationships hold in the real world. 6 The second reason is that Dr. Freedman looked at 7 a number of different pieces, if you will, of the smoothing 8 and the presmoothing operation and asked the question what 9 would happen if you eliminated this piece or you eliminated 10 these two pieces and made comparison across those, and that 11 seemed to me to be rather like asking if you were driving a 12 car down the road whether you took one of the spark plugs 13 out or took a wheel off. 14 I mean, basically, we are taking something that 15 was designed to go together and taking something out of it 16 and asking how well it worked, and that was not something I 17 found terribly persuasive in terms of criticizing the 18 procedures. 19 And the last, but not least, the comparisons he 20 made were by and large comparisons of rather larger 21 geographical areas, region would be an example, and the 22 smoothing model was really designed to reduce sampling 23 error, and in particular for smaller groups of the 24 population, and specifically to the extent smoothing works 25 or doesn't work, you got to look at smaller areas and if you 2577 1 look at these larger areas, particularly if you eliminate 2 the step that the Bureau took to try to make them agree, 3 ratio adjusting or benchmarking, it was not terribly 4 surprising to me that you would see some differences. 5 So that I didn't find that to be in some sense 6 information that would change my mind. 7 Q. I would like to go back to the first of the 8 points that you mentioned, Dr. Freeman's changing of the 9 Bureau's model. 10 If we could, I would like to walk through the 11 steps of the smoothing model. 12 A. Okay. 13 Q. Let's start with the functional form. What was 14 the Bureau's functional form for the smoothing model? 15 A. Okay. We are talking now sort of two steps, 16 first looking at the variances and then the raw adjustment 17 factors, so let's talk about the variances. 18 Q. Very well. 19 A. As I think was brought out in Dr. Freedman's 20 testimony -- let me start with the Bureau and then talk 21 about Dr. Freedman so we can keep things straight hopefully. 22 What the Bureau did was to use a functional form 23 to model the variance which, let me use the shorthand, did 24 not involve logarithms. 25 Dr. Freedman's model used, in one case, a model 2578 1 which involved logarithms, in another case a model which did 2 not involve logarithms, so that was one difference. 3 Q. What about the choosing of the variables for the 4 regression in the smoothing of variances? 5 A. Okay. 6 The Bureau, as I understood it from the 7 documentation, developed a candidate set of variables for 8 the regression for modeling the variances and then selected 9 those that had T values greater than 2. 10 Q. Do you remember what Dr. Freedman said about the 11 Bureau's method? 12 A. Memory isn't my strongest point, but my 13 recollection was he said he had a conversation I think with 14 Cary Isaki in which Dr. Isaki said something along the lines 15 of -- I can't quote him, but there wasn't a systematic 16 study devoted to selecting those factors. I don't recall 17 Dr. Freedman's words exactly. 18 Q. And what did Dr. Freedman do in selecting 19 variables in presmoothing? 20 A. A couple of different things. 21 In one of his models he used the same variables 22 that he had used in his regression model for the adjustment 23 factors themselves, for others of his models he used a set 24 of variables that the Bureau used. 25 Q. So some of his models weren't the same as the 2579 1 Bureau's? 2 A. The same as the Bureau's model. 3 Q. What about in selecting outliers in modeling of 4 variances, what did the Bureau do? 5 A. What the Bureau did was to label as an outlier a 6 point that was more than four standard errors away from the 7 regression surface. 8 What I understand Dr. Freedman to have done is in 9 some of his models not tried to identify outliers at all, in 10 others as I understood him he used the same post-strata as 11 the Bureau did. 12 Q. What about in ratio adjustment of the variances? 13 A. Again, what the Bureau did is they ratio adjusted 14 the variances. 15 I guess it's probably worth in hopes of the 16 extent we can clarify all this, Dr. Freeman's term for ratio 17 adjustment as indicated was benchmarking and I think in his 18 first paper he called it benchmarking the variances, I think 19 in the second paper, the thicker of the two, he calls it 20 scaling the variances, if I'm recalling correctly. 21 The Bureau did ratio adjusted variances. As I 22 understand it, Dr. Freedman did not some of the time and did 23 some of the time, so that would be the modeling of the 24 variances in the presmoothing. 25 Q. What about in estimating the covariance matrix? 2580 1 A. What the Bureau did to estimate the covariance 2 matrix was to model the variances or presmooth them as we 3 have discussed before and then estimate the covariance 4 matrix by multiplying those modeled variances times the 5 correlation matrix. 6 Q. What did Dr. Freedman do? 7 A. Dr. Freedman did that some of the time and didn't 8 do it some of the time. 9 Q. Dr. Rolph, moving on to smoothing the adjustment 10 factors themselves, how did the Bureau choose the variables 11 to be used in the regression equation? 12 A. For the variables themselves, the Bureau 13 developed two lists, basically. 14 The first list consisted of a must variables, 15 which were things like age, sex, race and so forth which 16 were designed to preserve the demographic information in the 17 adjustment factors. 18 The second list was what they called the 19 candidate list, and this consisted of a list that came out 20 of their several decades of experience in modeling the 21 undercount, studying the undercount as well as the dress 22 rehersal and TARO data. They then used the best subsets 23 technique to select from the candidate list to get a 24 different equation for each of the four regions. 25 Q. What is best subsets, Dr. Rolph? 2581 1 A. It's a method of choosing amongst -- choosing 2 from a group of variables to get one that fits the best as 3 defined in a suitable way. 4 Q. And what is Mallows C P? 5 A. Mallows C sub P is a particular method for 6 evaluating the fit of a regression equation. 7 Q. So is it fair to say it is part of the best 8 subsets regression? 9 A. My terminology would be it would be a method 10 which you could use in conjunction to do a type of best 11 subsets regression. 12 Q. Did Dr. Freedman use the same technique with the 13 Bureau in choosing his variables? 14 A. As I understand it, no. For some of his models 15 he took the Bureau's list of candidate variables, part of 16 that list, and then used a different method of selecting 17 from it, the method being if the T ratio was greater than 2, 18 I think he did a variant where the T ratio was greater than 19 one a half if I recall correctly, for some of his models, 20 for others -- I'm having a hard time keeping it straight -- 21 on others I think he did use the same set of variables, but 22 I'm a little unclear on that. 23 Q. In any case, did he use the same set of variables 24 in the same selection technique? 25 A. No. 2582 1 Q. With regard to the error structure used in 2 fittings the regression, what did the Bureau use? 3 A. The Bureau used what is called, pardon the term 4 of art, generalized least squareds with the relevant term 5 being our famous sigma squared I plus V 3. 6 As I understood Dr. Freedman, he used -- and 7 that's the appropriate thing to use for the model the Bureau 8 fit. 9 But what Dr. Freedman did, even with the same 10 basic error structure, used a different generalized least 11 squared formula which was not the appropriate one strictly 12 speaking for that model. 13 Q. What was the symbol for that model? 14 A. I think it was the symbol K and what is missing 15 is our friend sigma squared I. 16 Q. In identifying outliers, Dr. Rolph, what did the 17 Bureau do? 18 A. The Bureau defined as an outlier in the 19 adjustment regression points that were more than three 20 standard errors away from the regression surface. 21 Q. Is that what Dr. Freedman did? 22 A. In some of his models, as I understood it, he did 23 not do anything about outliers, in others he did. 24 Q. What did the Bureau use as an estimation method? 25 A. The Bureau used a procedure called maximum 2583 1 likelihood estimation, and Professor Freedman used for some 2 of his models, he used a method called the method of moments 3 and for others of his models he used a maximum likelihood 4 method, although I was not totally clear from the 5 documentation whether it was exactly the same one as the 6 Bureau. 7 Q. Is there a difference between those two? 8 A. Yes, I would describe method of moments, and I've 9 used it myself, not to knock it at all, as more of a quick 10 and dirty method and maximum likelihood as something which 11 you want to do when you want to all the bells and the 12 whistles. 13 Q. What about ratio adjustment of the adjustment 14 factors? 15 A. The Bureau did ratio adjustment of the adjustment 16 factors. Professor Freedman did it in some of his models 17 and not in some of his other models. 18 Q. Given all these differences, Dr. Rolph, what is 19 your opinion about whether Dr. Freedman's work provides a 20 valuable inference about the Bureau's smoothing process? 21 MR. SITCOV: I object. 22 THE COURT: Overruled. 23 A. As I indicated earlier, I found the, if you will, 24 analogy not close enough to really be persuasive to me in 25 making a statement about what the Bureau did. 2584 1 I'm sure there are valid inferences about what 2 Professor Freedman did, but I didn't find the analogies 3 strong enough to draw conclusions about what the Bureau did, 4 that is, the real world, as it were. 5 Q. Dr. Rolph, you also mentioned I think it was 6 point two that you made Dr. Freedman's analysis of the 7 interconnecting parts that made up the Bureau's smoothing 8 model. 9 Was this another term for testing the robustness 10 of the model? 11 A. Well, again, I believe in his testimony he termed 12 it robustness. 13 My understanding of the term robustness is 14 somewhat different than that. 15 What he did was to take particular pieces, if you 16 will, of this set of steps that was going into producing 17 these estimates and then removed them and then asked what 18 difference it made. 19 What I would view robustness as being is the 20 looking at departures from assumptions where you are looking 21 at the whole procedure and seeing how well it works. 22 Just to use the car example, at the risk of 23 overusing it, that would be the case where instead of 24 pulling off a spark plug you would ask whether it operated 25 well on wet streets, snowy streets, a hundred degree weather 2585 1 and so forth, really looking at variations at the whole 2 instead of taking the pieces apart. 3 Q. So in your view was Dr. Freedman's analysis a 4 proper robustness analysis? 5 A. Well, I certainly didn't find it persuasive as an 6 exercise in testing whether the model for robust. 7 Q. Dr. Rolph, did the Bureau do any tests of the 8 robustness of the smoothing model? 9 A. Yes, they did, as I understand it. 10 Q. I have handed you PX 118. 11 Can you tell the court what this is? 12 A. Yes. It's headed at the top, "Tom Belin, May 21, 13 1991, tech design meeting notes." 14 Q. What does this represent? 15 A. As I read it, it is looking at several 16 alternatives -- 17 THE COURT: Who is Tom Belin? 18 MR. SHERMAN: I'm not sure if he still is, your 19 Honor. 20 THE WITNESS: If I might, he was employed by the 21 Bureau, he was a graduate student I think at the time, he is 22 now on the staff at UCLA, actually. 23 Q. As I understand it from reading this memo, it is 24 Tom looking at a number of different alternative models in 25 an effort to see how sensitive the results are to slightly 2586 1 different specifications. 2 Q. Would you call this a test of the robustness of 3 the smoothing procedure? 4 A. I would call this a sensitivity analysis, and 5 that's the term I would use to test robustness. 6 MR. SHERMAN: We move Plaintiff's Exhibit 118, 7 your Honor. 8 MR. SITCOV: No objection. 9 THE COURT: 118 is admitted. 10 (Plaintiff's Exhibit 118 marked for 11 identification was received in evidence.) 12 BY MR. SHERMAN: 13 Q. I have handed you PX 126. 14 Can you tell the court what this is? 15 A. Yes. It's headed, "June 6, 1991, William Bell 16 statistical research division." 17 I mentioned Dr. Bell is a statistician at the 18 Bureau. 19 It's headed, "Raw and smoothed, from various 20 models, adjustment factors and census adjustments, tables 21 and graphs," and what it describes is some analysis which 22 looks at two alternative methods of smoothing labeled Fay 1 23 and Fay 2, which are identified as Bob Fay's model, and 24 presents results from those analyses. 25 So, again, I would call this a sensitivity 2587 1 analysis, an effort to make sure that what they were using 2 was reasonably robust. 3 MR. SHERMAN: We move 126, your Honor. 4 MR. SITCOV: No objection. 5 THE COURT: 126 is admitted. 6 (Plaintiff's Exhibit 126 marked for 7 identification was received in evidence.) 8 BY MR. SHERMAN: 9 Q. I have handed you PX 127. 10 Will you describe to the court what that is? 11 A. Yes. This is headed, "June 10, 1991, William 12 Bell, statistical research division," it is headed, 13 "Addition to handout to June 6, tilted," and it gives the 14 title as I gave to the previous document. 15 What it appears to be in its title and what is 16 tabulated on the page is some additional material relating 17 to the analysis on the previous document, again, an example 18 of, as I read it, sensitivity analysis of the model of 19 robustness. 20 MR. SHERMAN: We move 127. 21 MR. SITCOV: No objection. 22 THE COURT: 127 is admitted. 23 (Plaintiff's Exhibit 127 marked for 24 identification was received in evidence.) 25 BY MR. SHERMAN: 2588 1 Q. I have given you PX 129, Dr. Rolph. 2 Will you describe to the court what that is? 3 A. It's headed, "Census raw adjusted, modified 4 smoothed, smoothed counts," and at the top it says 10:44, 5 Tuesday, June, on my copy a blank, 1991. 6 Q. And what is it? 7 A. It appears to be a series of tables which 8 indicate the results of running or fitting different models 9 to the data, so it appear to be another example of looking 10 at the sensitivity of what your assumptions are. 11 Q. And, therefore, a test of the robustness? 12 A. Yes. 13 MR. SHERMAN: We move 129. 14 MR. SITCOV: I object and I will move to strike. 15 It starts out on page 6, your Honor. I don't 16 know what it is page 6 of. The witness says he only things 17 it appears to be something. 18 I certainly thing we are entitled to more than 19 that before this is admitted into evidence. 20 THE COURT: I also note the last two pages are 21 numbered 6 and 7 as well. 22 THE WITNESS: I thought that was only my copy. 23 MR. SHERMAN: I have no explanation, your Honor. 24 THE COURT: I will sustain your objection, Mr. 25 Sitcov, and strike the last statement of the testimony. 2589 1 MR. SHERMAN: My next one starts with page 2, so 2 I guess I will forego that. 3 BY MR. SHERMAN: 4 Q. Dr. Rolph, the documents that you have just gone 5 through with the court and me, are they all dated prior to 6 the Secretary's decision? 7 THE COURT: You can tell us that, can't you? 8 MR. SHERMAN: They are. 9 A. They are, I agree. 10 Q. Are you aware of any other sensitivity analysis 11 or tests for robustness done on the Bureau's smoothing model 12 prior to the Secretary's decision? 13 A. Yes. There were a number that were done which 14 are documented in the work that was work that was done by, 15 for example, the technical advisory committees work. 16 I have in mind -- what comes immediately to mind 17 is a study by Hoaglin and Glickman that looked at a number 18 of regression models -- 19 MR. SITCOV: Your Honor, I am going to object. 20 This is all hearsay. If there is such a study, let's see 21 it. But I certainly object to his testimony about other 22 studies without us being able to even look at them. 23 THE COURT: How do you know there are these other 24 studies? 25 THE WITNESS: I read the study that I am 2590 1 referring to right now. It's part of the report of Estrada, 2 Ericksen, Wolter and Tukey. 3 THE COURT: I will permit the testimony to stand 4 that he knows these things exist, that he read them, without 5 going into their context. 6 MR. ZIMROTH: It's also already in evidence, your 7 Honor. 8 THE COURT: Okay. 9 Q. Are you talking about the Hoaglin and Glickman 10 report? 11 A. Yes, I am. 12 MR. SHERMAN: For the record, your Honor, that is 13 appendix E to PX 195. 14 BY MR. SHERMAN: 15 Q. Will you describe to the court what that study 16 is? 17 A. Yes. That was exploring a number of different 18 regression models for the smoothing operation -- if I recall 19 I think there were 13 different ones -- that they looked at 20 and then looked at what the results were of using these 21 different models. 22 The tables that come to mind from that report of 23 ones that look at various evaluation of post-strata and 24 their overall conclusion was that they got approximately the 25 same result no matter which model they used. 2591 1 Q. In your view, Dr. Rolph -- let me ask you a 2 question before that. 3 Did the Hoaglin and Glickman study and the 4 various Bureau documents that we looked at do analysis of 5 the actual Bureau data? 6 A. Yes, that was analysis of Census Bureau data from 7 the 1990 census and PES. 8 Q. And in your view, were these proper tests of the 9 robustness of the smoothing model? 10 A. Yes. I thought they were carried out very well. 11 I have the highest regard for the statisticians at the 12 Bureau and I thought they did an excellent job. 13 Q. Dr. Rolph, what was the purpose of smoothing the 14 adjustment factors? 15 A. To reduce the sampling error of the raw 16 adjustment factors from the 1,392 post-strata. 17 Q. Do you have a view about whether that was 18 successfully done? 19 A. Yes. I believe in my direct testimony I even 20 referred to a chart, I think, which showed how much the 21 sampling error was reduced in various subgroups of the 22 population. 23 MR. SHERMAN: If I might, your Honor, can we 24 visit that chart just briefly? 25 THE COURT: All right. 2592 1 MR. SHERMAN: This is marked as PX 722, it has 2 been admitted. 3 BY MR. SHERMAN: 4 Q. Is that the chart you were referring to, Dr. 5 Rolph? 6 A. Yes. What that chart shows is for the different 7 racial groups, subgroups of the population, the difference 8 between the estimated standard error used in the smoothed 9 versus the unsmoothed models and what it shows is for the 10 racial minorities, you tend to get a considerable 11 improvement in reducing the sampling error in using 12 smoothing as opposed to not using smoothing. 13 Q. Dr. Rolph, did the testimony that you heard from 14 Dr. Freedman cause you to change your opinion about whether 15 smoothing was properly done by the Bureau? 16 A. No, it did not. 17 MR. SHERMAN: I have nothing else, your Honor. 18 THE COURT: Thank you, Mr. Sherman. 19 Any cross? 20 MR. SITCOV: Yes. 21 Could we have a short break, your Honor? 22 THE COURT: How long do you want? 23 MR. SITCOV: Half an hour? Will that be all 24 right? 25 THE COURT: Surely you jest. 2593 1 MR. SITCOV: Twenty minutes? 2 THE COURT: All right. 3 Is there anybody else today? 4 MR. ZIMROTH: Nobody else today. 5 MR. ZIMROTH: I have an application that I wanted 6 to make, but we will wait. 7 THE COURT: Until quarter of four? 8 MR. ZIMROTH: No, until after the examination. 9 (Recess) 10 11 12 (Continued on the next page.) 13 14 15 16 17 18 19 20 21 22 23 24 25 2594 1 MR. ZIMROTH: Your Honor, before we begin, may I 2 make an application? 3 THE COURT: Sure. 4 MR. ZIMROTH: Your Honor, at this time the 5 plaintiffs would like to renew their application to depose 6 and call as a trial witness former Secretary Robert 7 Mosbacher. Throughout this trial the defendants have hidden 8 behind his absence. When asked about a particular document 9 or page in his report, the witnesses have been cross- 10 examined on the grounds that it is pure speculation, they 11 don't know what was in the Secretary's mind when he wrote 12 whatever he wrote or whether he considered whatever he 13 considered. 14 In addition, they have shielded their own 15 witnesses from damaging documents and evidence on the 16 grounds that when we try to put the evidence in, we could 17 not prove what the Secretary actually saw or considered. I 18 give you just one example now. This is in Mr. Bounpane's 19 redirect testimony. In his cross-examination, Mr. Bounpane 20 was asked about the ethnographic studies, and he said that 21 he would be surprised if the Secretary didn't consider them. 22 On redirect, I believe it was Mr. Subar: 23 "Q. Mr. Cone asked you," he us talking to 24 Bounpane now, "some questions about ethnographic studies. 25 Do you know whether the Secretary of Commerce had the 2595 1 ethnographic studies available to him before he made his 2 decision? 3 "A. No, I don't. 4 "Q. Do you know whether Barbara Bryant had them 5 available to her before she made her recommendation? 6 "A. No, I don't." 7 Mr. Mosbacher's absence again was used as both a 8 shield and a sword here, which is exactly what the 9 defendants are doing. For that reason, we renew our 10 application to take his deposition and to call him as a 11 witness. 12 THE COURT: Thank you, Mr. Zimroth. 13 Mr. Millet? 14 MR. MILLET: Your Honor, there is nothing new 15 that has been proffered in this trial to prompt the Court to 16 reconsider the rulings that you have already issued in this 17 case. This is not rebuttal testimony that they are 18 offering. There is not anything that they are offering by 19 way of rebutting the direct testimony that we offered. That 20 is the point of this stage of the proceedings. 21 We are not using the former Secretary as a shield 22 or a sword in any manner. The only thing that Mr. Zimroth 23 has referred to that is in any way different are the 24 ethnographic studies. Your Honor, as we argued when these 25 studies were first attempted to be admitted and your Honor 2596 1 admitted them conditionally, on the face of most of those 2 studies, they post-dated the Secretary's decision. That was 3 the basis for our objection to those, and you acknowledged 4 that that was the reason for our objecting to them. Your 5 Honor allowed them to be admitted subject to plaintiffs 6 connecting them up at some point later in this proceeding. 7 That by itself, and it is the only thing I hear 8 Mr. Zimroth refer to, provides no reason to reconsider the 9 Court's earlier ruling on why the Secretary should or should 10 not be called. For that matter, he is the former Secretary, 11 and I'm not sure that we could find him at this point. He 12 doesn't work for us. 13 MR. ZIMROTH: Your Honor, I did just read one 14 example, but there are many others in the testimony. 15 Professor Cain on cross-examination was asked, "You don't 16 know, do you, that Secretary Mosbacher considered the 17 political implications of adjustment as an actual fact?" 18 The Court says, "It's an inference on your part? 19 "A. It's an inference based on the fact that 20 these things are obvious, that is correct. 21 "Q. For all you know, he made efforts to 22 isolate himself from political forces, is that correct? 23 "A. I don't know. Once you know something, I 24 don't know how you unknow it. He may have tried to unknow 25 things." 2597 1 The point being that they have been cross- 2 examining our witnesses again and again on the grounds that 3 our witnesses are in no position to know what Mr. Mosbacher 4 took into account and what he didn't take into account, what 5 influences were brought to bear and what not. I think that 6 that is grounds for our having a chance andm franklym for 7 the Court having a chance to hear Mr. Mosbacher. 8 THE COURT: I am not satisfied that the 9 plaintiffs have made the factual showing required by United 10 States v. Morgan. Therefore, the application to subpoena 11 Mr. Mosbacher is denied. 12 Cross-examination, Mr. Sitcov. 13 MR. SITCOV: Thank you, your Honor. I apologize 14 in advance if it is a little disjointed. 15 THE COURT: I won't be able to tell. 16 MR. SITCOV: Thank God this is not televised. 17 CROSS-EXAMINATION 18 BY MR. SITCOV: 19 Q. Dr. Rolph, you are aware, aren't you, that Dr. 20 Freedman replicated the Bureau's smoothing model? 21 A. Exactly? 22 Q. You heard him testify that he replicated the 23 Bureau's smoothing model, didn't you? 24 A. Quite honestly, I don't recall whether his words 25 said hta he exactly replicated. I know he replicated 2598 1 something he thought was very close to it. 2 Q. You didn't replicate the Bureau's smoothing 3 model, did you? 4 A. I certainly did not. 5 MR. SITCOV: May I enter the well for a second, 6 your Honor? 7 THE COURT: Sure. 8 MR. SITCOV: You'll love this, your Honor, trust 9 me. 10 Q. I am going to put it over here. Can you see 11 this? 12 A. Yes, I can see it. 13 Q. Do you see the second equation there, that says Y 14 equals X beta plus V? 15 A. Second from the bottom there or the top? Where 16 are we? 17 Q. I'm sorry. Third line from the bottom, Y equals 18 X beta plus V? 19 A. Yes, I do see that. 20 Q. The sampling covariance matrix is sigma, is that 21 right? 22 A. Not on that sheet. The sample covariance matrix 23 is V 3, if I understand that sheet. 24 Q. Let's do it hypothetically that Y equals X beta 25 plus B. In the sampling, the covariance matrix for V is 2599 1 sigma? 2 A. Okay. 3 Q. Then a GSL estimate for beta should be based on 4 sigma, is that right? 5 A. If that is all you are assuming, yes. 6 Q. You testified about the Hoaglin & Glickman report 7 on your direct examination. Do you recall that? 8 A. Yes, I do. 9 Q. Do you recall that when you were deposed last 10 month, you testified that you didn't fully understand 11 certain things about the Hoaglin and Glickman report? 12 A. I recall when I was deposed I was shown several 13 pages which at the time I don't believe I knew were from the 14 Hoaglin and Glickman report by memory, and I was not able to 15 decipher them for Mr. Baron, is my best memory. 16 Q. You recall saying that you don't fully understand 17 how the weighting is being plotted? 18 A. I suspect I said even vaguer things than that. 19 Q. Now, it is correct, isn't it, that you hadn't 20 performed any independent analyses of the 1990 production 21 smoothing model, is that right? 22 A. Yes. I still have not. 23 Q. You haven't performed any independent 24 calculations regarding how much uncertainty there is in the 25 variance of the smoothed adjustment factors, isn't that 2600 1 right? 2 A. That is also correct, yes. 3 Q. You haven't performed any statistical analyses on 4 any census-related adjustment issue since the time you have 5 been retained as a consultant in this case, isn't that 6 right? 7 A. That's correct, yes. 8 Q. It would be correct to describe the presmoothing 9 process as a situation where variances were estimated from 10 survey data by the jackknife and then were presmoothed by 11 regression, is that right? 12 A. That sounds about right to me. 13 Q. Do you recall testifying on May 15th in this case 14 that you couldn't think of any other example besides the 15 1990 proposed adjustment of where variances had been 16 estimated from survey data by the jackknife and then the 17 estimates were presmoothed by a regression? 18 A. I don't recall that precisely. But if you 19 represent that, I have no reason to disbelieve you. 20 Q. That is a true statement, isn't it? 21 A. I'm saying I don't disbelieve you. I can't 22 recall my exact answer. That sounds right. 23 THE COURT: Supposing he put the question to you 24 now afresh. 25 Q. You are not aware of any other examples where 2601 1 variances have been extimated from survey data by the 2 jackknife and then the estimates were smoothed by 3 regression? 4 A. Actually, if you look at some of the CPS 5 documents that I was asked about in my rebuttal testimony, 6 they do something very similar to that but not exactly that. 7 Q. Do you have Plaintiffs' Exhibit 763 in front of 8 you? 9 A. Yes, I do. 10 Q. The model variances in Plaintiffs' Exhibit 763 11 are not used as inputs to a smoothing model, are they? 12 A. Not to my knowledge. 13 Q. Do you have Plaintiffs' Exhibit 70? Do you have 14 that in front of you? 15 A. Seven zero? 16 Q. Yes. 17 A. Just a moment. Yes, I do. 18 Q. According to the third sentence in the first 19 paragraph, Bill Bell preferred using a simple model term 20 sigma squared I and the smoothed variances term V sub 3, is 21 that correct? 22 A. I think you have read that correctly, yes. 23 Q. It doesn't say why he preferred that, does it? 24 A. No, it does not say that here. 25 Q. Do you have Plaintiffs' Exhibit 87 in front of 2602 1 you? 2 A. Yes, I do. 3 Q. Exhibit 87 is based on 1988 data, is it not? 4 A. Yes, it is. 5 Q. It doesn't even refer to 1990 data, does it? 6 A. That's correct. 7 Q. In fact, on the second page, the third line from 8 the bottom, the whatever you want to call this, memorandum, 9 notes that it prefers sigma squared I to sigma squared V 10 because the former appears to give smaller estimated 11 standard errors, isn't that right? 12 A. It reads actually smaller standard errors, if 13 that is what you are referring to, yes. 14 Q. Yes. 15 A. Sigma squared V zero, I think. 16 Q. Yes. 17 A. Yes. 18 Q. Do you have Plaintiffs' Exhibit 92 in front of 19 you? 20 A. Yes, I do. 21 Q. Can you look at the second line in the first 22 paragraph. It says, "I computed AIC values." 23 A. Yes, I do. 24 Q. What does AIC stand for? 25 A. I would infer it means -- you are not going to 2603 1 like this -- Akaike's information criterion, I believe. 2 Q. Are you aware of any other work which uses 3 Akaike's criterion to compare nonnested models? 4 A. I guess I would have to say that I wouldn't know 5 one way or the other. I come across it being used, and the 6 answer is no, I guess. 7 Q. Do you see in the first line in the first 8 paragraph the reference to 2 log L? 9 A. Yes, I do. 10 Q. Do you know what L is? 11 A. I would infer from its use here that it is what 12 is called the likelihood function. 13 Q. Are you aware of any other work which uses 14 negative 2 log L to compare nonnested models? 15 A. I certainly am aware of using minus 2 log L 16 routinely. I would have a hard time saying whether they are 17 nested or nonnested without going back and looking at some 18 paper. But it is a quite standard thing to calculate. 19 Q. As you sit here today, you don't know of any 20 other work which uses their nonnested models? 21 A. Not one way or the other, no. 22 Q. The comparison in Plaintiffs' Exhibit 92 is of 23 two nonnested models using 1988 data, is that right? 24 A. As I read it, it refers to, I think, about four 25 models, if I am not mistaken. It talks about model 1, model 2604 1 2, model 4. I am somehow not focusing on model 3. So I 2 think it is more than two models as I read it. I'm sorry, I 3 lost the other part of your question. Oh, nonnested? 4 Q. Yes. 5 A. I am not sure from looking at this, but it could 6 be that the model which says T equals sigma squared C and T 7 equals sigma squared I would be nested. But it would depend 8 on some details which I am just not clear on from looking at 9 this memo. 10 So the answer is I'm not certain about the answer 11 to your question. 12 Q. Can you look at, I think it is, the third 13 sentence in the first paragraph, the one that says, "It 14 seems to me comparisons are only valid between models with 15 the same sampling covariance (S) and with regression 16 variables (M) determined the same way. 17 A. I see that same sentence. 18 Q. Could you look down into the last paragraph? 19 A. Okay. 20 Q. It says, "Based on your work to date, I would 21 conclude S equals V sub 3 and T equals sigma squared I are 22 the best choice you have got right now." 23 Do you see that? 24 A. Yes, I do. 25 Q. Those are two different covariance matrices, 2605 1 aren't they? 2 A. Yes, they are. 3 Q. Do you recall being asked at your deposition last 4 month the following questions and giving the following 5 answers? It is on page 914. 6 "Q. Let me switch gears for a moment. Do you 7 know what folded wings are? Have you ever heard of that 8 term? 9 "A. Only when making paper airplanes. 10 "Q. Not in statistical analysis? 11 "A. It doesn't come immediately to mind." 12 Do you recall that? 13 A. Yes, I do recall that particular question and 14 that answer. 15 Q. Do you have Plaintiffs' Exhibit 195 in front of 16 you, that is, the report from Hoaglin and Glickman that I 17 think you testified about earlier? 18 A. I don't, but if you would put it in front of me, 19 I would be happy to look at it. 20 Q. Would you turn to page 26. 21 A. Okay. 22 Q. There is a section there entitled "Fitting 23 Regressions to Adjustment Factors." At the bottom of that 24 first paragraph there it says, "Tukey developed these 25 alternative treatments of covariances as a companion to the 2606 1 prescription for dealing with the variances. The three are 2 as follows: Use raw correlations, use zero correlations, 3 use folded wings." 4 Do you see that? 5 A. I see that, yes. 6 Q. Could you turn over to page 31, to the section 7 number 6, "Comparing Fits." Paragraph number 2 says, "The 8 four combinations from 1" -- 9 A. Pardon me. Page 31? 10 Q. 31. There is a section entitled "6. Comparing 11 Fits." 12 A. Yes. 13 Q. Then there are some numbered paragraphs. Do you 14 see that? 15 A. Yes. Okay. 16 Q. The second paragraph, the one numbered 2, says, 17 "The four combinations from one of (Point 6, Point 10) with 18 one of (raw correlations folded wings)." Do you see that? 19 A. I see that, yes. 20 Q. Are you being compensated for your work for 21 plaintiffs in this case? 22 A. I have been in the past, yes. 23 Q. What is your rate of compensation? 24 A. It was $200 an hour. 25 Q. Is it the same rate for testimony in depositions? 2607 1 A. No. It is time and a half. 2 Q. So that is $300 an hour? 3 A. That's correct. 4 MR. SITCOV: No further questions. 5 THE COURT: Any redirect? 6 MR. SHERMAN: No, I have nothing, your Honor. 7 THE COURT: Thank you very much, Dr. Rolph. It 8 is a pleasure to hear from you again. 9 (Witness excused) 10 MR. MILLET: Your Honor, on the chance that this 11 may be the last day of trial, would it be appropriate to 12 talk about post-trial briefing 13 THE COURT: All that needs to be done is work out 14 a schedule, and I would ask you to meet with my clerk to 15 work that out. 16 MR. SOLOMON: I apologize for this. I should not 17 have to raise this. We asked your Honor after Dr. 18 Freedman's cross for what he had that underlay his 19 assertions concerning Professor Fisher. 20 THE COURT: Right. Mr. Sitcov said he would get 21 it for you. 22 MR. SOLOMON: Right. After lunch I was given a 23 disk which I think I should have been given weeks ago --but 24 put that aside -- which does have some data on it, and then 25 a sheet of paper, the sheet of paper purportedly being from 2608 1 what Dr. Fay did and the disk being some data. 2 We don't have the program. We don't have the 3 methodology that was done. This game should not be played. 4 We would request all of the data and the program so that we 5 can see what Dr. Freedman did. We request that your Honor 6 give us leave to take the deposition of this person this 7 evening. 8 THE COURT: What person? 9 MR. SOLOMON: Of Dr. Freedman, to find out what, 10 in fact, is going on. I am happy to take Mr. Sitcov's 11 representation that we have it if he will give it to us. 12 Otherwise, especially if we are thinking about calling 13 Professor Fisher, let's get the stuff so we can look at it. 14 MR. SITCOV: They had that, your Honor. They 15 have all of his data. 16 THE COURT: And the program that he says he 17 needs? 18 MR. SITCOV: They do not have the program. That 19 is not the data he used. The program is something that he 20 developed. They are entitled, under Rule 705 to the facts. 21 THE COURT: Can they interpret the data without 22 the program? 23 MR. SITCOV: Absolutely. They most certainly 24 can. 25 THE COURT: Then why do you need the program, Mr. 2609 1 Solomon? 2 MR. SOLOMON: First of all, I know of no rule 3 that limits Rule 705, the reference in 705, to the data and 4 not the programs. In fact, when I read to your Honor 5 yesterday from Judge Weinstein is exactly to the contrary. 6 Second, I took this disk and I gave it to the computer 7 people, and they came back and they said, what is it you 8 want us to do with this because we don't have the 9 methodologies and we don't have the program, and how can we 10 do it? 11 The witness told me yesterday morning, you will 12 remember, that without the program, if we have the data, 13 that what we can do is we can try to recreate what he did 14 and it will take between, in his view, two weeks and two 15 months. There is no reason for this. 16 MR. SITCOV: Rule 705 says, your Honor, "The 17 expert may in any event be required to disclose the 18 underlying facts or data." He has disclosed all of his 19 facts and data. 20 THE COURT: I don't have a clear picture in my 21 own head as to what the difference is between a program and 22 data. 23 MR. SITCOV: The data are the computer bits, that 24 is the information that is on the computer disk. The 25 program would be the original work that Dr. Freedman did to 2610 1 utilize that. 2 The plaintiffs ought to have programs that do 3 these things because this work essentially recreates Dr. 4 Fisher's stuff. 5 THE COURT: Why are you reluctant to give them 6 the program? I don't have a feel for that either. 7 MR. SITCOV: Mr. Solomon is acting as though we 8 are obliged by the rule to do so. 9 THE COURT: Assuming you are not, would you do it 10 as a favor for me? 11 MR. SITCOV: If we weren't obliged and we could 12 do it as a favor for you, I would be happy to do it if it is 13 possible to do easily. I don't know if it is. I suppose I 14 would have to ask Dr. Freedman. 15 THE COURT: Do you want to do that? Do you want 16 to caucus? 17 MR. SITCOV: Yes. Do you mind? 18 THE COURT: Not at all 19 (Pause) 20 MR. SITCOV: Your Honor, I am told by Dr. 21 Freedman that part of it still exists in some sort of hard 22 form, part does not. He could recreate that if your Honor 23 desires, and we could provide it to the plaintiffs. I 24 should note that according to Dr. Freedman, it would not 25 take any near weeks or days, but a short period of time. 2611 1 On the other hand, we have not had any access to 2 anything of Dr. Fisher's. We would certainly like to have 3 the programs that were used to create Dr. Fisher's charts, 4 and we would like them pretty quick. 5 THE COURT: Ly. 6 MR. SITCOV: Ly. We would like them fastly. 7 THE COURT: Sounds fair. Mr. Solomon? 8 MR. SOLOMON: The question of quickly is 9 completely unfair. This is the first we heard of this 10 request. As to whether to give it to them, we offered it to 11 them on May 1st. Of course there is no problem with that. 12 The facts, including the data and the programs, should be 13 made available upon request. We made that offer on May 1st. 14 I have no problem with that. 15 MR. SITCOV: We asked for them on Sunday. 16 MR. SOLOMON: What Sunday? 17 MR. SITCOV: This past Sunday. 18 MR. SOLOMON: Sorry. The request was not made of 19 me. If there is a problem, I see nobody coming up. There 20 is no problem. 21 THE COURT: In any case, you will do it, and work 22 out a schedule between you. 23 MR. SITCOV: Yes. 24 THE COURT: I appreciate your cooperativeness. 25 MR. RIFKIND: While we are making mutual 2612 1 requests, there is a disagreement between me and Mr. Sitcov 2 this morning. He made some representations about Dr. 3 Fisher's deposition with which I took exception. The 4 deposition wasn't present. He said it was in the next room. 5 I wish in the course of the evening he will let me know what 6 pages he was referring to in the Fisher deposition, because 7 I can't find them. 8 THE COURT: You will do that? 9 MR. SITCOV: Yes. 10 THE COURT: On the chance, paren read hope, that 11 there is no further rebuttal -- it would be tomorrow 12 afternoon -- I would like before we adjourn to commend 13 counsel. This case had the potential to be highly 14 acrimonious. I think it worked out rather well. 15 It is a tribute to the professionalism and the 16 civility of counsel for both sides that they were able to 17 resolve their many procedural and substantive differences 18 with the least bit of conflict. I have rarely seen a case 19 tried as well as this. Indeed, it may well have been the 20 best tried case I have ever presided over. I congratulate 21 counsel. 22 I would also like to just mention to the 23 statisticians here, I don't know whether you all know it or 24 not, but I have spent almost all of my professional life in 25 the groves of academe, including ten years as the dean of a 2613 1 law faculty. So I think I know good teaching when I hear 2 it. 3 I expected to come out of this trial as ignorant 4 as when I entered it. It may well be that I still am, but I 5 think I learned a great deal, and it was certainly a most 6 rewarding intellectual experience. I can understand now why 7 the faculty at Berkeley is so highly regarded. You may be 8 pleased to know that my next law clerk who will join me in 9 September is from the law school at Berkeley. A Jesuit, by 10 the way. 11 It really has been a pleasure meeting all of you. 12 If I don't get to see you again, I hope that I do have the 13 pleasure somewhere, sometime to deal with all of you again. 14 Thank you for an extremely well tried case. 15 (Adjourned) 16 17 18 19 20 21 22 23 24 25