Little Rock School District v. Pulaski County Special School District No. 1 , 470 F. Supp. 2d 963 ( 2004 )

470 F. Supp. 2d 963 (2004)

LITTLE ROCK SCHOOL DISTRICT Plaintiff
v.
PULASKI COUNTY SPECIAL SCHOOL DISTRICT NO. 1, et al. Defendants
Mrs. Lorene JOSHUA, et al. Intervenors
Katherine KNIGHT, et al. Intervenors

No. 4:82CV00866 WRWJTR.

United States District Court, E.D. Arkansas, Western Division.

June 30, 2004.

Philip E. Kaplan, Kaplan, Brewer, Maxey & Haralson, P.A., Little Rock, AR, for Plaintiff.

Clayton R. Blackstock, Mark Terry Burnette, Mitchell, Blackstock, Barnes, Wagoner, Ivers & Sneddon, PLLC, Little Rock, AR, Norman J. Chachkin, NAACP Legal Defense & Educational Fund, Inc., New York, NY, for Intervenor Plaintiffs.

John W. Walker, John W. Walker, P.A., Little Rock, AR, for Intervenor Plaintiffs/Defendants.

Stephen W. Jones, Jack, Lyon & Jones, PA., Sharon Carden Streett, Christopher J. Heller, Friday, Eldredge & Clark, LLP, Little Rock, AR, for Defendants.

James M. Llewellyn, Jr., William P. Thompson, Thompson & Llewellyn, P.A., Fort Smith, AR, for Intervenors.

MEMORANDUM OPINION^[1]

WILSON, District Judge.

I. Warning to the General Reader

The general reader, if any there be, should realize that educators, like lawyers, have developed their own language. To the extent that time, patience, and skill would permit, I have tried to Garnerize^[2] this Memorandum Opinion. I have fallen short, but hope the effort will be of some help.

II. Background

On September 13, 2002, I entered a Memorandum Opinion (the "September 13 Decision") holding that the Little Rock School District ("LRSD") had substantially complied with all of its desegregation obligations set forth in the January 16, 1998 Revised Desegregation and Education Plan (the "Revised Plan"),^[3] except those obligations contained in ž 2.7.1. LRSD v. Pulaski County Special Sch. Dist., et al., 237 F. Supp. 2d 988 (E.D.Ark.2002); aff'd, 359 F.3d 957 (8th Cir.2004). Section 2.7 of the Revised Plan obligated LRSD to "implement programs, policies and/or procedures designed to improve and remediate African-American achievement." Section 2.7.1 ensured that the promise made in ž 2.7 would have teeth by requiring that:

LRSD shall assess the academic programs implemented pursuant to ž 2.7 after each year in order to determine the effectiveness of the academic programs in improving African-American achievement. If this assessment reveals that a program has not and likely will not improve African-American achievement, LRSD shall take appropriate action in the form of either modifying how the program is implemented or replacing the program.

CX 871.

As stated above, in the September 13 Decision, I found that LRSD had substantially complied with its obligations under ž 2.7 of the Revised Plan; however, I determined there were numerous, substantial deficiencies in LRSD's efforts to comply with its obligations under ž 2.7.1. See LRSD, 237 F.Supp.2d at 1076-1082. The September 13 Decision gave LRSD until March 15, 2004, to demonstrate that it had substantially complied with ž 2.7.1 of the Revised Plan, as specified in subparts A, B, and C of the Compliance Remedy. LRSD, 237 F.Supp.2d at 1087-88.

LRSD has been involved continuously in desegregation litigation since 1956. See LRSD, 237 F.Supp.2d at 996 n. 18. As far as I can tell from the reported cases, LRSD now has the dubious distinction of having been under federal court supervision longer than any other school district in history. Thus, LRSD is well seasoned when it comes to court supervision and monitoring.

On November 12, 2002, Joshua Intervenors^[4] ("Joshua") appealed (docket no. 3704) the September 13 Decision. On March 2, 2004, the Eighth Circuit Court of Appeals affirmed. LRSD v. Armstrong, 359 F.3d 957 (8th Cir.2004). Thus, all aspects of the September 13 Decision are now final and the law of the case.

On March 12, 2004, LRSD filed its Compliance Report (docket no. 3837) seeking complete unitary status on the ground that it had substantially complied with the obligations imposed under the Compliance Remedy and ž 2.7.1 of the Revised Plan. On April 15, 2004, Joshua filed an Opposition to LRSD's Request for Release from Court Supervision of Its Desegregation Efforts (docket no. 3856), along with a supporting Memorandum (docket no. 3857). I must now decide whether LRSD has met its obligations under the Compliance Remedy, and whether it should be released from almost five decades of court supervision.

III. The September 13, 2002 Compliance Remedy

Almost 70% of LRSD's students are African-American. Historically, the academic achievement of many of these students, as ganged by standardized test scores, is low and poses a significant longterm challenge to LRSD teachers and administrators. Of course, because this so-called "achievement gap" is a nationwide phenomenon, it is a problem that educators must confront in schools throughout the country. See LRSD, 237 F.Supp.2d at 1073-74.

Importantly, ž 2.7 of the Revised Plan promised only that LRSD would "implement programs, policies and/or procedures designed to improve and remediate African-American achievement," See docket no. 3410 at 51. However, ž 2.7.1 went on to require LRSD to assess the ž 2.7 programs annually in order to determine their effectiveness, and to modify or replace any programs that were shown not to be working to improve African-American achievement. Id. at 148. Read together, the obligations set forth in ž 2.7 and ž 2.7.1 of the Revised Plan required LRSD not only to design academic programs that were intended to improve the academic achievement of African-American students, but also to make annual assessments of those programs to ensure that they were, in fact, effective in improving African-American achievement. Expressed in the vernacular of my native Scott County, Arkansas, ž 2.7 contained the sizzle and ž 2.7.1 contained the bacon. These two sections of the Revised Plan are crucially important to the future educational success of a large number of LRSD's current and future students.

During the November 2001 hearings on unitary status, Dr. Bonnie Lesley, LRSD's Associate Superintendent of Instruction and Curriculum, defined a program assessment as something that is "dynamic, it is interactive, it's ongoing, it happens frequently, and it is a measurement, along with the analysis that you would make of whatever results are available." LRSD, 237 F.Supp.2d at 1077. In contrast, she defined a program evaluation as "more long term, it may consider observations or measurements in addition to test scores, and is guided by a series of research questions that are usually provided by whoever the consumer is of that report." Id. In other words, a program assessment is a relatively informal process that may not result in much documentation, while a program evaluation is a formal process that always involves the preparation of an often lengthy written program evaluation which is centered around carefully prepared research questions that the evaluation is designed to answer.

Section 2.7.1 of the Revised Plan provided that LRSD must make assessmentsnot evaluations-of the ž 2.7 programs in order to determine their effectiveness in improving the academic achievement of African-American students. However, as early as March 15, 2000, LRSD acknowledged, in its own Interim Compliance Report (docket no. 3356), that ž 2.7.1 of the Revised Plan obligated it to prepare evaluations on the key ž 2.7 programs so that LRSD administrators could make an in formed decision on the effectiveness of those programs. See LRSD's Interim Compliance Report at 51-55. Furthermore, during her testimony in November of 2001, Dr. Lesley admitted that, even though ž 2.7.1 of the Revised Plan did not mention anything about LRSD's obligation to prepare program evaluations to determine the effectiveness of the ž 2.7 programs, she and other administrators interpreted that section of the Revised Plan as requiring LRSD to perform evaluations covering the most important ž 2.7 programs. LRSD, 237 F.Supp.2d at 1077. Because it is so important to an understanding of the Compliance Remedy, I want to be very clear on this point: The evidence overwhelmingly establishes that LRSD has always construed the obligations contained in ž 2.7.1 of the Revised Plan as requiring it to prepare formal program evaluations on the key ž 2.7 programs. See Interim Compliance Report at 51-55; Final Compliance Report dated March 15, 2001 (docket no. 3410) at 148; Dr. Lesley's testimony cited in the September 13 Decision, LRSD, 237 F.Supp.2d at 1077.

Since my decision will turn on whether LRSD has properly implemented the September 13, 2002 Compliance Remedy, it is set forth below in full:

VII. Compliance Remedy

Because LRSD failed to substantially comply with the crucially important obligations contained in ž 2.7.1, it must remain under court supervision with regard to that section of the Revised Plan until it: (a) demonstrates that a program assessment procedure is in place that can accurately measure the effectiveness of each program implemented under ž 2.7 in improving the academic achievement of African-American students; and (b) prepares the program evaluations identified on page 148 of the Final Compliance Report and uses those evaluations as part of the program assessment procedure contemplated by ž 2.7.1 of the Revised Plan. The details of this compliance remedy are set forth below:

A. For the entire 2002-03 school year and the first semester of the 2003-04 school year, through December 31, 2003, LRSD must continue to assess each of the programs implemented under ž 2.7 to improve the academic achievement of African-American students. LRSD now has over three years of testing data and other information available to use in gauging the effectiveness of those programs. I expect LRSD to use all of that available data and in formation in assessing the effectiveness of those programs and in deciding whether any of those programs should be modified or eliminated.

B. LRSD must maintain written records regarding its assessment of each of those programs. These written records must reflect the following information: (a) the written criteria used to assess each program during the 2002-03 school year and the first semester of the 2003-04 school year; (b) the results of the annual assessments of each program, including whether the assessments resulted in program modifications or the elimination of any programs; and (c) the names of the administrators who were involved with the assessment of each program, as well as at least a grade level description of any teachers who were involved in the assessment process (e.g., all fourth grade math teachers; all eighth grade English teachers, etc.).

C. LRSD must use Dr. Nunnery or another expert, from outside LRSD with equivalent qualifications and expertise to prepare program evaluations on each of the programs identified on page 148 of the Final Compliance Report.^[5] I will accept all program evaluations that have already been completed by Dr. Nunnery or someone with similar qualifications and approved by the Board. All program evaluations that have not yet been completed on the remaining programs identified on page 148 of the Final Compliance Report must be prepared and approved by the Board as soon as practicable, but, in no event, later than March 15, 2003. In addition, as these program evaluations are prepared, LRSD shall use them, as part of the program assessment process, to determine the effectiveness of those programs in improving African-. American achievement and whether, based on the evaluations, any changes or modifications should be made in those programs. In addition, LRSD must use those program evaluations, to the extent they may be relevant, in assessing the effectiveness of other related programs.

D. Joshua must monitor LRSD's compliance with ž 2.7.1 and must immediately bring to the attention of LRSD all problems that are detected in its compliance with its obligations under ž 2.7.1, as those obligations are spelled out in this Compliance Remedy. Thereafter, Joshua and LRSD must use the "Process for Raising Compliance Issues" set forth in ž 8.2, et seq., of the Revised Plan to attempt to resolve those compliance issues. If those efforts are unsuccessful, Joshua must present the issues to me for resolution, as required by ž 8.2.5. Any such presentation must be timely.

E. The 0DM must also monitor LRSD's compliance with ž 2.7.1 and help to ensure that LRSD fulfills its obligations, as specified in this Compliance Remedy.

F. On or before March 15, 2004, LRSD must file a Compliance Report which documents its compliance with its obligations under ž 2.7.1. Any party, including Joshua, who wishes to challenge LRSD's substantial compliance with ž 2.7.1, as specified above, may file objections with the court on or before April 15, 2004. Thereafter, I will decide whether LRSD has substantially complied with ž 2.7.1, as specified in this Compliance Remedy, and should be released from all further supervision and monitoring.

LRSD, 237 F.Supp.2d at 1087-88 (emphasis added).

Even a casual reading of this Compliance Remedy reveals that it imposed four essential obligations on LRSD, Joshua, and the Office of Desegregation Monitoring ("ODM"):

1. Preparation of Annual Assessments of ž 2.7 Programs.

LRSD was required to annually assess each of the programs implemented under ž 2.7 during the 2002-2003 school year and the first semester of the 2003-2004 school year and then use those assessments, the more than three years of testing data, and all other relevant available information to determine the effectiveness of those programs and to decide whether any of the programs should be modified or eliminated. LRSD also was required to maintain written records reflecting: (a) the criteria used to assess each program; (b) the results of the annual assessments of each program, including whether any programs were modified or eliminated; and (c) the administrators and teachers who were involved in preparing the assessment of each program.

Subparts A and B of the Compliance Remedy obligated LRSD to assess each of the ž 2.7 programs implemented during the 2002-03 school year and the first semester of the 2003-04 school year and to maintain written records of its annual assessments of each of those programs. I made no mention of LRSD preparing evaluations of ž 2.7 programs because, on its face, nothing in ž 2.7.1 of the Revised Plan obligated LRSD to perform "program evaluations." However, Dr. Lesley made it clear in her testimony that LRSD administrators knew and understood that the "assessment" obligation in ž 2.7.1 included the obligation of preparing "program evaluations." See discussion, supra, at 965-966. Therefore, I concluded it would be best to use the same terms in the Compliance Remedy that the parties themselves had chosen to use in ž 2.7.1 of the Revised Plan. Because I was tracking the parties' own language, I never dreamed the use of the terms assess and assessment would suddenly create confusion for LRSD administrators in deciding how to comply with subparts A and B of the Compliance Remedy.

2. Preparation of Page 148 Program Evaluations.

LRSD was required to hire experts to prepare the fourteen program evaluations identified on page 148 of its March 15, 2001 Compliance Report. The. Court agreed to accept all program evaluations that had already been prepared by outside consultants and ordered LRSD to hire outside consultants to complete the unfinished program evaluations, which were to be approved by the LRSD Board and filed with the Court no later than March 15, 2003. Finally, as those evaluations were prepared, LRSD was required to use themÔÇöto the extent they might be usefulÔÇöin its annual assessment of the effectiveness of the ž 2.7 programs.

3. Monitoring.

Joshua's counsel was required to continue with its monitoring of LRSD's implementation of the Compliance Remedy and to use the procedures set forth in ž 8.2 of the Revised Plan to resolve any compliance problems that might arise. If Joshua and LRSD were unsuccessful in using the ODM to facilitate and resolve those problems, they were required to bring those compliance issues directly to me for resolution. Finally, the ODM was directed to monitor LRSD's implementation of the Compliance Remedy and "to help ensure that LRSD fulfills its obligations" specified therein. The sole purpose of subparts D and E the Compliance Remedy was to ensure that, if the ODM was unable to successfully facilitate the resolution of any compliance issues raised by Joshua, those compliance issues would be brought to my attention so that I could resolve them on a timely basis, thereby avoiding any surprises when LRSD filed its Compliance Report.

4. Preparation of Compliance Report. LRSD was ordered to file a Compliance Report by March 15, 2004, documenting its substantial compliance with ž 2.7.1 of the Revised Plan and the Compliance Remedy.

IV. The Parties' Compliance Activities after September 13, 2002

A. The Court Clarifies Joshua's Monitoring Obligation.

After the entry of the September 13 Decision, LRSD did not seek clarification of any terms used in the Compliance Remedy or any of its compliance obligations. Based on its silence, I concluded that LRSD understood what it was required to do under the Compliance Remedy, and that it was proceeding apace to meet those obligations.

In contrast, on October 1, 2002, Joshua's counsel wrote a letter (docket no. 3680) objecting to the monitoring obligations imposed on them under subpart D of the Compliance Remedy. Among other things, Joshua's counsel challenged the Court's decision to: (1) impose monitoring obligations on them "that were contemplated to be the responsibility of the ODM"; and (2) place "a greater burden upon Joshua than it has imposed upon the ODM." Id. By way of relief, Joshua's counsel sought clarification of their monitoring obligations under subpart D of the Compliance Remedy and "a hearing on this matter so that an appropriate record on the issues of the role of the ODM monitoring and Joshua's monitoring may be fully developed." Id.

On October 11, 2002, I entered an Order (docket no. 3685) clarifying the monitoring obligations imposed on Joshua's counsel under the Compliance Remedy. After noting that Joshua's counsel had been engaged in monitoring LRSD's compliance with its desegregation obligations since at least 1990, I made it clear that subpart D of the Compliance Remedy only obligated Joshua's counsel "to continue to perform their monitoring role according to the same procedure they and LRSD have followed for many years in this case." LRSD, 237 F.Supp.2d at 1091. Because the October 1, 2002 letter could be read to suggest that Joshua's counsel would only continue their monitoring role, if ordered to do so by the Court, I also made it clear that:

I do not believe I can force Joshua's counsel to perform monitoring duties ÔÇö something that I may have mistakenly assumed they wanted to do. I will leave it up to Joshua's counsel to decide if they have an ethical and professional duty to continue monitoring LRSD's compliance with its sole remaining obligation under the Revised Plan. I hope Joshua's counsel resolves that question in favor of continuing their long-standing commitment to monitoring LRSD's compliance with its desegregation obligations. However, since they complain about my expressly directing them to continue monitoring LRSD's compliance with ž 2.7.1 of the Revised PlanÔÇösomething I never expected to hearÔÇöI believe I must now clarify Section VII.D. of the Memorandum Opinion to read as follows:

Joshua may monitor LRSD's compliance with ž 2.7.1 and, if they choose to do so, they should bring to the attention of LRSD, on a timely basis, all problems that are detected in its compliance with its obligations under ž 2.7.1, as those obligations are spelled out in this Compliance Remedy. Thereafter, Joshua and LRSD must use the "process for raising compliance issues" set forth in ž 8.2, et seq., of the Revised Plan to attempt to resolve those compliance issues. If those efforts are unsuccessful, Joshua shall present the issues to me for resolution, as required by ž 8.2.5. Any such presentation must be timely.

Id. at 1091.

Finally, I emphasized that, regardless of whether Joshua's counsel decided to continue to monitor LRSD's compliance with its obligations under ž 2.7.1 of the Revised Plan, "the ODM staff most certainly will continue their close monitoring of LRSD's compliance with that section of the Revised Plan." Id. at 1091. My final admonition on the subject of monitoring was as follows:

If Joshua's counsel decide to continue their monitoring role, which is independent from the monitoring work performed by the ODM, . . . I expect counsel for Joshua and LRSD to cooperate and work together to ensure that things go smoothly with regard to monitoring LRSD's implementation of its obligations under ž 2.7.1. However, if actual disputes arise regarding monitoring, I will be available to resolve them.

Id. at 1091.

B. LRSD Adopts Compliance Plan, Approves Regulation IL-R1, and Designates "Areas" for ž 2.7.1 Program Evaluations.

On October 10, 2002, LRSD's Board of Directors ("the Board") adopted a "Compliance Plan" that was specifically designed to satisfy the Court's Compliance Remedy. See Exhibit A to LRSD's May 14, 2003 Notice of Filing Program Evaluations (docket no. 3745).^[6] The Compliance Plan recognized that, in order for LRSD to meet its obligations under the Compliance Remedy, it would have to satisfy three core obligations: (1) "develop a written procedure for evaluating the programs implemented pursuant to ž 2.7 to determine their effectiveness in improving the academic achievement of African-American students"; (2) "maintain written records of the criteria used to evaluate each [ž 2.7] program"; and (3) "[p]repare a comprehensive program evaluation of each academic program implemented pursuant to . . . 2.7 to determine its effectiveness in improving the academic achievement of African-American students and to decide whether to modify or replace the program" Id. at 3-5 (emphasis added).

Significantly, there is nothing in the Compliance Plan adopted by the Board that suggests LRSD was confused about the meaning of any of the terms in the Compliance Remedy or any of its compliance obligations. Additionally, the Compliance Plan makes it clear that LRSD construed subpart A of the Compliance Remedy as requiring it to prepare "a comprehensive program evaluation of each academic program implemented pursuant to ž 2.7. . . ."

The Compliance Plan also included a detailed "Action Plan Time Line" that: (1) identified the LRSD employees who were responsible for implementing each "activity" necessary to satisfy the Compliance Remedy; and (2) provided a schedule for completing each of those activities. Dr. Bonnie Lesley, the Associate Superintendent for Curriculum and Instruction, and Dr. Ken James, LRSD's Superintendent of Schools, were assigned personal responsibility for each of the twenty-eight (28) activities identified by the "Action Plan Time Line." Id. at 7-10. Thus, it was up to Dr. Lesley and Dr. James to spearhead the timely implementation of all twenty-eight "activities" necessary to satisfy the Compliance Remedy.

Finally, at the same time it approved the Compliance Plan, the Board also adopted Regulation "IL-R1," which set forth "the written procedures for evaluating the programs implemented pursuant to ž 2.7 to determine their effectiveness in improving the academic achievement of African-American students." See Exhibit A at 2 (docket no. 3745).^[7] According to the Compliance Plan, Regulation IL-R1 established the criteria for preparing the program evaluations necessary to satisfy LRSD's obligations under subparts A and B of the Compliance Remedy.

On October 24, 2002, the Board approved a "Program Evaluation Agenda" for the 2002-03 school year that authorized the preparation of evaluations in three broad areas: (1) Elementary literacy; (2) Secondary literacy; and (3) the National Science Foundation ("NSF") K-12 Math and Science Project. See LRSD's Exhibit No. 3; ODM's March 30, 2004 Report on LRSD's Implementation of the Court's Compliance Remedy at 4 (hereinafter referred to as the "ODM's Compliance Report") (docket no. 3854).. LRSD subsequently construed the 2002-03 Program Evaluation Agenda as requiring it to prepare only two ž 2.7.1 evaluations in order to satisfy its obligations under subpart A of the Compliance Remedy: (1) a comprehensive "Literacy Evaluation"; and (2) a comprehensive "Math and Science Evaluation."

Of course, "Literacy" and "Math and Science" are not "programs" they are broad academic areas that roughly correspond to the grouping of college courses into "Arts" or "Sciences." Because LRSD administrators, such as Dr. Lesley, had always construed ž 2.7.1 of the Revised Plan as requiring LRSD to prepare evaluations of the key ž 2.7 programs implemented to improve African-American achievement, the Board should have been aware that they were being too general in dividing the academic universe into "Literacy" and "Math/Science" and then preparing a global evaluation of each of those areas. Nevertheless, in its March 12, 2004 Compliance Report (docket no. 3837), LRSD contends that these two ž 2.7.1 evaluations fully satisfy all of its obligations under subpart A of the Compliance Remedy.^[8]

C. Joshua Invokes Facilitation Provision of Revised Plan.

On November 4, 2002, Ms. Ann Marshall, the Director of the ODM, wrote Joshua's counsel a letter confirming that she: (a) had received their November 1, 2002 letter requesting that she facilitate^[9] the dispute that had arisen between Joshua and LRSD `regarding the adequacy of LRSD's Compliance Plan; and (b) was willing to facilitate that dispute.^[10] Thereafter, I was never contacted by Joshua's counsel or anyone else about the outcome of the ODM's facilitation efforts. Because subpart D of the Compliance Remedy obligated Joshua's counsel to contact me only if the facilitation was unsuccessful, I concluded from the parties' silence (and the ODM's silence) that the ODM had successfully resolved this dispute.

On November 6, 2002, I wrote Ms. Marshall to reinforce the crucially important role I expected her to play in monitoring LRSD's compliance with its obligations under the Compliance Remedy.^[11] The instructions to her were:

It seems to me it would be best if you worked with the parties toward implementing the remedies; but you should feel free to contact me in writing if a serious impasse develops. In other words, as long as everything is going along smoothly, I see no reason for you to make regular reports to me in this respect. I emphasize, however, that you should feel free to call on me if serious problems arise.

Neither the parties nor the ODM ever contacted me to request that I become involved in resolving any compliance disputes or other "serious impasses" between LRSD and Joshua, despite the requirement for such contact by subpart D of the Compliance Remedy and my November 6, 2002 letter.

D. The Six Completed or Substantially Completed Page 148 Evaluations.

The September 13 Decision stated that the Court "will accept all program evaluations that have already been completed by Dr. Nunnery or someone with similar qualifications and approved by the Board." LRSD, 237 F.Supp.2d at 1088. By the time I entered that decision, LRSD had completed or substantially completed six of the fourteen Page 148 Evaluations. These six evaluations covered the following programs: (1) Pre-K-3 Literacy; (2) NSF Math and Science Project; (3) Charter Schools; (4) English as a Second Language; (5) SEDL^[12] Program at Southwest Middle School; and (6) Collaborative Action Team Project. By December 31, 2002, LRSD's Board had approved all six of these Page 148 Evaluations.

E. The Eight Remaining Page 148 Evaluations.

LRSD contracted with Dr. Steven Ross, a program evaluation expert and a member of the faculty at the University of Memphis, to prepare guidelines for completing or revising the eight remaining evaluations.^[13] These eight evaluations covered the following programs: (1) Middle Schools; (2) Extended Year Schools; (3) HIPPY;^[14] (4) Campus Leadership Teams; (5) Summer SchoolÔÇöElementary; (6) Lyceum Scholars Program; (7) Onward to ExcellenceÔÇöWatson Elementary; and (8) Vital Link.

In late November of 2002, Dr. Ross prepared a document captioned "Guidelines for Completing Eight Program Evaluations in the LRSD."^[15] On January 10, 2003, LRSD contracted with Dr. Ross and two other, program evaluation experts, Dr. William Moore and Dr. Larry McNeal, to prepare these eight program evaluations. Dr. Ross prepared or completed evaluations on Vital Link, Onward to Excellence, HIPPY, and Campus, Leadership Teams; Dr. Moore prepared or completed evaluations on Middle School Transition and Extended Year Education; and Dr. McNeal completed evaluations on Lyceum Scholars and Elementary Summer School,

On March 14, 2003, LRSD filed all fourteen of the Page 148 Evaluations with the Court, as required by subpart C of the Compliance Remedy. See LRSD's Notice of Filing Program Evaluations (docket no. 3745). The six evaluations, which were substantially completed as of September 13, 2002, are bound together in Volumes I and II. The remaining eight Page 148 Evaluations are bound together in Volumes III and IV.^[16]

On April 14, 2003, Joshua filed "Comments on the Submission of Page 148 Evaluations." (Docket no. 3752.) These comments identify and discuss numerous alleged "deficiencies," most of which are contained in the eight Page 148 Evaluations that were completed and approved by the Board after January 1, 2003.

F. Dr. Lesley and Dr. James Resign.

On March 14, 2003, the same day LRSD filed the fourteen Page 148 Evaluations,^[17] Dr. Lesley, who was responsible for overseeing the preparation of those evaluations, resigned, and, two months later, Dr. James, LRSD's Superintendent, also resigned. As indicated previously, LRSD's Compliance Plan had assigned Dr. Lesley and Dr. James direct responsibility for each of the twenty-eight time line activities necessary for LRSD to implement the Compliance Remedy.

Thus, less than a year after the September 13 Decision, LRSD lost both of the crucially important leaders to whom all responsibility had been assigned for implementing LRSD's Compliance Plan. Both Dr. James and Dr. Lesley had been deeply involved for years in LRSD's implementation of its desegregation obligations under the Revised Plan, and both were thoroughly familiar with the intricacies of satisfying judicially imposed desegregation obligations. The almost simultaneous departures of Dr. James and Dr. Lesley during the early stages of LRSD's implementation of its Compliance Plan clearly created problems for LRSD in its compliance efforts. In the ODM's Compliance Report, the authors observe that the loss of Dr. Lesley and Dr. James at a crucial time in the implementation of LRSD's Compliance Plan, and the delays and difficulties LRSD encountered in filling those positions with "acting" or "interim" employees created "a period of some uncertainty" for LRSD. Id. at 5.

In June of 2003, LRSD appointed an Interim Superintendent to replace Dr. James. Later that month, on June 26, 2003, Mr. Dennis Glasgow, who previously had been the Director of LRSD's Math and Science Department, was appointed to succeed Dr. Lesley as Interim Associate Superintendent of Instruction and Curriculum. According to the Compliance Plan's "Action Plan Time Line," Mr. Glasgow originally was assigned responsibility for only two of the twenty-eight time line activities. Although it was not mentioned in LRSD's Compliance Report (docket no. 3837), during the recent compliance hearing, Mr. Glasgow testified that, on March 18, 2003, he assumed responsibility for all of Dr. Lesley's twenty-eight activities under the Compliance Plan.

In August of 2003, after the Interim Superintendent, hired only three months earlier, left that position, LRSD hired. Dr. Morris Holmes as the second Interim Superintendent. Dr. Holmes has continued to serve as Interim Superintendent through the date of this Memorandum Opinion.^[18] LRSD's Compliance Report does not address what role, if any, Dr. Holmes had in implementing the Compliance Plan. It seems that Dr. Holmes or someone else should have been assigned responsibility for the twenty-eight time line activities that had been assigned to Dr. James in the October 10, 2002 Compliance Plan; but the Compliance Report is silent on this point.

G. LRSD Files Its Compliance Report Seeking Release from Court Supervision.

When LRSD filed its Compliance Report (docket no. 3837) on March 12, 2004,^[19] it was supported by documents attached to it as Exhibits A through G. According to LRSD, the documents attached to the Compliance Report establish that it has substantially complied with the Compliance Remedy and is entitled to be released from all further court supervision and monitoring.

A careful reading of the Compliance Report reveals a number of revelations that cast LRSD's efforts in a less than favorable light. First, the Report states that: (a) in October of 2002, less than sixty days after the Court's September 13 Decision, Joshua's counsel "raised concerns about the Board-approved Compliance Plan"; (b) subsequently, Joshua's counsel "invoked the Process for Raising Compliance Issues set forth in Revised Plan ž 8.2" and "[the parties] met with Ms. Ann Marshall to facilitate an agreement"; and (c) the parties' last such meeting with Ms. Marshall was on February 28, 2003, but the parties failed to reach an agreement on Joshua's objections to the adequacy of the Compliance Plan. See Compliance Report (docket no. 3837) at 1-2. Thus, LRSD's March 12, 2004 Compliance Report constituted my first notice that Ms. Marshall's facilitation efforts in late 2002 had failedÔÇöneither the parties nor the ODM brought this to my attention, although they were required to do so by subpart D of the Compliance Remedy and by my November 6, 2002 letter to Ms. Marshall.

Second, LRSD contends that, by failing to bring the parties' disagreement over the Compliance Plan to the Court's attention, "Joshua waived any objections to the Board-approved Compliance Plan." Id. at 1-2.

As I explained earlier, the sole purpose of subpart D of the Compliance Remedy was to require the parties to bring potential compliance problems to my attention, as soon as they arose, so that I could resolve them early enough to prevent them from becoming stumbling blocks to LRSD's compliance with its obligations under the other subparts of the Compliance Remedy. After the ODM's facilitation efforts failed, it must have soon become obvious to LRSD's counsel and the ODM that Joshua's counsel was not going to notify the Court. It seems to me that, at this point, both LRSD and ODM should have realized that it was in LRSD's best interest to let me determine, in March of 2003ÔÇöwhile there was still time to do something about itÔÇöif the compliance issues raised by Joshua had merit.

In a school desegregation case that has its origins in the infamous 1957 Little Rock school desegregation crisis, no court is likely to hold the silence of Joshua's counselÔÇöeven if they are to be criticizedÔÇö against the African-American students they represent, and who now fill almost 70% of the total number of seats in LRSD's classrooms. I believe I would be ill advised to adopt "waiver" as a way to avoid reaching the merits of the adequacy of the board-approved Compliance Plan. In other words, I find that it would be inappropriate for me to default approximately 17,000 LRSD students.

Third, LRSD acknowledges in the Compliance Report (docket no. 3837) that it was required to do two things to satisfy the core obligations imposed by subparts A and B of the Compliance Remedy: (1) "develop written procedures for evaluating the programs implemented pursuant to ž 2.7 of the Revised Plan to determine their effectiveness in improving the academic achievement of African-American students"; and (2) "prepare a comprehensive program evaluation of each academic program implemented pursuant to ž 2.7 of the Revised Plan to determine its effectiveness in improving the academic achievement of African-American students and to decide whether to modify or replace the program." See Compliance Report at page 2, paragraph 4 (docket no. 3837; LRSD's Exhibit No. 14) (emphasis added).

To satisfy this first core obligation, which is contained in subpart B of the Compliance Remedy, the Compliance Report states that the Board adopted Regulation IL-R1. As indicated previously, the Board adopted Regulation IL-R1 on October 10, 2002. But, LRSD did not make this key Regulation an Exhibit to either its March 14, 2003 Notice of Filing Evaluations Pursuant to Paragraph C of the Compliance Remedy (docket no. 3745) or its March 12, 2004 Compliance Report (docket no. 3837). On May 12, 2004, after I had been unable to locate this document anywhere in the record, I entered an Order (docket no. 3864) requiring that LRSD provide a copy of Regulation IL-R1. The next day, LRSD filed its Response (docket no. 3865), which attached Regulation IL-R1 as Appendix I to Exhibit A of that document.

Regulation IL-R1 contained the following procedures that were to be followed in preparing all future ž 2.7.1 program evaluations:

(1) "Write a clear description of the curriculum/instruction program that is to be evaluated, with information about the schedule of its implementation."

(2) "Agree on the necessary research questions that need to be established in addition to the question: `Has this curriculum/instruction program been effective in improving and remediating the academic achievement of Mrican-American students?'"

(3) "If program modifications are suggested, the steps that the staff members have taken or will take to implement those modifications. If abandonment of the program is recommended, the steps that will be taken to replace the program with another program with more potential for the improvement and remediation of African-American students. (See ž 2.7.1 of the Revised Designation and Education Plan and Judge Wilson's Compliance Remedy.)"

(4) "Plan ways to provide regular progress reports (e.g. dissemination of meeting minutes, written progress reports, oral reports to Superintendent; Cabinet and/or Compliance Team) to stakeholders, including the Associate Superintendent for Instruction, the Superintendent of Schools, the ODM (until Unitary Status is achieved) and the Joshua Intervenors (until Unitary Status is achieved)."

(5) The team preparing the program evaluations had to meet "to monitor the completion of assignments"; "to review drafts and provide feedback to the writer"; and "to formulate recommendations . . for program improvement, especially to decide if a recommendation is required to modify or abandon the program if the findings reveal that the program is not being successful for the improvement of African-American achievement."

(6) A final draft of the program evaluation had to be submitted to the Associate Superintendent for Instruction "at least one month before placing the report on the Board's agenda for review and approval."

(7) After the program evaluation was approved by the Board, a copy of the complete report had to be made available to "the ODM and Joshua Intervenors (until Unitary Status is achieved)."

(8) "Each program evaluation team shall meet with the Associate Superintendent for Instruction after the completion of the work to evaluate the process and product and to make recommendations for future program evaluation."

See Regulation IL-RI at 3-7 (emphasis added).

In paragraph 8 of the Compliance Report, LRSD states that it satisfied the second core obligation specified in subpart A of the Compliance Remedy, by doing the following things:

(1) Dr. Steve Ross was hired "to prepare evaluations of the District's elementary and secondary literacy programs." Ultimately, Dr. Ross authored the global "Literacy Evaluation," which the Board approved in November of 2003. See Exhibit F to Compliance Report (docket no. 3837).^[20]

(2) Dr. Don Wold, a retired member of the faculty at UALR, authored the global "Math and Science Evaluation," which covered the overall math and science curricula (grades K-12) that LRSD had implemented with the grant it received from the NSF. The Board approved this evaluation in December 2003. See Exhibit G to Compliance Report (docket no. 3837).

According to the Compliance Report, these two global evaluations satisfy all of LRSD's obligations under subpart A of the Compliance Remedy and constitute "substantial compliance" with the obligations contained in ž 2.7.1 of the Revised Plan. Ominously, the Compliance Report says nothing about whether these two evaluations complied with the mandatory requirements of Regulation IL-R1.

H. The ODM's Compliance Report on LRSD's Implementation of the Compliance Remedy.

On March 30, 2004, the ODM filed its Compliance Report (docket no. 3854)^[21] commenting on various aspects of LRSD's implementation of the Compliance Remedy. The first four pages of the ODM's Compliance Report re-plow the now largely irrelevant events related to LRSD's earlier compliance efforts under the Revised Plan.^[22] The remaining nineteen pages of the Compliance Report contain "Findings and Conclusions" that, for the most part, criticize LRSD's compliance efforts. Among the more significant "Findings and Conclusions" in the ODM's Compliance Report are the following:

(1) As of September 13, 2002, the date the Court imposed the Compliance Remedy, LRSD had implemented at 'least forty-six programs designed to improve African-American achievement under ž 2.7 of the Revised Plan.^[23]Id. at 10.

(2) The ODM had difficulty getting LRSD administrators to identify the specific ž 2.7 programs that would be evaluated under ž 2.7.1 of the Revised Plan. LRSD administrators finally acknowledged to the ODM that "the program evaluation agenda for the 2002-03 school year would include only elementary literacy, secondary literacy, and the NSF math and science project." LRSD administrators never explained to the ODM which ž 2.7 programs would be covered in the literacy and math and science evaluations. Id. at 12.

(3) While the Compliance Remedy directed LRSD to use "all available testing data" in assessing the effectiveness of the ž 2.7 programs, LRSD only did so in the Evaluation of the Math and Science Programs. (Exhibit G to LRSD's Compliance Report.) In the Literacy Program Evaluation (Exhibit F to LRSD's Compliance Report), LRSD limited the testing data to "the SAT9 and the benchmark literacy exams." Id. at 14.

(4) Subpart B of the Compliance Remedy required LRSD to maintain written records reflecting "the results of the annual assessments of each program, including whether the assessments resulted in program modifications or the elimination of programs." LRSD, 237 F.Supp.2d at 1087-88. The ODM concluded that, while Exhibit C to LRSD's Compliance Report describes the annual program modifications for elementary and secondary literacy and math, there is no discussion of annual program modifications for science.^[24]Id. at 15.

(5) The ODM's Compliance Report noted numerous shortcomings in the Literacy Program Evaluation. Id. at 1677. However, the most serious criticism was that "the evaluation draws no conclusions about the extent to which student performance might be affected by program components, such as Reading Recovery or Accelerated Reader, nor does it correlate any teaching practices with student achievement." Id. at 18.

(6) The ODM's most serious criticism of the Math and Science Evaluation was that it "does not offer data relative to the level of uniformity of program implementation" and does not identify which ž 2.7 programs most directly improved the academic achievement of African-American students in math and science courses. Id. at 19.

(7) By the time the Court entered its September 13 Decision, six of the fourteen Page 148 Evaluations had already been completed or substantially completed by outside consultants. Subpart C of the Compliance Remedy provided that "the Court will accept all program evaluations that have already been completed by Dr. Nunnery or someone with similar qualifications and approved by the Board." Between October 24, 2002, and December 19, 2002, the Board voted to approve these six evaluations which covered the following programs: (a) Pre K-3 Literacy; (b) NSF Math and Science; (c) Charter School; (d) English as a Second Language; (e) SEDL ProgramÔÇöSouthwest Middle School; and (f) Collaborative Action Team. The ODM's Compliance Report does not contain any significant criticism of these six evaluations. Id. at 20.

(8) As of September 13, 2002, eight of the Page 148 Evaluations still remained to be prepared. Under subpart C of the Compliance Remedy, LRSD was obligated to hire outside consultants to prepare those eight evaluations and then file them with the Court by March 15, 2003. The eight, programs that were the subject of these evaluations were as follows: (a) Middle Schools Implementation; (b) Extended Year Schools; (c) HIPPY; (d) Campus Leadership Teams; (e) Elementary Summer School; (f) Lyceum Scholars Program; (g) Onward to ExcellenceÔÇö Watson Elementary; and (h) Vital Link. Id. The last four of these programs had already been discontinued by the time I entered the September 13 Decision. Thus, only the evaluations on the first four of these programs (Middle Schools program, Extended Year Schools program, HIPPY program, and Campus Leadership Teams program) had any potential to be useful to LRSD.^[25]

(9) All eight of the Page 148 Evaluations prepared by outside consultants were heavily criticized by the ODM. The cited source for much of this criticism was Dr. Ross, who prepared two of the four evaluations for ongoing programs and two of the four evaluations for discontinued programs. According to the ODM's Report, Dr. Ross told the ODM that: (a) "[t]he evaluations of the subject programs `were worthless,' both as originally prepared and currently" and that the discontinued programs he was hired to evaluate were "dead wood"; (b) he did not understand why LRSD had declined to provide additional updated information and data on the four ongoing programs "in order to facilitate more effective evaluations"; and (c) "[t]he evaluations were of little or no use to the district."

(10) Echoing Dr. Ross's criticism of the eight Page 148 Evaluations, the ODM observed that: "[LRSD's] asking the outside evaluators to rewrite the evaluations using the same data from the original, unacceptable versions [prepared internally by Dr. Lease] is similar to asking mechanics to tear down a Yugo and put it back together as a Lexus." Id. at 21-22. The ODM concluded this section of the Report by noting that: "Sadly, the evaluations are so flawed that they reveal little of substance that would enable the district to draw conclusions about the programs, ascertain the effectiveness of their component parts, point out appropriate modifications, or decide whether to drop a program altogether." Id. at 22. (The direct charge to the ODM was to work closely with LRSD to help to ensure it met its obligations under the Compliance Remedy. Thus the ODM's stinging criticism of LRSD's compliance efforts is, at least in part, an admission of its own shortcomings. It seems to me that the ODM should have been less harsh in its criticismÔÇöit should attempt to pour oil on troubled waters, when reasonably possible.)

I. Joshua's Opposition to LRSD's Compliance Report.

On April 15, 2004, Joshua filed an Opposition To LRSD's Motion to Be Released from Further Supervision and Monitoring of Its Desegregation Effort (docket no. 3856), along with a supporting Memorandum (docket no. 3857). Joshua begins their Memorandum by framing the compliance issue with the following quotes from the September 13 Decision:

I find that the purpose of ž 2.7.1 was to make sure that the programs under ž 2.7 actually worked to improve the academic achievement of African-American students. I further find that LRSD's substantial compliance with ž 2,7.1 was crucial to its commitment to improve the academic achievement of African-American students; for, without performing rigorous annual assessments of each of the many dozens of programs implemented under ž 2.7, it would be impossible to determine which programs were working and should be continued and which programs were not working and should be discontinued, modified, or replaced with new programs.

* * * * * *

I conclude that the Court should continue supervision and monitoring of LRSD's compliance with this crucially important section of the Revised Plan in order to ensure that LRSD has in place an effective assessment program that will allow it to identify and improve those programs that are most effective in remediating the academic achievement of African-American students.

LRSD, 237 F.Supp.2d at 1076, 1086.

Joshua's Memorandum goes on to discuss the following areas in which they allege that LRSD has failed to substantially comply with its obligations under subparts A, B, and C of the Compliance Remedy:

(1) During the entire 2002-03 school year and the first semester of the 2003-04 school year, the Compliance Remedy obligated LRSD to assess each of the dozens of programs it implemented pursuant to ž 2.7 of the Revised Plan in order to determine if those programs were effective in improving the academic achievement of African-American students. However, in the two global program evaluations, attached as Exhibits F and G to its March 12, 2004 Compliance Report, LRSD does not identify with, any degree of precision which of the dozens of ž 2.7 programs are included in each evaluation, much less address the effectiveness of each of those programs in improving African-American achievement. Joshua's Memorandum (docket no. 3857) at 2-3.

(2) Regulation IL-R1 explicitly required each ž 2.7.1 program evaluation to set forth and' answer the following crucially important research question: "Has the curriculum/instruction program been effective in improving and remediating the academic achievement of African-American students?" See Appendix 1 to Exhibit A at  6(D) (docket no. 3865). However, this mandatory research question is not included among the research questions set forth in the Literacy Program Evaluation.^[26] Joshua's Memorandum (docket no. 3857) at 4.

(3) Both the Literacy Evaluation and Math and Science Evaluation contain only broad generalizations about student achievement based solely on standardized test data, rather than a specific analysis of which of the dozens of ž 2.7 programs actually worked in improving African-American achievement in the areas of literacy, math, and science and which of those programs need to be modified or eliminated. Id. at 5-6.

(4) Regulation IL-R1 explicitly required that: (a) a program evaluation include, as part of its "program documentation," a clear and accurate description of the program being evaluated (see Appendix 1 at 3 to Exhibit A of docket no. 3865); and (b) the team leader ensure that each program evaluation contain "a clear description of the curriculum/instruction program that is to be evaluated . (See Appendix 1 to Exhibit A at page 5, paragraph 6(c) (docket no. 3865).) According to Joshua, the Literacy Evaluation contained only a cursory description of a few of the ž 2.7 programs implemented to improve the literacy of African-American students and the Math and Science Evaluation contains only a list of the math and science courses that were available to LRSD students at various grade levels. Id. at 6-8.

(5) LRSD's Page 148 Evaluations, filed on March 13, 2003 (docket no. 3745), are inadequate because: (a) they do not satisfy the standards for program evaluations contained in Regulation IL-R1; (b) according to Dr. Ross, who was involved in preparing those evaluations, they were "for the most part . . . worthless . . . [and] of little or no use to the district" (see ODM's Compliance Report at 21); and (c) LRSD failed to use the Page 148 Evaluations, as part of the assessment process, to determine the effectiveness of the ž 2.7 programs in improving African-American achievement, as required by subpart C of the Compliance Remedy. Id. at 8-9.

(6) Regulation IL-R1 provided that the team leader for each evaluation shall "provide regular progress reports (e.g., dissemination of meeting minutes, written progress reports, oral reports to the superintendent's cabinet and/or compliance team) to stakeholders, including . . . the ODM (until unitary status is achieved), and the Joshua Intervenors (until unitary status if achieved)." According to Joshua, LRSD failed to provide them or the ODM with the progress reports on the preparation of the program evaluations, as required by Regulation IL-R1.

J. LRSD's Proposed Compliance Plan and Proposed Regulation IL-R2 Finally Surface.

On May 12, 2004, I entered an Order (docket no. 3864) asking LRSD to immediately provide me with a copy of Regulation IL-R1. In its May 13, 2004 Response (docket no. 3865), LRSD attached a copy of IL-R1,^[27] along with the two documents that were not requested:

(1) LRSD's "Proposed Compliance Plan."^[28]

(2) LRSD's `Proposed IL-R2."^[29]

LRSD does not explain why it decided to bring these last two documents to my attention at this late date in the compliance process.

The "Proposed Compliance Plan," which was prepared within thirty days after the entry of the September 13 Decision, contains numerous questions about the meaning of various terms used in the Compliance Remedy. For example, the Compliance Committee states that they are "confused" by the Court's use of the term "assessment" in subparts A and B of the Compliance Remedy, something that required "the Compliance Committee to determine the District Court's intended meaning."^[30]See Proposed Compliance Plan at 3-4. Additionally, this document contains a lengthy discussion about the meaning of certain obligations in subpart C of the Compliance Remedy, which required LRSD to complete and file the eight unfinished Page 148 Evaluations by March 15, 2003. Id. at 7-8.

As previously discussed, on October 10, 2002, the Board approved the final "Compliance Plan," which LRSD attached as Exhibit A to its March 14, 2003 Notice Of Filing Program Evaluations Required By Paragraph C Of The Compliance Remedy (docket no. 3745). LRSD's May 13, 2004 Response does not explain why none of the many questions raised in the Proposed Compliance Plan were mentioned in the final Compliance Plan, which was filed on March 14, 2003. It seems clear to me that these questions should have been brought to my attention when they were first put on paperÔÇöroughly a year and a half ago.

"Proposed Regulation IL-R2" was intended to govern the preparation and content of "Informal Program Evaluations." The stated purpose of this regulation was "to ensure that a written record exists explaining a decision to significantly modify an academic program." See Appendix 2 to docket no. 3865. On page 6 of the "Proposed Compliance Plan" (Exhibit A to docket no. 3865), it noted that Proposed Regulation IL-R2 "was drafted to address" the following specific concern that was raised in the September 13 Decision:

I have grave reservations about anyone this side of Solomon being wise enough to use two or three semesters' worth of erratic composite test scores to make reliable decisions about which remediadon programs for LRSD's African-American students are actually working.

LRSD, 237 F.Supp.2d at 1079. According to the "Proposed Compliance Plan," Proposed Regulation IL-R2 satisfied my concern by "prohibiting substantial program modifications from being made without a written record as required by Paragraph B [of the Compliance Remedy]."

Proposed Regulation IL-R2 clearly was intended to meet certain of the obligations imposed on LRSD under subpart B of the Compliance Remedy. LRSD, 237 F.Supp.2d at 1087-88. Yet, LRSD's May 13, 2004 Response (docket no. 3865) does not explain why the Board failed to adopt Proposed Regulation IL-R2. Likewise, the ODM's Compliance Report sheds no light on why the Board did not adopt Proposed Regulation IL-R2. Instead, the ODM merely notes in passing that, after drafting Proposed Regulation IL-R2, LRSD "ultimately decided [it] was unnecessary and did not adopt [it]."

It appears now that, from the very outset of its efforts to fulfill its obligations under the Compliance Remedy, LRSD was badly confused about what it was required to do. It is hard to understand why this confusion was not brought to my attention. I would have been more than happy to explain precisely what I meant in the Compliance Remedy if this matter had been brought to my attention. Be all of this as it may, LRSD now finds itself in the position of arguing that it has substantially complied with desegregation obligations that its Compliance Committee found to be confusing and difficult to understand. And this means, of course, that LRSD's "puzzlement" was unknown until it was too late for it to cure the problems.

V. Compliance Hearing on LRSD's Request for Release from Court Supervision and Monitoring

On June 14 and 15, 2004, I conducted a compliance hearing on LRSD's request "that the Court find that LRSD has substantially complied with Revised Plan ž 2.7.1, as specified in the Compliance Remedy; that LRSD is unitary with regard to all aspects of [its] school operations; and that it be released from all further supervision and monitoring of its desegregation effort." See Compliance Report at 4 (LRSD's Exhibit No. 14). LRSD called the following witnesses to testify in support of its having met its obligations under the Compliance Remedy: (1) Dr. Bonnie Lesley, LRSD's former Associate Superintendent for Curriculum and Instruction; (2) Dr. Steven M. Ross, the Director of the Center for Research in Education Policy at the University of Memphis; (3) Ms. Vanessa Cleaver, LRSD's Director of the NSF Grant; and (4) Mr. Dennis Glasgow, LRSD's Interim Associate Superintendent for Curriculum and Instruction.

After LRSD rested, Joshua moved for Judgment as a Matter of Law, pursuant to Fed.R.Civ.P. 52(c), on the ground that LRSD had failed to establish a legally sufficient evidentiary basis for finding that it had substantially complied with its obligations under subparts A, B, and C of the Compliance Remedy. Ruling from the bench, I granted Joshua's Motion, in part, by finding that LRSD had failed to establish a sufficient evidentiary basis to support its contention that it had substantially complied with its obligation under subparts A and B of the Compliance Remedy. However, I denied Joshua's Motion with respect to LRSD's proof that it had substantially complied with it obligations under subpart C of the Compliance Remedy.

Joshua's counsel then stated, on the record, that they were withdrawing their challenge to LRSD's substantial compliance with its obligations under subpart C of the Compliance Remedy. For that reason, Joshua called no witnesses. After explaining to counsel that I intended to prepare detailed Findings of Fact and Conclusions of Law to support my ruling on Joshua's Rule 52(c) Motion, I adjourned the compliance hearing.

VI. Findings of Fact and Conclusions of Law

A. Burden of Proof

1. Section 11 of the January 16, 1998 Revised Plan (CX 871) explicitly provided that "any party challenging LRSD's [substantial] compliance [with its desegregation obligations] bears the burden of proof." Because Joshua was the "challenging party" in the 2001-02 unitary status hearings, I held that Joshua had the burden of proof in establishing that LRSD had failed to substantially comply with its desegregation obligations in each of the challenged areas. LRSD, 237 F.Supp.2d at 1033-35. Ultimately, Joshua met their burden of proof only with regard to LRSD's failure to substantially comply with its obligations under ž 2.7.1 of the Revised Plan.

2. The Compliance Remedy required LRSD to fulfill its obligations under ž 2.7.1 of the Revised Plan. Thus, the issue now before me is whether LRSD has substantially complied with my judicial remedyÔÇönot the previously litigated question of whether LRSD substantially complied with its obligations under the Revised Plan.

3. While ž 11 of the Revised Plan contained the parties' binding contractual agreement on the allocation of the burden of proof, for purposes of the unitary status hearings, that section of the Revised Plan no longer controls my decision on whether LRSD has met its obligations under the Compliance Remedy. It is black letter law that a school district seeking an end to court supervision has the burden of proving substantial compliance with the judicially imposed remedy. Freeman v. Pitts, 503 U.S. 467, 494, 112 S. Ct. 1430, 118 L. Ed. 2d 108 (1992); Belk v. Charlotte-Mecklenburg Bd. of Educ., 269 F.3d 305, 318 (4th Cir.2001); Jenkins v. Missouri, 216 F.3d 720, 725 (8th Cir.2000). Thus, I conclude that LRSD has the burden of proving by a preponderance of the evidence that it has substantially complied with each of the obligations contained in subparts A, B, and C of the Compliance Remedy.

B. LRSD's Efforts to Substantially Comply with Subparts A and B of the Compliance Remedy.

1. Throughout this litigation, standardized test scores have revealed that the academic achievement of a high percentage of LRSD's African-American students lags far behind their white counterparts. In the September 13 Decision, I discussed this "achievement gap" at some length and summarized LRSD's efforts to narrow this gap by implementing dozens of ž 2.7 programs, between 1998 and March of 2001, that were designed to remediate African-American achievement. See LRSD, 237 F.Supp.2d at 1070-72.

2. Importantly, in the September 13 Decision, I rejected Joshua's argument that ž 2.7 of the Revised Plan obligated LRSD to eliminate the achievement gap between African-American and white students.^[31] As a result, I found that the disparity in standardized test scores between African-American and white studentsÔÇöwhile "far from where they need to beÔÇöfailed to prove that LRSD has not lived up to its obligations under ž 2.7 of the Revised Plan." Id. at 1017.

3. Of even greater importance, however, I also found that ž 2.7.1 of the Revised Plan required LRSD to assess annually each of the key ž 2.7 programs designed to improve African-American achievement in order to ensure that those programs are effective. It is impossible to overstate the importance of ž 2.7.1 to LRSD's African-American students. Unless something is done to improve their academic achievement, many of them, who do not possess proficient skills in reading and math, will face difficult and uncertain futures. Because 70% of its students are African-American, LRSD should be devoting a substantial percentage of its educational resources to solving this crucially important problem that will burden the lives and career trajectories of so many of its students. It is my fervent hope that LRSD's administrators and its Board realize that LRSD must make the long-term commitment to solve this problem, not because a federal court says that it must, but because it is the right thing to do.

4. In the September 13 Decision, I held that: (a) LRSD's efforts to comply with ž 2.7.1 of the Revised Plan had fallen well short of the mark; (b) LRSD's obligations under ž 2.7.1 were "crucial to its commitment to improve the academic achievement of African-American students"; and (c) Court supervision and monitoring of LRSD must continue until it complies with "this crucially important section of the Revised Plan in order to ensure that LRSD has in place an effective assessment program" that is capable of determining not only the specific ž 2.7 programs that are most effective, but also the programs that need to be improved or eliminated. Id. at 1017, 1086. I find that the clear language of the September 13 Decision made it plain that I expected LRSD to devote all of the time and resources necessary to comply with the spirit and letter of the commitment contained in ž 2.7.1 of the Revised Plan. I further find that LRSD knew, or certainly should have known, that, in order to honor its commitment under ž 2.7.1, it was required to evaluate the key ž 2.7 programs.

5. Subpart A of the Compliance Remedy provides that LRSD must devise and implement a comprehensive process for assessing, on a year-to-year basis, the effectiveness of each of the key ž 2.7 programs in remediating the academic achievement of African-American students. Because LRSD's battle to improve the academic achievement of African-American students may go on for many years, the comprehensive process for assessing ž 2.7 programs must become a deeply embedded part of LRSD's elementary and secondary curriculum. Only then can I have the necessary assurance that LRSD will dutifully continue the comprehensive program assessment process (which also must include periodic program evaluations) until the need for ž 2.7 programs no longer exists. I assumedÔÇöI now realize incorrectlyÔÇöthat LRSD would recognize that part of its burden of proof would include demonstrating that an effective program assessment process has been implemented and deeply embedded in its curriculum.

6. Because the ž 2.7.1 program assessment process must be an essential part of LRSD's curriculum for the foreseeable future, I thought LRSD would understand that it must hire a highly qualified Assistant Superintendent to replace Dr. Lease as the head of its Department of Planning, Research, and Evaluation ("PRE") and reinvigorate PRE with a team of highly trained professionals capable of devising a systematic process for assessing the ž 2.7 programs; identifying the key ž 2.7 programs that required evaluations and overseeing the preparation of those evaluations; making annual determinations about the effectiveness of the ž 2.7 programs; and structuring and implementing the required changes in those programs in order to make them more effective. LRSD did not find a qualified replacement for Dr. Lease, and it allowed its PRE to cease operating as a viable department; so, LRSD did not formulate an effective program assessment process, and did not deeply embed that process as part of its curriculum.

7. During the recent compliance hearing, the evidence established that, after Dr. Lesley's departure, LRSD failed to hire someone with similar qualifications and extensive experience in preparing and, analyzing program evaluations to serve in the crucially important position of Associate Superintendent of Curriculum and Instruction. Mr. Glasgow, the acting Associate Superintendent of Curriculum and Instruction, admitted that PRE was "short on personnel" at the present time and that he intended to propose to the Board that it set a high priority on hiring a team of well qualified and experienced professionals capable of reinvigorating PRE. In addition to hiring a new director of PRE, Mr. Glasgow testified that LRSD needed to hire a "research specialist" with extensive experience in preparing and overseeing preparation of program evaluations. This is something it should have done immediately after I entered the September 13 Decision. This would have allowed LRSD to use its PRE Department to oversee the preparation of the key ž 2.7 program assessments by outside consultants, such as Dr. Ross, and to perform the "heavy lifting" necessary to deeply embed an effective program assessment process as a pertinent part of LRSD's curriculum for the foreseeable future.

8. LRSD should have included in its program assessment process everything necessary for it to make an informed decision about whether each of the key ž 2.7 programs designed to improve African-American achievement was actually working to accomplish that objective. While I am not a professional educator, common sense dictates that some of the more important elements of an effective program assessment process should include the following: (a) program evaluations specifically designed to determine whether key ž 2.7 programs, as implemented, are actually working at each school to improve the academic achievement of African-American students; (b) input from a broad cross section of the teachers responsible for implementing each of the key ž 2.7 programs-about the effectiveness of these programs at their respective schools and whether a ž 2.7 program should be modified or eliminated; (c) an assessment of whether each of the key ž 2.7 programs is being implemented with the same degree of effectiveness and success at each participating school; (d) an assessment of whether more teaching specialists or other personnel are needed at some schools in order to enhance the effectiveness of the key ž 2.7 program; and (e) administering consistent standardized tests at various grade levels so that the academic achievement of African-American students can be accurately followed, and valid statistics can be compiled regarding their year to year academic achievement.

9. In the Compliance Remedy, I was reluctant to set forth too much detail about how LRSD should structure its program assessment process. Professional educators ought to be able to do a better job than I could in formulating and implementing this process; but LRSD is found wanting in its handling of its duties under subparts A and B of the Compliance Remedy.

10. LRSD did not follow the plain language of subpart A of the Compliance Remedy, and I find this quite puzzling. In any event, it prepared two "comprehensive" evaluations that covered only some of the ž 2.7 programs and that reflected-at some grade levels and on some tests-an improvement in the achievement level of African-American students in the areas of literacy, math, and science. Somehow, LRSD concluded that those two global evaluations would be sufficient to prove the effectiveness of each of the key ž 2.7 programs, as they have been actually implemented, and to satisfy me that it had in place a deeply embedded and effective ž 2.7.1 program assessment process.

11. Having reviewed a mountain of test score results from 1998 through the first semester of the 2003-04 school year, it appears to me that most of LRSD's testing data is highly variable. At some grade levels, in some years, there is improvement, and at other grade levels, in other years, test scores decline. In dealing with a problem that is as challenging as narrowing the achievement gap, such an ebb and flow of test scores over only a few years should come as no surprise. That is why I believe it is so important for LRSD to deeply embed an effective ž 2.7.1 program assessment process in its curriculum and then make certain it has a team of highly trained specialists administering that process for years to come. LRSD should pay close attention to the next point: If it were to implement, properly staff, and deeply embed an effective ž 2.7.1 program assessment process, LRSD could still prove that it had substantially complied with its obligations under ž 2.7.1 and subpart A of the Compliance RemedyÔÇöeven if test results did not establish that there had been much, if any, improvement in the academic achievement of African-American students. This is because ž 2.7.1 does not require an improvement in the academic achievement of African-American studentsÔÇöit requires only that LRSD have in place an effective and deeply embedded program assessment process that is capable of assessing the effectiveness of its key ž 2.7 programs.

12. I find LRSD's Literacy Program Evaluation fails to substantially comply with subparts A and B of the Compliance Remedy. I further find some of the most significant deficiencies in this Evaluation to be as follows:

(a) LRSD's Compliance Plan^[32] provided that:

LRSD will: . . . (4) Prepare a comprehensive program evaluation of each academic program implemented pursuant to Revised Plan ž 2.7 to determine its effectiveness in improving the academic achievement of African-American students and to decide whether to modify or replace the program. . . .

(Emphasis added.) The Literacy Evaluation does not evaluate each ž 2.7 academic program implemented to improve the academic achievement of African-American students in the area of literacy, much less analyze "whether to modify or replace" any of those programs.

(b) Regulation IL-R1 provided that any ž 2.7.1 program evaluation must answer the following research question: "Has this curriculum/instruction program been effective in improving and remediating the academic achievement of African-American students?" See docket no. 3865, Appendix 2 at 5.^[33] The Regulation went on to state that this research question was required to be included in any ž 2.7.1 program evaluation based on: "Policy IL, ž 2.7.1 of the Revised Desegregation and Education Plan and Judge Wilson's Compliance Remedy." Id. at 5. Incredibly, the research questions set forth on page 1 of the Literacy Program Evaluation do not include this essential research question. During the compliance hearing, Dr. Ross testified that, read together, research questions 2 through 6 were aimed at getting at the same information sought in the one essential research question that was required by Regulation IL-R1. Nevertheless, as Dr. Ross conceded, he should have included that research question in the Literacy Program Evaluation in order to comply with Regulation IL-R1.^[34]

(c) Regulation IL-R1 explicitly required the "team leader" of the ž 2.7.1 program evaluation to write "a clear description of the curriculum/instruction program that is to be evaluated, with information about the schedule for its implementation." Id. at 5. The Literacy Program Evaluation does not sufficiently identify the specific ž 2.7 programs that are covered by the evaluation ÔÇö a direct violation of Regulation IL-R1. At the elementary school level (grades K-5), the Literacy Program Evaluation cites "results" from five programs: (a) Early Literacy Learning in Arkansas ("ELLA") for grades K through 2; (b) Effective Literacy for grades 3 through 5; (c) Signature Reading Series for grades 1 through 5; (d) Reading Recovery for 1st graders; and (e) Success For All ("SFA") for grades K through 5. Id. at 10.^[35] At the middle school level (grades 6-9), the Evaluation states only that "all grades participated in Reading and Writing Workshop concepts [whatever that means]," without indicating whether "Reading and Writing Workshop" was a ž 2.7 program or whether there were any other ž 2.7 programs designed to improve the literacy of African-American middle school students. Id. at 11. Finally, at the high school level (grades 9-12), the Evaluation merely cites "block scheduling of English classes to accommodate different levels of learners . . . and teacher professional development . . . based on national research findings, trends, and issues in literacy development [whatever that means]." Dr. Ross conceded that these program descriptions were "amorphous." While Dr. Ross and LRSD administrators may know what these descriptions mean, I do not. Thus, I am left to guess at whether I should consider "block scheduling of English classes" and "teacher professional development" as a ž 2.7 program and wonder if there were any other ž 2.7 programs that LRSD implemented to improve the literacy of African-American high school students. In the future, LRSD should keep in mind that its program evaluations of key ž 2.7 programs need to be written in such a way that they not only comply with Regulation IL-R1, but also are readily understandable to me.

(d) In the "Summary and Conclusions" section, Dr. Ross cites a number of negative comments about the overall effectiveness of LRSD's literacy program. For example, LRSD elementary teachers said in interviews that some of the most effective ž 2.7 reading programs, such as ELLA and Effective Literacy, were "often perceived as separate and distinct entities instead of integral to the district comprehensive literacy program." Id. at 43. Furthermore, the authors noted that: (1) "middle school and high school teachers' comments seemed to indicate that they also did not perceive themselves as being involved in a literacy plan beyond the traditional roles they had as English teachers"; and (2) "teachers' perceptions of the impact of literacy programs were extremely mixed." Id.

(e) Finally, Regulation IL-R1 required LRSD to provide regular "progress reports" on the Program Evaluations to "stakeholders," including the ODM and Joshua. See Joshua's Exhibit No. 2 at 5; docket no. 3864, Exhibit A, Appendix. 1 at 5. During the evidentiary hearing, the testimony established that LRSD failed to provide the ODM and Joshua with any "progress reports" on the Literacy Program Evaluation as required by Regulation IL-R1.

13. I find LRSD's Math and Science Evaluation also falls short of substantial compliance with subparts A and B of the Compliance Remedy. The more obvious deficiencies in this Evaluation are as follows:

(a) As indicated previously, LRSD's Compliance Plan specifically obligated it to "prepare a comprehensive program evaluation of each academic program implemented pursuant to Revised Plan ž 2.7 in order to determine its effectiveness in improving the academic achievement of African-American students and to decide whether to modify or replace the program."^[36] To meet this requirement of its own Compliance Plan, LRSD was obligated to: (1) identify each of the key ž 2.7 math and science programs that were in place during the 2002-03 school year and the first semester of the 2003-04 school year; and (2) prepare a comprehensive program evaluation of each of those academic programs. In violation of its own Compliance Plan, LRSD prepared a comprehensive Math and Science Evaluation that examined "areas of knowledge"ÔÇönot specific ž 2.7 programs. Furthermore, the Evaluation makes no attempt to analyze which of the ž 2.7 math and science programs need to be modified; the nature of the modifications that should be made; and whether any of those programs should be eliminated and replaced with the more effective programs.

(b) The Math and Science Evaluation lists "Milestones" in implementing the "educational system reforms" but only a few ž 2.7 programs are specifically identified, e.g., the Summer Mathematics Advanced Readiness Training Program ("SMART"), THRIVE, a Saturday math academy implemented in collaboration with Philander Smith, SE CME, a program for students interested in engineering, and unspecified "additional support programs in collaboration with the Arkansas Museum of Discovery, UALR, and UAMS." See Exhibit G at 6-9 (docket no. 3837). Furthermore, one of the "Milestones" is described as the implementation of a "high quality standard-base curricula for all students, grades K-12, in math and science." Id. at 9 (emphasis added). After carefully reviewing the long list of "Milestones," I find that the Math and Science Evaluation was intended to cover all curriculum changes made to LRSD's math and science programsÔÇönot the specific ž 2.7 programs that were designed and implemented to improve the academic achievement of African-American students in math and science. Ms. Vanessa Cleaver, the former Project Director of the Comprehensive Partnership for Math and Science Achievement, testified that LRSD prepared the Math and Science Evaluation according to NSF evaluation standards, which differed from those contained in Regulation IL-R1. She also testified that LRSD prepared the Math and Science Evaluation to satisfy the requirements of the NSF grant. I find the purpose and objective of the NSF grant were not the same as subpart A of the Compliance Remedy. Thus, LRSD should not have assumed that satisfying the requirements of the NSF grant would automatically satisfy the requirements of subpart `A of the Compliance Remedy.

(c) The Math and Science Evaluation consists primarily of an analysis of test scores demonstrating that LRSD has made some progress: (1) in increasing the number of African-American students who have moved from the Below Basic Achievement Level in math and science to the Basic Achievement Level; and (2) increasing the enrollment of African-American students in more rigorous math and science courses. However, much less progress has been made in increasing the number of African-American students who are achieving at or above the Proficient Level in math and science. The only statistically supportable conclusion that can be drawn from this Evaluation is that LRSD appears to be doing "something" right in its math and science curriculum because a significant number of African-American students have moved from the Below Basic Level to the Basic Level in math and science. However, the Math and Science Evaluation makes no attempt to address, much less answer, the essential questions that LRSD's own Regulation IL-R1 required it to answer: Which specific ž 2.7 programs are actually working and most effective in improving the academic achievement of African-American students in math and science? What changes need to be made to improve those programs? What programs are not effective and should be eliminated? In her testimony, Ms. Cleaver admitted that the primary focus of the Math and Science Evaluation was on a statistical analysis of test scoresÔÇönot a determination of the effectiveness of specific ž 2.7 programs such as THRIVE and SMART. Finally, Ms. Cleaver acknowledged that the Math and Science Evaluation did not address the effectiveness of the new math and science curriculum, as it had been implemented at each school, in improving the academic achievement of African-American students.

(d) The Stanford Achievement Test is a "norm referenced" test, which means that it compares the achievement of Arkansas students with a national sample of students who took the same exam. As a result, scores are based on national performance quartiles. Thus, if a fifth grade student performs at the 20th percentile in math on the Stanford Achievement Test, it means 80% of the fifth grade students in the country exceeded his or her performance. In contrast, the state Benchmark Exam is a "criterion-referenced" test that was created for use within Arkansas to measure students' mastery of state math and literacy standards. Performance on the Benchmark Exam is measured in four broad categories: Below Basic; Basic; Proficient; and Advanced. According to Mr. Glasgow, students who perform at the Below Basic level have a very limited understanding of the concepts being tested. Mr. Glasgow also testified that it was much easier to improve student test scores on the state Benchmark Exam than on a national "norm referenced" test like the Stanford Achievement Test. It is possible to devise a statistically defensible "equating formula" to allow educators to compare percentiles of performance among "norm referenced" tests. However, in the Math and Science Evaluation, the authors take the extraordinary statistical liberty of trying to convert quartiles of performance on the "norm referenced" Stanford Achievement Test into the four much different categories of performance that are measured on the "criterion-referenced" Benchmark Exam. Furthermore, this appears to have been done to create the false impression that African-American students have improved their performance on the Stanford Achievement Test. In my experience, you can't mix apples and oranges and then claim you've got tangerines. I find the charts and data that are displayed on pages 49-55 to be of no statistical value in determining whether African-American students have improved their level of performance on the Stanford Achievement Test.

(e) Finally, I find that LRSD failed to provide regular "progress reports" on the Math and Science Evaluation to the ODM and Joshua, as required by Regulation IL-R1. See docket no. 3864, Exhibit A, Appendix 1 at 5.

14. Dr. Ross testified that his Literacy Evaluation, which he referred to as a "big picture" evaluation, was the necessary first step in complying with LRSD's obligations under ž 2.7.1 of the Revised Plan. According to Dr. Ross, the global Literacy Evaluation reached a narrow but very important conclusion: The various ž 2.7 literacy programs, which are described on pages 10-11 and 34, are equally "effective" or "ineffective" in improving the academic achievement of African-American students. As Dr. Ross explained, "beauty is in the eye of the beholder." Therefore, some readers of the Literacy Evaluation might conclude that the relatively modest improvement in African-American test scores was sufficient to deem those programs "effective," while other readers might conclude, based upon the same data, that they were "ineffective." Most importantly, however, Dr. Ross testified that his statistical analysis demonstrated that none of the ž 2.7 programs aimed at improving literacy produced an unhealthy outcome for African-American students. By unhealthy outcome, Dr. Ross meant that the test scores for African-American students did not go down.

15. Dr. Ross made it clear that it is now crucially important for LRSD to take the second step and perform program evaluations to determine the effectiveness of the specific ž 2.7 literacy programs as they have been implemented at each of the schools in the district. He testified that the step 2 evaluations were necessary and essential in order for LRSD to determine: (a) whether, as implemented; on a school-by-school basis, the key ž 2.7 programs were effective in improving the academic achievement of African-American students; and (b) whether any program changes needed to be made to improve the effectiveness of the key ž 2.7 programs as they were being implemented at each school. Dr. Ross testified that, without these step 2 evaluations, LRSD cannot continue to make progress in improving the scores of African-American students on the Arkansas Benchmark Exam and the Stanford Achievement Test.

16. In the 2001-02 unitary status hearings, the evidence established that LRSD has a number of elementary schools in which the students are entirely or almost entirely African-American. Likewise, some of LRSD's middle schools and high schools have a disproportionately high number of African-American students compared with other middle schools, and high schools. Only by performing these step 2 evaluations of the key ž 2.7 programs, as they have been implemented on a school-by-school basis, will LRSD be able to determine their effectiveness at the classroom level. Because that is where "the rubber meets the road," I find the data that will be gained from these step 2 evaluations is a vital and essential part of LRSD's assessment obligation under ž 2.7.1 of the Revised Plan and subpart A of the Compliance Remedy.

17. Dr. Lesley testified that, because each of the key ž 2.7 programs that LRSD implemented had already been found to be successful in improving the academic achievement of African-American students in other school districts where those programs were pioneered and developed (e.g., the Chicago School District and the Washington D.C. School District), there was no need to perform step 2 evaluations of the effectiveness of those programs, as implemented in LRSD. However, Dr. Ross testified that his own extensive experience in evaluating virtually identical programs designed to improve the academic achievement of African-American students in different school districts revealed that some programs work well in one district but fail in another. Furthermore, Dr. Ross indicated his own personal experiences with the highly variable success rates of the same programs implemented in different school districts was borne out by all of the research and studies on the subject. Thus, the ELLA literacy program might work in LRSD but fail to achieve results in the Memphis School District. I find Dr. Ross's testimony on this point to be authoritative and credible. I reject Dr. Lesley's testimony that "one size fits all" and that it can be taken on faith that, if LRSD implements a literacy program which was proven to be highly effective in the Washington D.C. elementary schools, the program can be assumed to be equally effective as implemented in LRSD.

18. Dr. Lesley also testified that, in order to meet its obligations under ž 2.7.1 and subpart A of the Compliance Remedy, she believed LRSD was required to perform only annual informal program assessments. She testified that, at the time she left LRSD in March of 2003, she and other LRSD administrators were fully capable of assessing test data, interviewing teachers, and coming up with an informal assessment or evaluation of which ž 2.7 programs were effective and which were not. Even if Dr. Lesley had remained with LRSD, I would have serious doubts about the value and reliability of such informal assessments or evaluations. However, after Dr. Lesley resigned in March of 2003, there is no evidence that any of LRSD's remaining administrators had the experience or training to perform such informal assessments or evaluations.

19. Dr. Ross was emphatic in testifying that LRSD must perform formal evaluations of the key ž 2.7 programs as implemented ÔÇö not informal "assessments" or informal program evaluations. He testified that it is easy for "informal evaluations" never to be performed, or, worse yet, result in bad statistical analysis that leads a school district to claim test score improvements that, in reality, do not exist.

20. LRSD's counsel asked Mr. Glasgow, the interim Superintendent of Instruction and Curriculum, if he agreed that requiring African-American students to take more rigorous academic courses was the most effective way to improve African-American achievement. Obviously, counsel expected a "yes" answer, but Mr. Glasgow responded "No." He testified that the single best way to improve the academic achievement of African-American students was to evaluate, on a school-by-school basis, the effectiveness of specific ž 2.7 programs, as they have been implemented at the classroom level. By way of example, Mr. Glasgow referred to Reading for All, one of the key ž 2.7 literacy programs. Mr. Glasgow testified that, because deviations existed from school to school in the way in which that program had been implemented, some students ÔÇö at some schools ÔÇö may have been helped more by that program than students at other schools where it had not been implemented with the same degree of success. Thus, Mr. Glasgow's testimony strongly supported Dr. Ross's testimony that the step 2 program evaluations were a necessary part of LRSD's program assessment obligations under subpart A of the Compliance Remedy.

21. I find that Dr. Ross is a well-qualified expert in preparing program evaluations and that he has extensive knowledge about programs and strategies that schools can implement to improve the academic achievement of African-American students. I further find that Dr. Ross's testimony was both informative and credible regarding the Literacy Evaluation that he prepared. According to Dr. Ross, he was instructed to prepare the comprehensive Literacy Evaluation but then had very little, if any, substantive contact with LRSD's compliance team regarding his work on that evaluation. Dr. Ross indicated that he was disappointed by LRSD's lack of participation in the Literacy Evaluation.

22. I do not put much stock in Dr. Lesley's testimony at this last hearing. (Her testimony in the unitary status hearings in 2001 was quite helpful.) Her answers to pointed questions were often indirect and marked by semantics. I got the distinct impression that she wanted to avoid giving answers that would be harmful to LRSD's position. In his preparation of one or more of the Page 148 Evaluations, Dr. Ross advised Dr. Lesley that she was improperly using test score results to reach unsupportably favorable conclusions about improvements in African-American achievement. Dr. Lesley testified that, after Dr. Ross called this matter to her attention, she "toned down" some of her statistical conclusions. Yet, in the compliance hearing, Dr. Lesley hailed the Literacy Evaluation's data on the improvement in African-American test scores as a cause for great celebration. Dr. Ross, however, made it clear that, while African-American students at some grade levels had made modest improvements on the state Benchmark Exam, very little progress had been made in improving African-American achievement in the "norm referenced" Stanford Achievement Test. In other words, there is no cause for great celebration on this point, at this time.

23. As I have repeatedly emphasized, LRSD's Compliance Plan interpreted subpart. A of the Compliance Remedy as requiring it to prepare "a comprehensive program evaluation of each academic program implemented pursuant to Revised Plan ž 2.7 to determine its effectiveness in improving the academic achievement of African-American students and to decide whether to modify or replace the program." See docket no. 3745, Exhibit A at 3 (emphasis added). I find it impossible to construe LRSD's specific obligation to prepare a comprehensive program evaluation on "each academic program" to now mean that LRSD was only required to evaluate two broad areas of learning: "Literacy" and "Math and Science." While the fields of "Literacy" and "Math and Science" may be convenient ways to divide academic knowledge, they most certainly do not constitute specific ž 2.7 "academic programs" (e.g., Reading for All, Early Literacy Learning, Reading Recovery, or Effective Literacy) that LRSD implemented, on a school-by-school basis, to improve the academic achievement of African-American students.

24. I find it relevant that, as early as October 10, 2002, less than thirty days after I entered the September 13 Decision, LRSD's Compliance Plan construed subpart A of the Compliance Remedy as requiring it to prepare "comprehensive evaluations of each academic program implemented pursuant to ž 2.7" ÔÇö not "comprehensive evaluations of Literacy and Math and Science." If LRSD believed subpart A of the Compliance Remedy imposed on it obligations that went beyond ž 2.7.1 of the Revised Plan, it should have appealed that issue to the Eighth Circuit It is too late in the game for LRSD to cry foul, and contend that its own construction of subpart A of the Compliance Remedy, as set forth in its own Compliance Plan, is too onerous, and goes beyond its original obligations in ž 2.7.1 of the Revised Plan.

25. Dr. Ross's "big picture evaluation" of literacy test score results has established that, on balance, none of the ž 2.7 programs have produced any "unhealthy outcomes" for African-American students. Thus, I find that LRSD has taken the first step required by subpart A of the Compliance Remedy. Having received this "good news" from Dr. Ross's global Literacy Evaluation, LRSD is now in a position to complete its obligation under subpart A of the Compliance Remedy by preparing step 2 evaluations of each of the key ž 2.7 programs, as they have actually been implemented, in order to determine their effectiveness and the changes and modifications that may need to be made to improve their effectiveness.

26. Finally, Dr. Ross testified that large urban school districts in Chicago and Washington D.C. sometimes perform ten or more step 2 program evaluations each year. While he conceded that it would be unrealistic to expect a district the size of LRSD to perform that many program evaluations, Dr. Ross testified that, in his opinion, it was reasonable to expect LRSD to perform four or five step 2 program evaluations each year in order to continue to make progress in improving the academic achievement of its African-American students.

27. If LRSD had implemented and deeply embedded an effective program assessment process in the 2002-03 school year, assembled "a highly qualified team of professionals to oversee that process, prepared the big picture step 1 evaluations of its Literacy and Math and Science programs, and four or five step 2 evaluations of specific ž 23 programs, with the intention of continuing to perform such evaluations annually for the foreseeable future, I would have had no difficulty concluding that it had met its obligations under, subpart A of the Compliance Remedy. Unfortunately, because almost none of these things were accomplished, I must conclude that LRSD has failed to establish a legally sufficient evidentiary basis for finding that it has substantially complied with its obligations under subpart A of the Compliance Remedy.

28. Subpart B of the Compliance Remedy required LRSD to maintain written records of its assessment of each ž 2.7 program. These written records were required to include the following: (a) the criteria that LRSD used to assess each ž 2.7 program during the 2002-03 school year and the first semester of the 2003-04 school year; and (b) the "results of the annual assessment of each program, including whether the assessments resulted in program modifications or the elimination of any programs." LRSD, 237 F.Supp.2d at 1088. Proposed Regulation IL-R2, which was never approved by the Board, appears to have been designed to satisfy the requirements of subpart B of the Compliance Remedy. According to Proposed Regulation IL-R2, anytime LRSD significantly modified an academic program in "English/Language Arts, Mathematics, Science or Social Studies," it was required to: (a) explain in writing the decision to significantly modify the academic program; (b) set forth "the written criteria used to evaluate the program" in which the changes are to be made; and (c) summarize "the student assessment data [test results] on which the decision was based [to modify the program]." See docket no. 3865, Exhibit A, Appendix 2 at 1. If LRSD had approved, implemented, and complied with Proposed Regulation Il-R2, it might have substantially complied with its obligations under subpart B of the Compliance Remedy.

29. The evidence established that LRSD failed to maintain any of the separate written records on each of the ž 2.7 programs that were required by subpart B of the Compliance Remedy. Instead, LRSD attached to its Compliance Report three documents that contain a confusing compilation of random changes in various vaguely described academic programs during the 2001-02, 2002-03, and 2003-04 school years. See docket no. 3837, Exhibits C, D, and E. The programs are not well enough described so that I can determine which of them are ž 2.7 programs designed to improve the academic achievement of African-American students. Furthermore, while Exhibits C, D,. and E catalogue numerous changes that were made to the vaguely identified programs, they contain no explanation of how each of the changes will increase the effectiveness of the programs in improving the academic achievement of African-American students. Thus, I find that Exhibits C, D, and E to LRSD's Compliance Report fall well short of substantially complying with the requirements of subpart B of the Compliance Remedy.

C. LRSD's Efforts to Comply with Subpart C of the Compliance Remedy.

1. On page 148 of its March 15, 2001 Compliance Report, LRSD erroneously stated that its PRE Department had prepared Program Evaluations on fourteen listed programs. See docket no. 3410 at 148. It is unclear how, LRSD determined that it was necessary to prepare these fourteen Page 148 Evaluations in order to satisfy its obligations under ž 2.7.1 of the Revised Plan. Furthermore, at least one of the listed programs, English as a Second Language ("ELS"), clearly is not a ž 2.7 program. I do not understand how LRSD came to believe that it was required to evaluate an ELS program in order to satisfy its obligations under ž 2.7.1 of the Revised Plan. However, even if LRSD had properly completed all fourteen of the Page 148 Evaluations, I still would have held, for the other reasons enumerated in the September 13 Decision, that it had failed to substantially comply with its obligations under ž 2.7.1 of the Revised Plan. See LRSD, 237 F.Supp.2d at 1076-82.

2. In subpart C of the Compliance Remedy, I required LRSD to complete and file the fourteen Page 148 Evaluations to send a clear message that I expected LRSD to honor all of its commitments even if it now meant preparing evaluations that would be of little or no use in assessing the effectiveness of the ž 2.7 programs. Because I was requiring LRSD, retrospectively, to prepare the fourteen program evaluations that it should have completed on or before March 15, 2001, I expected LRSD to use only the data that would have been available through that date. However, I also knew that, under subpart A of the Compliance Remedy, LRSD was required to formulate and implement a comprehensive program assessment process and then use it to assess each of the key ž 2.7 programs, using the most recent data available from the 2002-03 and 2003-04 school years. Thus, I viewed the comprehensive program assessments LRSD was required to prepare under subpart A of the Compliance Remedynot the Page 148 Evaluations-as the touchstones that LRSD would use to determine the effectiveness of each of the key ž 2.7 programs.

3. Subpart C of the Compliance Remedy provided that the Court "will accept all [Page 148 Evaluations] that have already been completed by Dr. Nunnery or someone with similar qualifications and approved by the Board." See LRSD, 237 F.Supp.2d at 1088. LRSD had completed or substantially completed six of the fourteen Page 148 Evaluations by the time I entered the September 13 Decision. These six evaluations covered the following programs: (a) Pre-K-3 Literacy; (b) NSF Math and Science Project;^[37] (c) Charter Schools; (d) ELS; (e) SEDL Program at Southwest Middle School; and (f) Collaborative Action Team Project. By the end of December, 2002, LRSD's Board had approved all six of these Page 148 Evaluations.

4. On March 14, 2003, LRSD filed these six program evaluations with the Court. See Volumes I and II (docket no. 3745). The ODM's Compliance Report contains no significant substantive criticism of these evaluations. See ODM's Compliance Report at 20 (docket no. 3854). Similarly, Joshua's objections to these six evaluations are primarily confined to alleged shortcomings such as a failure to "properly disaggregate data," or overstating the positive conclusions that might fairly be drawn from some of the data presented. See Joshua's Comments on the Page 148 Evaluations at 3-5 (docket no. 3752).

5. Of the eight remaining Page 148 Evaluations, LRSD had discontinued four of the programs by the time I entered the September 13 Decision. Thus, the Page 148 Evaluations on LRSD's Elementary Summer School Program, the Lyceum Scholars Program, the Onward to Excellence Program at Watson Elementary, and the Vital Link Program were utterly useless in determining the effectiveness of any of the ongoing ž 2.7 programs. Because these four Page 148 Evaluations are essentially "obituaries" on discontinued programs, I find the ODM's and Joshua's criticism of these evaluations falls squarely in the category of "beating a dead horse."

6. The last four Page 148 Evaluations cover Middle School Implementation, Extended Year Schools, HIPPY, and Campus Leadership Teams. See Volumes III and IV (docket no. 3745). While these four evaluations could be improved, I find they are adequate to fulfill the limited purpose for which they were prepared.

7. As previously stated, I never expected for any of the Page 148 Evaluations to be of much, if any, use to LRSD in assessing the effectiveness of the current ž 2.7 programs. Rather, I required LRSD to prepare those evaluations to fulfill the promises it made in the March 15, 2001 Compliance Report. For that reason, any flaws in the Page 148 Evaluations are of little, if any, substantive importance in determining the effectiveness of the ž 2.7 programs.

8. As previously discussed, after. LRSD rested its case in the compliance hearing, I granted Joshua's Rule 52(c) Motion with respect to LRSD's proof that it had complied with its obligations under subparts A and B of the Compliance Remedy. Joshua then withdrew its challenge to LRSD's evidence that it had substantially complied with subpart C of the Compliance Remedy. Thus, based upon Joshua's withdrawal of their challenge to LRSD's compliance with subpart C, and my own independent findings on the merits of that compliance issue, I conclude LRSD has substantially complied with subpart C of the Compliance Remedy and is released from all further supervision and monitoring with regard to those obligations.

VII. Compliance Remedy

Because LRSD failed to substantially comply with the crucially important obligations contained in ž 2.7.1 of the Revised Plan, it must remain under court supervision for two more complete school years, 2004-05 and 2005-06. To avoid any "misunderstanding" regarding this Compliance Remedy, I will be specific. The new Compliance Remedy is as follows:

A. LRSD must promptly^[38] hire a highly trained team of professionals to reinvigorate PRE. These individuals must have experience in: (a) preparing and overseeing the preparation of formal program evaluations; and (b) formulating a comprehensive program assessment process that can be used to determine the effectiveness of specific academic programs designed to improve the achievement of African-American students. I expect the director of PRE to have a Ph.D.; to have extensive experience in designing, preparing and overseeing the preparation of program evaluations; and to have a good understanding of statistics and regression analysis. I also expect LRSD to hire experienced statisticians and the other appropriate support personnel necessary to operate a first-rate PRE Department.

B. The first task PRE must perform is to devise a comprehensive program assessment process.^[39] It may take a decade or more for LRSD to make sufficient progress in improving the academic achievement of African-American students to justify discontinuing the need for specific ž 2.7 programs. For that reason, the comprehensive program assessment process must be deeply embedded as a permanent part of LRSD's curriculum and instruction program. Only then will I have the necessary assurance that LRSD intends to continue using that process for as long as it is needed to determine the effectiveness of the various key ž 2.7 programs in improving the academic achievement of African-American students. Part of LRSD's proof, at the next compliance hearing, must include evidence that it has devised and implemented a comprehensive program assessment process, which has been deeply embedded as a permanent part of its curriculum and instruction program. I suggest that LRSD use Dr. Ross to assist in developing this comprehensive program assessment process; then be sure that he approves that process before it is finalized and implemented.

C. During each of the next two academic school years (2004-05 and 2005-06), LRSD must hire one or more outside consultants to prepare four (4) formal step 2 evaluations^[40] Each of these step 2 evaluations must cover one of the key ž 2.7 programs, as it has been implemented in schools throughout the district. Thus, over the course of the next two academic school years, LRSD must hire outside consultants to prepare a total of eight (8) formal step 2 evaluations of key ž 2.7 programs. During the recent compliance hearing, Dr. Ross made it clear that LRSD must conduct these formal step 2 evaluations of the key ž 2.7 programs in order to continue to make progress in improving the academic achievement of African-American students. Again, I suggest that LRSD hire Dr. Ross-to perform the following tasks: (1) identify the four key ž 2.7 programs that should be formally evaluated during the 2004-05 school year and the four key ž 2.7 programs that should be formally evaluated during the 2005-06 school year; and (2) prepare as many of the eight step 2 evaluations as possible. If Dr. Ross cannot prepare all eight of the step 2 evaluations, I recommend that LRSD hire someone that Dr. Ross recommends as possessing the experience and ability necessary to prepare those evaluations.

D. Each of the eight step 2 evaluations must answer the following essential research question: "Has the ž 2.7 program being evaluated improved the academic achievement of African-American students, as it has been implemented in schools throughout the district?" The eight step 2 evaluations may also answer as many other research questions as the designers of each evaluation deem necessary and appropriate. Each of the step 2 evaluations must be organized and written in such a way that it can be readily understood by a lay person. I will allow the outside experts preparing each of these evaluations to decide on the appropriate number of years of test scores and other data that need to be analyzed in preparing each evaluation. PRE must: (1) oversee the preparation of all eight of these step 2 evaluations; (2) work closely with Dr. Ross and any other outside consultants hired to prepare these step 2 evaluations; and (3) provide the outside consultants with any and all requested assistance and support in preparing these step 2 evaluations.

E. In order to streamline LRSD's record-keeping obligation,^[41] I am going to require that each of the eight step 2 evaluations contain, in addition to the traditional information and data, a special section which: (1) describes of the number of teachers and administrators, at the various grade levels, who were interviewed or from whom information was received regarding the effectiveness of the key ž 2.7 program being evaluated; (2) lists each of the recommended program modifications, if any, that were deemed necessary in order to increase the effectiveness of each of the ž 2.7 programs in improving the academic achievement of African-American students; and (3) briefly explains how each of the recommended modifications, is expected to increase the effectiveness of the ž 2.7 program. This requirement is intended to relieve LRSD of any independent recordkeeping obligations under ž 2.7.1 of the Revised Plan and the Compliance Remedy.

F. As soon as PRE and Dr. Ross identify the eight ž 2.7 programs targeted for step 2 evaluations, PRE must notify the ODM and Joshua in writing of the names of those eight programs. In addition, after PRE and Dr. Ross have formulated a comprehensive program assessment process and reduced it to a final draft, PRE must provide a copy to the ODM and Joshua at least thirty days before it is presented to the Board for approval. I expect the Board to approve LRSD's comprehensive program assessment process no later than December 31, 2004.

G. PRE must submit quarterly written updates on the status of the work being performed on the four step 2 program evaluations that will be prepared during the 2004-05 school year and the four step 2 program evaluations that will be prepared during the 2005-06 school year. These quarterly updates must be delivered to the ODM and Joshua on December 1, March 1, June 1, and September 1 of each of those two academic school years. As soon as each of the eight step 2 evaluations has been completed and approved by the Board, LRSD must provide a copy to the ODM and Joshua.

H. The ODM must actively assist LRSD in meeting its obligations under this Compliance Remedy. If problems arise, the ODM must work with LRSD to solve them. The ODM's primary mission is to do everything necessary to ensure that LRSD substantially complies with all of its obligations under this Compliance Remedy.

I. I expect Joshua to continue to fulfill its traditional role of monitoring LRSD's compliance obligations. If problems arise between LRSD and Joshua regarding the formulation and implementation of the comprehensive program assessment process, the key ž 2.7 programs that will be the subject of the eight step 2 evaluations, or any other aspect of LRSD's compliance obligations, the parties must bring such problems to my attention so that I can resolve them. I want to be very clear on this point ÔÇö if compliance problems arise, the parties must immediately bring them to my attention so that I can resolve them while there is still time for LRSD to make "mid-course corrections."

J. The four step 2 program evaluations for the 2004-05 school year must be filed with the Court no later than October 1, 2005. The four step 2 program evaluations for the 2005-06 school year must be filed with the Court no later than October 1, 2006.

K. On or before October 15, 2006, LRSD must file a Compliance Report documenting its compliance with its obligations under ž 2.7.1 of the Revised. Plan, as specified in this Compliance Remedy. If Joshua wishes to challenge LRSD's substantial compliance, they must file objections on or before November 15, 2006. Thereafter, I will schedule a compliance hearing and decide whether LRSD has met its obligations under the Compliance Remedy and should be released from all further supervision and monitoring.

L. This Compliance Remedy is intended to supersede and replace the identical compliance obligations that I imposed on LRSD, albeit with less specificity, in subparts A and B of Section VII of the September 13 Decision.

If the parties and the ODM are not fully advised in the premises by the above, now, not later, is the time to contact the Court.

VIII. Final Thoughts on LRSD's Compliance Efforts

After extensive negotiations between LRSD and Joshua, which took place throughout much of 1997, the parties voluntarily entered into the January 15, 1998 Revised Desegregation and Education Plan (CX 871). See LRSD, 237 F.Supp.2d at 1013. On April 10, 1998, my predecessor in this case, the Honorable Susan Webber Wright, entered a Memorandum Opinion and Order (docket no. 3144) approving the Revised Plan. I review this history to make it crystal clear that LRSD-not the Court-formulated all of the program assessment/evaluation obligations contained in ž 2.7.1 of the Revised Plan and voluntarily agreed to comply with all of those obligations.

I know it will be quite a burden for LRSD to formulate, implement, and deeply embed in its curriculum an effective ž 2.7.1 program assessment/evaluation process that will allow it to determine the effectiveness of each of the key ž 2.7 programs. But this is the medicine that LRSD knowingly and voluntarily decided it must take in an attempt to cure the historically low academic achievement of so many of its African-American students. I know that: (1) substantially complying with the obligations in ž 2.7.1 of the Revised Plan will require LRSD to form a team of highly trained professionals (many of whom may have to be hired from outside the ranks of its current teachers and administrators) to formulate, implement, and deeply embed an effective comprehensive ž 2.7.1 program assessment/evaluation process; and (2) it will be expensive for LRSD to hire and assemble this team of professionals to oversee the ž 2.7.1 program assessment/evaluation process for the foreseeable future. But again this is the medicine that LRSD voluntarily prescribed for itself. Finally, I realize that an effective and comprehensive ž 2.7.1 program assessment process will require LRSD to use the services of outside consultants such as Dr. Ross, who will be required to prepare the step 2 evaluations of the key ž 2.7 programs. But, this is a crucial part of the obligation that LRSD voluntarily undertook when it agreed in ž 2.7.1 of the Revised Plan to annually assess the key ž 2.7 programs "in order to determine the effectiveness of the academic programs in improving African-American achievement."

It seems to me there is no higher and better use of LRSD's resources than to seek to improve the academic achievement of the overwhelming majority of its students who are currently scoring at a discouragingly low level on standardized tests.

I would expect LRSD to continue doing a splendid job of providing outstanding educational opportunities for its best and brightest students, African-American and all others. However, we can only celebrate when a significant number of African-American students, so many of whom start life facing serious disadvantages, are able to step into the academic equivalent of a batter's box and hit a home run by scoring at or above the proficient level in reading, math, and science. To this end, LRSD must do what it promised to do, and what it has been ordered to do because of this promise. In the words of the poet of the Yukon, Robert Service, "a promise made is a debt unpaid."

IT IS THEREFORE ORDERED that LRSD must satisfy the Compliance Remedy described in Section VII in order to substantially comply with ž 2.7.1 of the Revised Plan.

IT IS FURTHER ORDERED that LRSD is hereby released from all further supervision and monitoring in connection with its obligations under subpart C of the Compliance Remedy contained in the September 13 Decision.

EXHIBIT A

OFFICE of DESEGREGATION MONITORING

United States District Court Eastern District of Arkansas

Ann S. Marshall, Federal Monitor

One Union National Plaza

124 West Capitol, Suite 1895

Little Rock, Arkansas 72201

(501) 376-6200 Fax (501) 371-0100

November 4, 2002

Mr. John Walker

1723 Broadway

Little Rock, AR 72206

Dear Mr. Walker:

I've received your November 1, 2002 request to invoke Section 8.2.4 of the LRSD's revised desegregation plan in relation to the district's new Compliance Plan, developed in response to the Court's recent orders on program evaluation. We'll be happy to assist you and LRSD representatives, assuming their willingness to participate in discussions. However, I need to inform you of a problem that will delay our assistance.

ODM associate Gene Jones is responsible for monitoring program evaluation in the LRSD, which he's been following closely for a long time. Gene has been involved in an accident, resulting in several broken bones and other injuries. He's been out of the office for two weeks, and I don't know when his doctors will allow him to return to work. In view of Gene's long-term monitoring of program evaluation, I believe it will be helpful to all of us to schedule a discussion of your concerns about the Compliance Plan after his return. If his absence should become protracted, we may decide to start without him.

I'll keep you posted on Gene's progress and his forecasted return-to-work date when I know it.

Sincerely yours,

Ann S. Marshall

cc: Chris Heller

Clay Fendley

Counsel of Record

EXHIBIT B

UNITED STATES DISTRICT COURT

EASTERN DISTRICT OF ARKANSAS

600 W. CAPITOL ROOM 423

LITTLE ROCK, ARKANSAS XXXXX-XXXX

(505) 604-5140

Facsimile (501) 604-5149

BILL WILSON, JUDGE

November 6, 2002

Ms. Ann Marshall

Office of Desegregation Monitoring

124 West Capitol, Suite 1895

Little Rock, AR 72201

CONFIDENTIAL

Dear Ms. Marshall:

Attached is your copy of my letter to Mr. Fendley.

It seems to me that it would be best if you would work with the parties towards implementing the remedy; but you should feel free to contact me in writing if a serious impasse develops. In other words, as long as everything is going along smoothly, I see no reason for you to make regular reports to me in this respect. I emphasize, however, that you should feel free to call on me if serious problems arise.

Many thanks.

Cordially, Wm. R. Wilson, Jr.

cc: The Honorable J. Thomas Ray ÔÇö Confidential

NOTES

[1] I would be seriously remiss if I did not once again note the tremendous amount of work United States Magistrate Judge Joe Thomas Ray has done on this case.

[2] Bryan Garner, of Dallas, Texas, has published several excellent books and articles for the legal profession on the use of plain, understandable English.

[3] During the 2001-02 unitary status hearings, the Revised Plan was introduced into evidence as CX871.

[4] The Joshua Intervenors are a group of African-American school children, some of whom are enrolled in each of the three Pulaski County school districts. Thus, Joshua serves as the class representative for all African-American students enrolled in LRSD, the Pulaski County Special School District, and the North Little Rock School District.

[5] On page 148 of its March 15, 2001 Final Compliance Report (docket no. 3410); LRSD flatly stated that it had prepared program evaluations on fourteen separate programs listed on that page. During the 2001-02 unitary status hearings, the evidence overwhelmingly established that, as of March 15, 2001, LRSD had not prepared any of those fourteen program evaluations. See LRSD, 237 F.Supp.2d at 1079-80. Henceforth, I will refer to these fourteen program evaluations as the "Page 148 Evaluations."

[6] During the June 14 and 15, 2004 compliance hearing (hereafter referred to as the "compliance hearing"), the Proposed Compliance Plan was introduced into evidence as LRSD's Exhibit No. 2. In most respects, the Proposed Compliance Plan is identical to the final Compliance Plan approved by the Board on October 10, 2002. However, the Proposed Compliance Plan raises a number of questions about the meaning of subparts A and B of the Compliance Remedy. All of those questions were deleted from the final Compliance Plan approved by the Board on October 10, 2002. As will be discussed later, LRSD did not bring any any of those questions to my attention, or otherwise seek clarification of the requirements of the Compliance Remedy.

[7] During the compliance hearing, Regulation IL-R1 was introduced into evidence as Joshua's Exhibit No. 2.

[8] As I explain later in some detail, I was not made aware of LRSD's Proposed Compliance Plan and Regulation IL-R1 until two months after LRSD filed its March 12, 2004 Compliance Report. LRSD did not file the October 10, 2002 Compliance Plan with the Court until March 14, 2003.

[9] Section 8.2 of the Revised Plan required the parties to use the ODM to "facilitate" the resolution of any compliance issues. Since the words "facilitation" and "facilitate" come directly from the Revised Plan, I will use them.

[10] A copy of Ms. Marshall's November 4, 2002 letter, marked Exhibit A, is attached to this Memorandum Opinion.

[11] A copy of my November 6, 2002 letter, marked Exhibit B, is attached to this Memorandum Opinion.

[12] SEDL is an acronym for "Southwest Educational Development Laboratory." See ODM's Compliance Report at 8 (docket no. 3854).

[13] A number of years ago, Joshua formally agreed that Dr.. Ross has the qualifications necessary to prepare program evaluations.

[14] HIPPY is an acronym for "Home Instruction for Parents of Preschool Youngsters." See ODM's Compliance Report (docket no. 3854) at 8.

[15] During the compliance hearing, this document was introduced as LRSD's Exhibit No. 5.

[16] During the compliance hearing, these fourteen program evaluations were introduced as LRSD's Exhibit No. 13.

[17] During its implementation of the Revised Plan in 1998 through early 2001, LRSD originally intended to use its own Department of Planning, Research, and. Evaluation ("PRE") to prepare the fourteen Page 148 Evaluations. Dr. Kathy Lease was the Assistant Superintendent who headed PRE and reported to Dr. Lesley. See ODM's Compliance Report at 2.

According to Dr. Lesley's testimony during the November 2001 hearings on unitary status, Dr. Lease "dropped the ball" in preparing these evaluations, which resulted in only a few partially completed and woefully inadequate evaluations being available on March 15, 2001, the deadline for LRSD to file its Compliance Report seeking unitary status. Additionally, Dr. Lesley testified that, in her opinion, no one in PRE-including Dr. LeaseÔÇö had the expertise to prepare program evaluations. See LRSD, 237 F.Supp.2d at 1077-81.

In early 2001, Dr. Lease resigned and left LRSD for other job opportunities. Since that time, PRE has functioned with only a statistician and several support employees, but no one was hired to replace Dr. Lease or take over and operate the department. In November of 2003, LRSD appointed the statistician as the acting head of PRE. As a result, since Dr. Lease's departure in early 2001, LRSD has essentially functioned without a meaningful Department of Planning, Research, and Evaluation. See ODM's Compliance Report at 6.

[18] On June 11, 2004, the Board announced that it hired Mr. Roy Gregory Brooks to serve as LRSD's new Superintendent of Schools. It is my understanding that sometime in July or August Mr. Brooks will assume his new duties as Superintendent.

[19] During the compliance hearing, this document was introduced into evidence as LRSD's Exhibit No. 14.

[20] As indicated previously, during the compliance hearing, the Compliance Report was introduced into evidence as LRSD's Exhibit No. 14.

[21] During the compliance hearing, the ODM's Compliance Report was introduced into evidence as LRSD's Exhibit No. 15.

[22] The September 13 Decision addressed in detail all of the events described in these four pages of the ODM's Compliance Report.

[23] During the compliance hearing, LRSD witnesses correctly pointed out that the chart on page 10 of the ODM's Compliance Report was flawed. Among other things, a number of programs are listed twice, and some of the things that are listed as "programs" are not.

[24] Although not mentioned in the ODM's Compliance Report, Exhibit C also fails to discuss the reasons for making each of the modifications in the math and literacy programs, or and how those modifications were expected to improve the effectiveness of those programs.

[25] As indicated previously, LRSD originally intended to use Dr. Kathy Lease and its PRE Department to design and prepare these eight evaluations. When Dr. Lease resigned and left her position with LRSD in early 2001, none of these evaluations had been prepared.

[26] See Exhibit F at 1 (docket no. 3837).

[27] See Appendix 1 to docket no. 3865; see also Joshua's Exhibit No. 2.

[28] See Exhibit A to docket no. 3865; see also LRSD's Exhibit No. 2.

[29] See Appendix 2 to docket no. 3865.

[30] Section 2.7.1 of the Revised Plan obliged LRSD to "assess" the effectiveness of the ž 2.7 programs and, if that assessment revealed the programs were not likely to improve AfricanÔÇöAmerican achievement, LRSD was required to modify or replace those programs. Nowhere in ž 2.7.1 does it mention anything about LRSD being required to prepare "program evaluations" in order to determine the effectiveness of the ž 2.7 programs. However, as I explained previously on pages 4-5, LRSD unquestionably construed ž 2.7.1 as requiring it to perform "evaluations" of the key ž 2.7 programs in order to determine their effectiveness in improving the academic achievement of AfricanÔÇöAmerican students.

[31] During the May 1996 hearings in this case, Dr. Walberg, Dr. Armor, and Dr. Orfield, three nationally recognized experts in desegregation litigation, testified at length about the numerous causes for this achievement gap. Both Dr. Armor and Dr. Walberg testified that the socioeconomic differences between many African-Americans and whites make it virtually impossible for schools to eliminate the achievement gap. See LRSD, 237 F.Supp.2d at 1073-74.

[32] See docket no. 3745, Exhibit A at 3; LRSD's Exhibit No. 2.

[33] During the recent compliance hearing, this document was introduced as Joshua's Exhibit No. 2.

[34] In Dr. Ross's defense, he testified that he had little,,if any, communication with LRSD's staff members or administrators after he was hired to prepare the Literacy Evaluation. Thus, it is unclear whether he was provided with a copy of Regulation IL-R1, or otherwise advised of its purpose and requirements.

[35] As indicated previously, LRSD has never provided me with a list of all of its ž 2.7 literacy programs. Therefore, I have no way of knowing if these were the only ž 2.7 literacy programs implemented at the elementary school level.

[36] See docket no. 3745, Exhibit A at 3.

[37] This evaluation included data and test scores from school years 1998-99 through 2000-01. See Volume II (docket no. 3745).

[38] By "promptly hire," I mean, if possible, before school starts in August of this year. Because this team must consist of experienced and highly trained professionals, LRSD may not be able to accomplish this task by August. However, if LRSD believes that it needs more than ninety days to assemble this team of professionals, its attorneys should immediately notify me so that I can schedule a hearing on this matter.

[39] By "comprehensive program assessment process," I mean everything necessary to accurately assess the effectiveness of the key ž 2.7 programs in improving the academic achievement of African-American students. As explained in detail in subpart C of this Compliance Remedy, part of the "comprehensive program assessment process" must include formal step 2 evaluations of certain key ž 2.7 programs, as they have been implemented in schools throughout the district. I also expect part of LRSD's "comprehensive program assessment process" to include preparing informal program assessments that involve interviews with teachers, informal evaluations of test scores, and the other things normally associated with the more dynamic program assessment process. While it should already be crystal clear, I am not using the term "assessment" to mean "testing."

[40] For a detailed discussion of what I mean by "step 2 evaluations," see paragraphs 14 through 26, supra, at 991-994.

[41] Subpart B of the Compliance Remedy in the September 13 Decision required LRSD to maintain certain written records regarding its assessment of each of the key ž 2.7 programs. See LRSD, 237 F.Supp.2d at 1087-88. LRSD had considerable difficulty compiling and properly maintaining those written records.