From Washington Post comes this counterpoint to the Joel Klein column from yesterday:

This is the latest in a back-and-forth between Joel Klein, former New York City Schools chancellor, and Aaron Pallas, a Teachers College professor and statistician.

The debate started with this Klein op-ed published in The Washington Post two weeks ago about the new generation of school reformers who model their work after Klein’s. Pallas responded with this piece, which argued that the reformers use accountability systems that don’t measure real learning, to which Klein responded on this blog here, arguing that Pallas was wrong. Here’s Pallas’s latest response to Klein. You won’t be surprised to find that, once again, he vociferously disagrees with Klein.

Pallas writes the Sociological Eye on Education blog (on which his latest response appeared) for The Hechinger Report, a nonprofit, non-partisan education-news outlet affiliated with the Hechinger Institute on Education and the Media. Pallas has also taught at Johns Hopkins University, Michigan State University, and Northwestern University, and served as a statistician at the National Center for Education Statistics in the U.S. Department of Education.

Before becoming chancellor in 2002, Klein had been a lawyer in public and private practice, serving as the lead prosecutor in the antitrust case United States v. Microsoft. He quit as chancellor late last year and is now executive vice president at Rupert Murdoch’s News Corp.

By Aaron Pallas

It’s an appealing narrative: A new generation of school leaders takes on entrenched interests. Reforms are new and bold. Outcomes soar.

This is the story Joel Klein sought to tell in his op-ed in The Washington Post two weeks ago. He named a number of alumni from his administration who had taken on leadership positions in school districts and states across the country, and — coupling them with other recent appointments offsetting his and Michelle Rhee’s resignations — proclaimed, “What a difference a few months can make!” [Rhee resigned last October as D.C. schools chancellor.]

Pointing to the success of graduates of two KIPP charter schools, Klein claimed that the KIPP students “essentially eliminated the achievement gaps with respect to race, ethnicity and poverty.” He then generalized from these two cases to argue that all schools can do much better, relying on reforms such as holding teachers accountable for student performance and providing greater choice for families.

The narrative falls apart, however, if one looks beyond the two KIPP schools to a broader array of evidence about the impact of reforms Klein has championed. In my blog post on Klein’s op-ed, I suggested setting the KIPP story aside—because Klein and the other reformers he m

entioned had nothing to do with the apparent success of those schools.

Instead, I drew attention to student performance during the era that Klein and his like-minded colleagues were at the helm of the New York City schools. After eight years, did achievement rise precipitously? Did New York City make significant progress in closing the achievement gaps that separate students of different racial and ethnic backgrounds?

Relying mainly on analysis of the performance of New York City students on the National Assessment of Educational Progress, often referred to as the “Nation’s Report Card,” I concluded that the modest gains over time observed in New York City were comparable to those in other parts of the country, and that the achievement gap was nearly as large at the end of Klein’s tenure in 2010 as when he took over in 2002.

Not surprisingly, Klein took umbrage at my characterization of what was achieved during his tenure. He likes KIPP a lot, and took some time to rebut my concern about the claims that might be drawn from the data on those two KIPP schools, but that’s a distraction from his administration’s record in New York City — as are his references to studies of charter-school effects in Boston and New York City. (No mention of Margaret Raymond’s finding that only 17 percent of charter schools show gains that are better than their traditional public-school counterparts — but why mess up a good story?) Instead, Klein relies heavily on a study of gains on state tests in reading and math and in graduation rates conducted by my colleague Jim Kemple.

Beyond the name-calling, there is a profound difference in how Klein and I make sense of the available data. I have a great deal of ambivalence about state tests and high-school graduation rates because they have proven vulnerable to manipulation, which undermines their validity as measures of student achievement. Tests and graduation rates can tell us something about what students have learned, but there is a great deal of uncertainty about what these measures mean. This uncertainty argues for caution in drawing claims about the success of school-reform efforts based on these data.

Joel Klein, on the other hand, created an accountability system in New York City that treated test scores and graduation rates as virtually the only outcomes of schooling worthy of attention—ascribing in the process a precision to these numbers that is unwarranted. It makes one wonder what specifically Klein thinks schooling is about and what an educated student ought to know upon leaving high school. I consider myself a pretty careful observer of Klein and his record, but I cannot recall ever having seen a detailed account in his own words of what students should know and be able to do.

Why am I ambivalent about the New York State tests in reading and math? Over the past several years, there has been a sharp divergence between trends on federal tests and those reported by New York State. The New York State Department of Education acknowledged last year that state tests in reading and math administered in grades 3-8 became easier and more predictable over time, calling into question their ability to measure what students have learned. When the state recalibrated the scores, most of the “gains” of the previous eight years vanished.

And while Jim Kemple’s analysis found that proficiency rates on state tests rose faster between 2002 and 2010 in New York City than in other parts of New York State, those gains were rooted in a set of tests and proficiency cutoffs that the state of New York has asserted were too easy and too predictable. I’m reluctant, then, to treat this as strong evidence of meaningful academic growth during the Klein era.

As for high-school graduation rates, it’s true that these rates have been rising rapidly in New York City. Whereas 47 percent of the entering ninth-grade class of 2001 graduated from high school in four years, 65 percent of those entering ninth grade in 2006 did so. And the gains observed for black and Hispanic students were especially large, rising from 40 to 61 percent, and 37 to 58 percent, respectively. Taken at face value, these patterns suggested that school achievement had risen steadily and the racial/ethnic achievement gap had shrunk substantially.

But consider that three out of every four New York City high-school graduates who enter the city’s community colleges require remediation in basic skills — and that fully 50 percent of those who enter the CUNY system, including its senior colleges, need remediation. One wonders, then, whether completing high school in New York City truly represents a high level of academic accomplishment.

Not according to the state of New York, which is in the process of phasing out a tiered system of diplomas that distinguished among Local Diplomas, Regents Diplomas, and Regents Diplomas with Advanced Designation.

Whereas all students have been required to accumulate 44 credits in various subject areas in high school, the type of diploma awarded has rested on performance on the Regents exams. A Local Diploma could be earned by achieving a score of 55 or higher on each of five Regents exams in English, mathematics, science, U.S. history and government, and global history and geography. A Regents Diploma had a higher threshold of performance; students could earn a Regents Diploma by scoring 65 or higher on each of these five exams. An Advanced Regents Diploma required a 65 or higher on additional examinations in mathematics, science and a language other than English. The gains in the graduation rate observed in New York City have been more pronounced for the less rigorous Local Diploma, which is the one being phased out.

The accountability system that Joel Klein imposed on high schools created incentives for awarding credits to students who hadn’t demonstrated mastery of course content. Regents exams — scored by a school’s own teachers — have shown a peculiar bulge in performance at the threshold score of 65, suggesting there are pressures to pass students just below the cutoff score. This casts serious doubt on whether a New York City high-school diploma is a reliable indicator that a young person is prepared for life after high school.

But scoring a 65 is not a high level of performance on the Regents exams, because many students scoring at this level are subject to remediation in college. This year, the New York State Board of Regents released a new measure of college readiness characterized as an aspirational performance measure. It tabulates the number of students from each entering ninth-grade high school cohort who scored high enough on the Regents exams to predict placing out of remedial courses or earning at least a C grade in college-level courses at two- and four-year institutions in New York City and around the state of New York.

How do the black and Latino youth in New York City’s high schools, who make up 70 percent of the system’s enrollment, fare on the state’s measures of college readiness? You won’t find this number in any of the PowerPoint decks prepared by the New York City Department of Education, but it’s easy enough to calculate from publicly available data.

For black and Latino students who entered high school in 2006—and who thus had the benefit of four years of elementary and middle school, and all four years of high school, under the policies and practices of Joel Klein’s administration—the proportion emerging ready for college in 2010 was 12 percent.

Twelve percent of black and Latino students graduating from high school ready for college—after eight years on Chancellor Klein’s watch.

Proven reforms? You be the judge.