Opening salvo -- Two thumbs down for Collateral Damage

I started reading Collateral Damage: How High-Stakes Testing Corrupts America's Schools on the bus on Thursday and it made me so angry I read the whole thing in a day. The book is propaganda, not research. The authors (Sharon Nichols and David Berliner) start by comparing NCLB to 9-11 and Hurricane Katrina, claim that high-stakes testing is always bad, and then find anecdotes of bad things that have happened since NCLB.

This is shoddy research, based in ideology and calculated to fire up opponents of accountability reform. Here are my responses to the book's two major claims:

• "Campbell's Law" is bunk. Nichols and Berliner contend that "Campbell's law stipulates that the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to monitor." They elevate this "law" to the level of the Heisenberg uncertainty principle and use it throughout the book as an carte blanche excuse for not assessing school performance.
  
  If you search for "campbell's law" on google or google scholar, the vast majority of sites and articles you turn up will refer to Collateral Damage. Most of the rest are written by authors named Campbell. This "law" was not recognized or circulated prior before Nichols and Berliner's book.
  
  Don't go calling it "Berliner's law" either -- it simply isn't true. Example: grades are the sole measurement for evaluating college student performance. Do you cheat to improve your grades -- always? If Berliner's word is law, you must. Nichols and Berliner cite examples of doctors, teachers, salespeople, CEOs, politicians, auto mechanics, etc. cheating to improve their bottom lines. Do they all cheat? Of course not. Campbell's "law" is a vast overgeneralization.
  
  The truth is that performance metrics put pressure on those being evaluated. Organizational theorists have a good framework for looking at cheating and corruption called the "fraud triangle." In the fraud triangle, cheating happens when pressure, opportunity, and rationalization come together. In stark contrast to Nichols and Berliner's claim, pressure from testing alone isn't enough to induce teachers to cheat.
  



• The authors of Collateral Damage also contend that performance evaluation is an "archaic theory of management" "commonly used to manage workers in the nineteenth century--a workforce consisting mostly of laborers whose intellects were not valuable and who were completely replaceable (pieceworkers in shirt factories, fruit pickers, nonunion laborers)." Nichols and Berliner try to argue that modern businesses don't attachthreats and incentives to the performance of knowledge workers. Read this section again and you'll notice the lack of any citations for this ludicrous idea.
  
  The truth is that most professionals are paid largely for performance. Most hiring, firing, and promotion decisions are made on the basis of performance evaluations. Incentives and performance pay are the rungs of the most career ladders. Managers, entrepreneurs, and increasingly, everyday employees, are compensated with stock options. Lawyers, wall street analysts, consultants, and CEOs count on annual review bonuses as a substantial portion of their income. The trend in medicine for the last 30 years has been towards performance management. And that's to say nothing about the fact that all businesses live and die by the ultimate performance evaluation -- the ability to generate revenue in a competitive environment.
  
  From the early experiments with incentivized performance in the 1800's, performance pay has become increasingly widespread, sophisticated, and effective. Education is one of the last industries in the modern information economy that has held out against performance evaluation. I don't know if Nichols and Berliner missed this trend because of ignorance or ideology, but it is a glaring omission and a bad foundation for a book about measuring performance.
  

Collateral Damage never tries to verify either of these two very large claims. It simply takes them as given and then chronicles supporting anecdotes. Itdoesn't address alternative hypotheses. It doesn't consider context. It calls for no further research -- according to its authors, Collateral Damage is the definitive end to the debate about accountability.

To my mind, this is an unproductive path for policy discussion. It starts from ideology, not reality, and leaves no room for discovery and discussion. There are plenty of legitimate criticisms and questions about accountability policy: What are rational ways to choose proficiency levels? Could we improve testing by using different formats? How can we prevent cheating by teachers and students? These need to be resolved or high-stakes testing might end up hurting a lot of kids.

Unfortunately, Nichols and Berliner ignore the real criticisms in favor of false generalizations, made-up "laws," and extreme anecdotes. It's writing like this that makes reform such a combatative and political process. Two thumbs down for the closed-mindedness and lack of real analysis that produced Collateral Damage.