Once upon a time, there was a cowboy cook who acknowledged that his “rabbit stew” included horsemeat. “It’s 50-50,” he said. “One rabbit, one horse.”
Emails from Good Schools Pennsylvania, which lobbies for higher state spending on education, reminds me of that story. They rarely fail to mention a $650,000 “costing-out study” released in 2007 by the State Board of Education. It purported to show that the state’s education budget should be increased by 20%. That conclusion may be right, but it’s impossible to tell the extent to which it’s based on hard data, as its supporters imply.
The study’s long section on methodology explains that its findings were based on a mix of empirical data and the opinions of people predisposed to think that the state should spend a lot more on education. The reader is left to guess which factor – data or opinion – was “rabbit” and which was “horse. In response to my question, a spokesperson for the State Board of Education acknowledged that this haziness was intentional.
The section on methodology describes three approaches. One is called “evidence-based,” a review of published literature. A second approach is called “expert judgment,” meaning the consensus opinion of a panel that consisted mostly of professional educators. (Incidentally, these seem to be two ways of describing one process – the evaluation of existing evidence by people prominent in the field. There’s nothing wrong with this, but the potential for bias is clear. “Experts” in any field tend to have a stake in their field’s growth. Panels of oil industry experts are apt to find a need for more drilling. Panels of high-ranking military officers are apt to find a need for higher defense spending. Should anyone expect panels of school administrators and university departments of education faculty to be different?)
A third approach – called an “effective schools” approach — looked less subjective. The idea was to identify about 80 schools whose test scores showed strong progress toward meeting state performance goals. That approach could not pinpoint how much money is “enough,” but it could at least set a reasonable floor. And that’s how the study was marketed: as providing hard evidence about how much money would be needed to provide high-quality education across the state.
For example, suppose that the overwhelming majority of these high-performing schools spent between $12,000 and $14,000 per student per year. That would suggest that a floor of $12,000 will almost always be “enough,” while anything over $14,000 is probably “unnecessarily expensive.” As the analysts correctly note, the analysis would be complicated and requires attention to factors like demographics and the incidence of special-needs students. However, even if the range of spending in these schools (all pre-selected for high academic quality) were found to be very wide, the median spending level (call it “X” dollars) would at least serve as a reality check on expert opinion.
So how large was “X”? Don’t ask. The study report devoted a lot of space to explaining background material – for example, how a sample of approximately 80 high-performing schools was constructed and how the data was collected. But it stopped short of disclosing the two key points: (1) the actual data from individual schools used to estimate “X” and (2) the importance accorded to “X” (the weight given to hard data relative to the softer approaches). After many pages on possible stew recipes, it left us with “one rabbit, one horse.”
Was this an oversight? I emailed the State Board of Education, asking to see the actual cost data collected on high performing schools. I also asked how much weight it was given, relative to the softer “evidence-based” and “expert judgment” approaches. Here’s an excerpt from the reply. (I’m not including the name of the staffer who replied, since there’s no reason to link him personally to a political decision above his pay grade.)
The Board [my correspondent wrote] does not have additional information to share beyond that which posted on our web site. We do not have additional analysis, working papers of technical documents. Much of the work conducted by APA is proprietary. [Emphasis supplied.]
(By the way, I don’t buy the notion that the Commonwealth of Pennsylvania should pay $650,000 for a “proprietary” analysis whose reasonableness cannot be checked by independent reviewers. But there’s more.)
You should note [my correspondent’s words, but my italics] that the State Board explicitly directed APA [Augenblick, Palaich & Assocaties, the Denver-based consulting firm that did the analysis] to not separately break out the results of each of the methodologies it used to determine the base cost and weights. This decision was based on our observations on how the costing out studies in other states was used. … [Omitted here is a mini-lecture on how all methodological approaches have strengths and weaknesses.]
The Board wanted the focus of the discussion about school funding to be about adequacy and equity and not about which of the methods supported a given political position.
All this makes a case for transparency in school finance analysis, not a black-box (or closed stew pot) approach. It’s easy to dismiss the 2007 costing-out study as an expensive piece of political theater, but it was also a missed opportunity. It would have been good then to have had actual data on how high-performing schools spent money, especially those schools that managed to do well while spending less than their academic peers. It would be great now to have baseline data collected about four years ago on a sample of 80 high-performing schools. If we had that, I’d lobby for spending on a follow-up study to see if those schools maintained their fine results and to see if their spending patterns prevailed. That we don’t have any of this – by design – is worth remembering the next time you hear anyone calling for more research on school cost-effectiveness.
