Friday, 15 April 2011

Merit Pay and Pisa

A study in Education Next claims to show a link between countries using performance related pay for teachers and better results in OECD's 2003 PISA test.

The study is by by Ludger Woessmann and is available here and in longer form here 

It is a strangely unconvincing study. We have, on the one hand, a country test score. On the other hand we have survey results for a collection of countries on different kinds of extra monetary incentives that country may employ to encourage or reward teacher performance. As you can imagine the variety of such schemes is extensive, some employed at local levels, some cash rewards from school principals etc. In a country like the USA those schemes, such as they are, are likely to vary from state to state. Note that while the sampling procedure for PISA in a given country is designed to ensure a representative sample for student acheivment that does not mean it was designed in such a way to ensure a represantive sample across regional policies such as merit pay. Further turning such schemes into a metric suitable for cross national comparisons sounds difficult

First though, a sniff test. Britain, Canada, Australia, New Zealnd, USA are all democratic, industrialised, developed nations with strong (at the time) economies, similar(ish) education systems but different scores on PISA. Australia and the UK (in PISA speifically England). Is merit pay a likely factor in explaining those differences? Doesn't seem likely. The merit pay study doesn't include Canada as apparently it didn't provide salary data. Of those nations the USA and the UK have performed not so great on PISA 2003 and Australia and New Zealand performed fairly well. Yet under the classifcation scheme US and NZ both have some performance related pay (1s in two columns in table 1 page 24 of pdf) and the UK has little and Australia none. On the face of it merit pay is an unlikely factor in describing the differences.

Of course the study isn't so naive as to simply compare scores and some combination of yes/no variables. The scores are first adjusted according to a model that will account for more gross differences in factors that will effect performances. That all seems reasonable, although the robustness of any model that accounts for such factors will clealry have an impact on the results of this merit pay study. For example from the graph (figure 2 on the HTML version figure 1 on the pdf) you can see that the UK score has been adjusted so that it is lower than the US score - such an adjustment is a legitimate process but clealry the extent of it has a huge bearing on whether the addition of perfomance pay in the US improves its score above the UK [on the unadjusted scores the UK performed better than the US but there are clear demographic differences so we might expect their positions to reverse if that was accounted for].

Eyeballing the graph tells us a lot. Look at the scatter of countries. "ISL" (pdf) "Iceland (HTML) lies near the bottom left hadn corner. Portugal lies on the right fairly high up. Without these two countries the scatter of nations would look more like an undiferentiated almost vertical blob. The vertical position of those two nations is almost all down to the adjustments made to account for other factors. Iceland is actually a high scoring nation and Portugal a lower scoring one - but clearly they are quite different nations economically (at least in 2003 pre-GFC).

In addition not only do we have reason to be a little sceptical about the adjusted scores but much also rests on the perfomance-pay metric used. Little account seems to have been made of both the frequency and size of any merit pay involved - a scheme that exists mainly on paper or whose financial rewards are slight isn't much of a scheme, or a scheme in which the rewards are common and easily acheieved is probably not much of a scheme either!

In short: interesting but uncovincing study.
See also

No comments:

Post a Comment