Buying Quality Performance

In the US a significant proportion of charter schools are delivering weaker than expected performance, with little consequence for the contract holders. On Monday (19^th March) Chris Grayling, the UK Minister for Employment, faced the Department for Work and Pensions (DWP) Select Committee and underlined his commitment to culling failing Work Programme providers and transferring clients to their more successful competitors.

Proponents of opening up public service provision often focus on the ability of new organisations to deliver innovation and drive up performance. There are examples of this happening – in the best cases lessons learnt have been applied by providers from all sectors and improvement has been felt across the board. However, what happens when organisations brought in to raise performance fail to do so? Do they face real consequences for missing targets? Or are they protected by the very contract procurement and management systems that should hold them accountable?

The American charter school model has been much discussed on both sides of the Atlantic. Some charter schools have improved educational outcomes for their pupils, on occasions in remarkable ways. However, it is currently the case that a third of charter schools are performing worse than the public schools they were set up to replace (and 47% demonstrate little difference). The theory was that in exchange for taxpayer funds and flexibility to move beyond the traditional school format, charters would be highly accountable. Closing ineffective schools was central to this theory. This accountability has rarely appeared in practice.

“The issue of quality is the most pressing problem that the charter school movement faces,” said Dr. Margaret Raymond, director of CREDO[1] at Stanford University, “…it needs to…focus on removing the barriers to exit, which means closing underperforming schools.”

In the 2010-2011 school year 6.2% of charter schools did not have their contracts renewed – about a fifth of those that are underperforming. This is down from 8.8% and 12.2% in the previous two years. The declining closure rate is not indicative of continually improving performance – indeed Greg Richmond, President and CEO of the National Association of Charter School Authorizers (NACSA), suggests that a more appropriate annual rate of closure for failure to meet targets should be 15-20%.

The UK welfare to work market tells a similar story. There is a sense that there is no consequence for poor performance. Contracts are rarely pulled, and previous performance is not considered during tendering processes. This has a de-motivating effect on other providers, both prime and sub contractors, as well as Jobcentre Plus managers in those Districts. It encourages lower standards with contractors allowed to prioritise bottom line performance over service impact.

Part of the problem is that it is not always as easy as it should be to comprehend how successful introducing additional providers has been. Understanding how different providers (from public, private and voluntary sectors) are performing is made difficult both by the inaccessibility and lack of transparency of much of the required data and the measures selected for analysis. There is no, or very limited, comparable benchmarking.

This forms a strong contrast with the Australian welfare to work star rating system, which compares performance of providers using regression analysis to control for exogenous factors such as labour market conditions and personal characteristics of the jobseekers. On the back of this evidence base, good performance can be rewarded with increased market share and contract renewal.

In the UK, comparative analysis for those performance league tables that are published is generally simplistic. It is not designed to capture and analyse in depth the complexity of performance. It plays no role in the reassignment of contracts or the award of new ones. There are, of course, issues with the validity of classification but the importance of examining, understanding and weighting the various factors that make up provider performance is essential.

Private and voluntary sector providers delivering a variety of public services have been given increasing levels of freedom to deliver success. Flexibility is central to delivering innovation and with it, quality performance. This needs to be matched by accountability. Challenging and ultimately removing contracts from those organisations that do not meet expectations is a sign of the strength and sophistication of the contract, rather than the weakness of the programme or the political policy. If this holding to account occurs within a context of continuous improvement, clear measurement and transparent reporting then it is the sign of a healthy system.

Richard has discussed some of the practical constraints on contractor change in his piece ‘The myth of multiple contractors’. However, this is not an argument for abdicating accountability. If anything, it reinforces the argument, with rich performance analysis the best possible way to predict and thereby head-off failure.

Dr. Raymond again: “If the charter school movement is to flourish, a deliberate and sustained effort to increase the proportion of high quality schools is essential. The replication of successful charter school models is one important element of this effort. On the other side of the equation, however, authorizers, charter school advocates and policymakers must be willing and able to fulfill their end of the original charter school bargain, which is accountability in exchange for flexibility.”

There are a number of steps that are essential to achieve accountability:

1. Improve performance measurement – both quality and transparency;

2. Ensure quality information is given to bidders, and when mistakes are discovered in this information, or the contact changes, allow reasonable amendments to agreed performance targets;

3. Assess ability to deliver – previous performance needs to form an important part of the bid assessment process. New entrants to the market can be scored differently to ensure equity, though should, anyway, be able to reference some sort of relevant, quantifiable performance;

4. Sit accountability at the heart of all contracts – remove failing providers and support successful models to expand, with operational contingencies in place to enable this.