Canberra Evaluation Forum: Evaluation And Australian Public Service Reform

Steve Sedgwick
Public Service Commissioner

17 February 2011

 

Good morning.

Let me begin by acknowledging the traditional owners of this land, and pay my respects to elders both past and present.

Introduction

I am pleased to have the opportunity to speak to you today.  I’m aware that it has become something of a tradition at the Canberra Evaluation Forum to begin with a keynote address on a broad but important subject in relation to evaluation in the public sector. 

Previous speakers on such topics have included the Auditor-General, Ian McPhee, and the Secretary of the Department of Finance and Deregulation, David Tune. 

As many here would be aware, particularly those who may have read my overview in the most recent State of the Service Report or my piece in the December issue of the Public Service Informant, it was quite a year in the APS and at the Commission.

The Review of Australian Government Administration, colloquially known as the APS Reform Blueprint, was completed and its 28 recommendations accepted by the government. We had a new prime minister and a federal election – resulting in a hung parliament, a new ministry - indeed a new minister for the Commission.

The APS delivered a huge agenda for the government and people of Australia, including a raft of programs conceived and executed quickly as a response to the GFC.

However there were also serious failings and consequent damage to the public service’s reputation resulting from the fall-out from the OzCar affair and the deficiencies of the Home Insulation and Green Loans programs; and some issues in some states about implementation of “Building the Education Revolution”.

As I said, quite a year.

But a year that reminded us, yet again, of the considerable long-term challenges facing the APS.

These challenges include meeting the combined pressures of:

• increasing governmental and citizen expectations of the quality and relevance of government programs;
• growing policy complexity and growing acceptance that issues should be approached holistically;
• growing perceptions that there is a much broader range of administrative and analytical  tools now available to address complex issues holistically and that governments should use them to solve almost any problem;
• as well as a tight fiscal environment, which is likely to endure for some years.

The Government remains committed to the Blueprint reforms, even though the Budget situation requires the Commission to implement them as far as possible within a more heavily constrained budget. 

Resourcing is a major challenge and will slow down – and possibly curtail – some of the implementation, but not the thrust of the government’s APS reform agenda or the reshaping of the strategic direction of the Commission. 

Consequently, our future strategic direction remains clear. 

We believe the reforms will make a significant positive difference to the future APS and its capacity to develop and implement effective policy responses to the challenges that confront us as a nation.

And I’m sure, as evaluators, you will be relieved that in creating the Blueprint we examined the evidence, analysed causes and effects and identified necessary reforms, a number of which have been proven internationally to produce better outcomes for governments and citizens.

David Tune made the point in his speech last August that in terms of evaluation, Ahead of the Game is quite critical of elements of the APS’ collective performance and sees a clear need to build and embed a stronger evaluation and review culture, noting a possibly lower investment in evaluation in Australia in comparison to other countries.

He also noted that while ‘some agencies maintain a best practice, coherent and well coordinated evaluation function, with well developed and stable internal evaluation capability and partnerships with external expert consultants, others appear to be less focussed and there can be questions about usefulness, objectivity, transparency and openness.’  A point backed up by some ANAO reports.

And you may well remember that an important recommendation in the Blueprint is the growing need to focus on whole-of-government outcomes (since the problems are joined up in the citizen and often require more than one program or agency to address them), which in turn challenges us to design and evaluate cross-agency and whole of government programs.

In a recent speech, Ann Sherry, a member of the Blueprint Advisory Group provided a hypothetical case in point.

She envisioned a world in which a new mother registered online and then had access to all of the Government services she needed to receive including midwife support, information on immunisation, an invitation to a local mother’s group and a range of other local services.

Of course this picture is far removed from current reality.

It implies joined up delivery not just at federal, state and local government levels but also at the community sector.

Such a high level of integration would be difficult to achieve: but it is a legitimate aspiration.

Purpose and Scope

However I digress.

David Tune spelled out for you the opportunities that the Blueprint has identified to improve program evaluation across the APS.

Today, I want to make a similar point regarding another area of APS activity, namely our management of the human capital of the APS.

Because we have room here also to benefit from the application of the evaluator’s arts.

And we have work to do to refresh or develop appropriate evaluation tools and to make those tools and a concern for program effectiveness part of the tool kit available to every senior manager.

So the plan for today is as follows.

As the title of this speech suggests, I will start with some background on recommendations arising out of the Blueprint for the Reform of Australian Government Administration. 

I won’t go into the detail of the process of the Review itself as I’m sure that most of you are well aware of that process.

But I will talk about some of the reforms in light of their evaluative characteristics and the strong emphasis in the Blueprint for evidenced-based approaches to policy development and program assessment.

There are indeed a few Blueprint initiatives that we ourselves are implementing that we need to evaluate as a matter of course.  I’ll outline these as we proceed.

Two specific recommendations in the Blueprint that I will discuss in more detail relate to Human Capital – including Workforce Planning – and Capability Reviews. 

The Commission is in the process of developing these recommendations into useful tools and services that we can deliver to the wider APS. 

Also, in respect, particularly of Human Capital, I’ll spend a bit of time discussing benchmarking. 

Benchmarking is something of a theme in the Blueprint.

It is, of course, a distinct form of evaluation practice that is often talked about, but rarely implemented or understood properly.

I’ll conclude with some remarks about the direction the APS is taking in respect of evidence and evaluation and the sorts of things I think agencies need to consider in developing their workforces to be as effective as possible in the political, social, cultural and international environments that we see now and expect in the future.

The Blueprint and Evaluation

The Blueprint concludes that the goal of the reforms ‘is to transform the APS into a strategic, forward looking organisation, with an intrinsic culture of evaluation and innovation’.

Developing innovative policy, programs and regulations based on evidence seems to be a no-brainer. 

Putting the principle into practice, however, is not always straight forward, especially if there are not well developed measures of outcomes achieved; or established causal links between actions taken and outcomes observed.

And sometimes the time frames to develop policy positions are short, which may challenge us if we have not previously anticipated the issue and begun to assemble the right analytical tools and an evidence base.

Ideally, we need systems that are informed by evidence at every stage, from policy development to service delivery; from when an issue is first identified, to the development of the most appropriate response, and the subsequent evaluation of its effectiveness.

This is even more challenging when dealing with complex problems, like Indigenous disadvantage and climate change, when the evidence on which responses must be based is shifting rapidly, requires a number of assumptions and complex modelling and involves many different interactive elements. 

So, then, there is room and a need for all kinds of research, review and evaluation along the continuum from formative to summative.

This means not only collecting data and investing in research, but, importantly, ensuring that policy makers have the right skills to discriminate between evidence which is reliable and useful, and that which is not.

The development of effective academic partnerships is an underdeveloped way to ensure that there is sufficient rigour around our work.

The Blueprint was strong on the need for the APS to establish wide networks, including with academics and academic institutions because the problems are often multidisciplinary and require fresh, possibly systems thinking.

We in the Commission are certainly establishing these to assist us in our own work.

The Blueprint was also strong on the need for the APS to improve its capacity to look forward and begin to research emerging issues before they become too pressing and there are demands for “instant” solutions.

And the Blueprint is also strong in the view that the APS requires integrated information systems and management strategies to ensure more coordinated, responsive and effective service delivery across the APS.

And it aims to embed a culture of continuous improvement, stating that by developing better data, more responsive systems and regular reviews, the APS can become a more flexible, innovative organisation that is alert to the needs and preferences of the community and focussed on remaining effective as their circumstances evolve over time.

The Blueprint specifically recognises the value of citizen feedback and, indeed, of the importance of building strong links between front line staff and policy makers. 

Without access to information that links like that can provide, it can be difficult to adjust or re-design policies and programs in ways that best meet citizen expectations or needs as they change over time.

Moreover, a number of jurisdictions have shown the value of capturing the qualitative experience of both citizens and employees, since program data alone may not tell us whether we are achieving our goals. 

These tools can also provide a read out on the degree of trust that citizens have in government – the maintenance of public trust is an important underpinning of a well performing APS, and has become a key measure of government and public service performance in some other countries, notably Canada and New Zealand.

Many agencies already collect information and feedback from citizens about programs and services but there is scope to improve the comparability of the data collected.

This is one reason the Blueprint proposes that Australia conduct a national Citizen Survey and adopt some common questions for agency surveys (known as the Common Measurements Tool), as Canada and New Zealand currently do. 

Human Capital

However the key point that I want to bring to your attention today is that the Blueprint challenges us to take evidence based approaches and the tools of evaluation into the very management of the APS itself. 

This is because the Blueprint has a lot to say about the need to plan for and develop APS capability - to ensure that the APS is both well equipped to meet today’s challenges but is also developing the skills necessary to meet evolving issues that can be anticipated and is resilient in the face of the unexpected.

And it calls on the APS to invest more systematically and more purposefully – i.e. more effectively - in the development of its human capital. 

I’ve discussed how Human Capital management differs from current practice in a speech last October, which is available on the Commission’s website, and I won’t repeat that in any detail here.

Suffice it to say that human capital is an inherently dynamic concept, which acknowledges our renewed emphasis on the fact that the APS is an enduring institution which needs to plan for and invest in its workforce – an institution, in short, with obligations to the future as well as to the present.

The Blueprint adopted the term ‘Human Capital’ to signal a distinct shift towards such a more systematic and future-oriented approach to the building and sustaining of APS workforce capability in order to maximise the effectiveness of each business or organisation. 

In this framework, the performance of people matters, because of their impact on the success of the business. 

Indeed, we are seeking not just to build the capabilities of individuals but, just as importantly, to build organisational capability, and to do so not just in respect of individual agencies but also of the APS as a whole.

There is a very strong “One APS” theme in the Blueprint. 

This has implications for the leadership capabilities the APS requires to respond to our future challenges, the way these capabilities are mobilised and used across the APS, and more generally the workforce planning practices we use to anticipate and build our APS workforce capability.

I won’t say much more about this issue at this juncture, except to say that one of the fundamental features that unites our “one APS” is the set of values that we hold in common.

You may recall that we have been consulting members of the APS and the community at large about a proposal in the Blueprint to simplify our current set of values and make them more meaningful for our work.

The feedback that we have received in this process has been very helpful and has directly influenced the proposals that we will be putting to the government in the near future.

If you were one of those who responded to our invitation to contribute your views then please accept my grateful thanks.

We hope to be in a position to talk further about proposals to go forward to parliament in the near future.

However, let me return to the issues raised in the Blueprint and subsequently about the capability of the APS.

An important point to make here is that organisational capability is not just about skill sets of staff.

Rather organisational capability is determined by an agency’s people, processes, systems, structures and culture – and how effectively they are combined to produce outcomes.

ANAO reports over the years have attested to the fact that significant failures frequently occur in organisations because one or more of those systems failed or did not work harmoniously together – the Home Insulation program and Green loans scheme are cases in point but there are earlier examples, for example in the previous Immigration department (see the Comrie and Palmer Reports).

We invest in people not only for the good for the individual, but also for the value that can accrue to the organization - value that can be understood and verified in terms of improved business productivity and performance.

Indeed, one of the challenges thrown up by such an approach is to develop the metrics and evaluative tools that enable us to assess whether our investments in enhanced systems and in the skills of our people actually enable us to create the workforce capabilities that effectively meet the future needs of the community.

Good metrics and analytics help to keep the people strategy targeted and focused on real business results. 

An example I’ve used before is that we need to know, not just whether participants were satisfied with the training course, but whether business outcomes improved as a result. 

For example, we need to show that any future investment we make to develop leadership capability actually leads to behavioural change back in the workplace and that this in turn contributes to the achievement of better program outcomes.

This is an extremely challenging area of research, but one that we must pursue.

We also have a broader responsibility to contribute to the academic evidence base in this area.

As I said earlier, this is one reason why it’s so important that we establish a range of academic partnerships to assist at each of the research, design, delivery and evaluation stages of our work.

For many APS agencies this task will present a challenge because we do not have a well established culture of measurement or evaluation on this aspect of APS activity.

However, I believe that our strengthened focus on Human Capital and the development of organisational capability naturally lends itself to the methods of evaluation.

And we have incentives to develop these skills and techniques.

Because, until we develop reliable measures of Human Capital and organisational capability in the public sector it will be difficult to present convincing evidence to senior managers and external observers of the efficacy of the investment of additional resources in the development of people and systems to manage them.

This does not mean merely picking up the measures used by the business community and applying them to the public sector.

There is much we can learn from business but there is also much that is distinctive about the public sector.

Accordingly, we will make recourse to public sector evidence and data a central feature of how we in the APSC pursue the human capital challenge that the Blueprint has laid down.

The Blueprint also challenges those of us who work in the Commission to do more than just measure, record and describe.

We too, are challenged to become more business driven, to pursue improvements in human capital practices which improve outcomes for citizens or lower costs. 

This gives us an added incentive to better understand APS agencies and their challenges.

How will we know whether a stronger focus on the APS’s human capital is successful or not? 

Human Capital Benchmarking

A component of our proposed approach is through Human Capital Benchmarking, which the Blueprint has recommended that the Commission embark upon. 

Some of the data for this benchmarking might be drawn from the proposed new agency Capability Reviews (about which I will speak later) and, to a lesser extent, the Citizen Survey.

Benchmarking is a commonly used term, but there is confusion about what is actually meant by it. 

In an article that some of you may be familiar with, John Alstete tells us that benchmarking was conceived originally as a continuous quality improvement process, in which an organisation striving for better practice compares its performance with another in an effort to learn and progress. 

However, it is often operationalised now as a simple ranking process.

Terms such as ‘standard’, ‘target’, ‘benchmark’, ‘benchmarking’, ‘performance measurement’, performance statistics’ and ‘performance indicators’ are often used interchangeably.

It is important to make the distinction between the establishment of a point of reference and undertaking a comprehensive continuous quality improvement process. 

We intend to approach human capital benchmarking in the manner that Alstete suggests, namely as a continuous quality improvement process –

• One that is used to identify and understand the practices exhibited by the best in their field;
• to adapt and improve those practices, for the purpose of reaching the targeted level of excellence; and
• then surpassing it with even better practice.

Our approach to Human Capital benchmarking will also be consistent with the Blueprint’s focus on improving the APS’s organisational capability (which, let me remind you, concerns the combination of people, processes, systems, structures and culture that ensures effective performance).

Human Capital benchmarks, then, are measures of the mix and level of the workforce’s attainment or embodiment of these characteristics compared to the business requirements of the organization.

For example, agency workforce plans could be benchmarked against a set of standards or criteria, possibly developed by the Commission, which seek to assess the quality of links between workforce planning and strategic planning within organisations.

Because the real value in workforce planning lies in the links drawn to the strategic priorities of the organisation and how to achieve them and not in some mechanistic application of input – output techniques to the workforce.

Similarly, the Blueprint also recommends that the capability of the APS to provide high-quality strategic policy advice also be benchmarked.

The Prime Minister’s Department has started work to define what good strategic policy is. 

Once that work is completed agencies may well be able to compare how their policy advising function compares with other APS agencies, or even with agencies in other domestic and international jurisdictions.

We know from past experience that evaluation of policy advising presents particular challenges – however that should not prevent us from looking afresh at this issue.

In both of these cases, however, an important component of the work will be to establish approaches that lend themselves ultimately to an evaluation of their impact on the performance of organisations.

That will be an interesting intellectual and practical challenge for our evaluators – including because, unlike the private sector, we cannot reduce everything to an assessment of the impact on the financial bottom line.

Capability Reviews as Evaluation

One emerging tool in this area is known as a Capability Review. 

This tool presents us with a different kind of public sector evaluation.

You might recall that the Blueprint proposed that the Commission undertake regular and systematic review of the capabilities of the major APS agencies, in order to promote improved capability in the key agencies and to assess the institutional effectiveness of the Service as a whole. 

The Commission is currently developing an approach to implement Capability Reviews, drawing on the UK’s experience. 

This will be trialled shortly in two agencies, beginning with the Department of Climate Change and Energy Efficiency.

The government will decide, in due course, whether or not to proceed with the approach that I am discussing here, informed amongst other things by an evaluation of how well the UK’s concepts translate into our context.

Incidentally, the idea of assessing organisational capability is not new in Australia. 

The concepts of capability reviews have been explored in the Commission’s Contemporary Government Challenges series over recent years.  These have examined emerging capability issues in the APS.

The concept is broader than a routine efficiency review or performance review, and it pays more attention to culture and governance issues than does a skills audit.

As one submission to the Advisory Group that developed the Blueprint put it: ‘The policy and delivery challenges in the future are likely to be far greater than experienced in the past and a range of new capabilities are required to respond to this environment.’

We need to continually reassess whether that statement is true and, if it is, identify what capabilities are required to meet emerging needs.

If it is done well, the capability review has the potential to provide an independent and well-rounded perspective on the totality of agency capability - a practical perspective and diagnosis forged in collaboration with an agency’s leadership that they can take on board, to make sustainable improvements in their implementation and delivery.

As previously noted, the proposed approach draws substantially on the United Kingdom (UK) Capability Review Programme. 

Also, we were mindful of what New Zealand and Canada have been implementing in this field.

The UK model was initiated in 2005. 

An evaluation of it last year by the National Audit Office in the UK found that as it has evolved and matured, it has become a catalyst for strengthening the UK Civil Service’s corporate capability and had put capability improvement onto management board agendas across Whitehall. 

Later in 2009, the UK Cabinet Secretary issued a progress report on the capability reviews. 

By then, twenty-two widely varying UK government departments had been reviewed.  Sixteen of these had been re-reviewed.

The follow-up reviews showed that capability had improved in key departments. 

Areas needing development had often been addressed, while continuing areas of weakness were now identified in plain terms.

The UK approach targets the underlying capability issues that tend to determine the effectiveness in delivery of public services - questions such as:

• Do departments have the right strategic and leadership capabilities?
• Do they know how well they are performing, and do they have the tools to fix their problems when they underachieve?
• Do their people have the right skills to meet current and future challenges?
• Do they engage effectively with their key stakeholders, partners and the public?

You will recognise that these are the sorts of broad questions that you might start out with in scoping an evaluation.

In this sense Capability Reviews are a form of evaluation, but one that is crucially different to the sorts of evaluation that we are used to. 

That is because the subject of the evaluation or review is not a program, process or service, but an organisation or agency.

It is in fact an entirely new approach to evaluating effectiveness in the public sector in Australia.

Before taking this argument further, let’s explore a little further what we mean by capability.

It is a word often used in the public and private sectors but less often defined.

Earlier on, I said that organisational capability was ‘determined by the combination of people, processes, systems, structures and culture’. 

The capability of the APS might also be described as:

• a distinctive set of APS qualities – for instance, our APS Values might be seen as central to our capability
• a set of abilities that the APS maintains for government in order to create public value – broadly these might be policy advice, regulatory services, a capacity for service delivery or program implementation
• a set of core organisational competencies such as agility and responsiveness
• various characteristics that are embedded in the assets the APS maintains – say, its people, its technology, and its regional presence.

Indeed, capability has all these meanings.  

But, in the case of the APS, practical organisational success also depends, I believe, on harnessing our business systems and our processes so as to convert these qualities, abilities, competencies, and assets, into tangible capabilities that consistently deliver good value to government and to citizens.

Drawing on the UK’s experience we will operationalise these concepts for the purposes of the Australian capability reviews by assessing how well the agency exhibits capability in three key areas. 

These are the quality of the people, processes, and practices, in place to support the three ‘pillars’ of:

• leadership,
• strategic policy development, and
• service delivery and implementation.

This concept of capability, I might add, links closely to the concept of human capital that we discussed earlier.

Indeed, we know that much of the capability that resides in organisations is built on the social capital that exists between people and around teams.

This capability is enhanced by people developing, sharing and exchanging information. 

A web of social relationships is constantly being formed and reformed, and similar could be said for the relationships between the agencies themselves.

If the APS is to be more coordinated and collaborative in its approach so as to address complex, multidisciplinary issues as those that are “joined up” in the citizen, then we should understand better this web of interactions within and across agencies, in order to understand, assess and improve our capability.

The 2009 UK audit, which I referred to earlier, touched on the importance of understanding the quality of institutional relationships as a way of understanding capability. 

The audit, and the ensuing progress report by the Cabinet Secretary, said that the UK model for capability review could be improved by strengthening the links between capability and delivery, and by placing greater weight on collaboration and working together across ‘complex delivery chains’.

So, while it is essential to assess the components of capability in terms of leadership, strategic policy development and service delivery, it is the way that these three ‘pillars’ interact and combine to deliver outcomes that is the key. 

We will certainly bear this in mind if we subsequently proceed to fully implement capability reviews in Australia.

Importantly, we would be striving to establish capability improvement as an established feature of the APS strategic agenda, as an expected feature of each APS agency’s executive or board deliberations.

An ongoing cycle of capability reviews would provide ongoing practical assessments as to whether agencies have the right people, processes, systems and culture to deliver the highest quality services for the Australian public, and the best forward-looking advice to the government.

Moreover, as agencies become more familiar with the process, we would hope to see that measurement of organisational capability had become a key consideration in the strategic planning processes of each agency, leading to fewer instances of incomplete or ineffective implementation of programs and similar issues. 

We at the Commission believe that well constructed capability reviews could provide a powerful addition to an agency head’s armoury of diagnostic tools and would provide a platform for identifying best practice, and for facilitating its service-wide adoption through information-sharing networks.

Cumulatively, the reviews would help the APS to identify its ‘centres of excellence’ on the one hand, or its recurring capability gaps on the other hand.

However, at the moment all of that lies in the future.

The trials that we plan to complete over the next six months or so will give us a better basis for establishing how well this promise may be realised in practice.

The reviews are not cheap so we will plan to evaluate our experience with the trials before taking any proposition to the government.

Conclusion

We have covered a fair bit of territory in the last few minutes so let me conclude with a quick summary of the key messages of this presentation.

The first key point is best stated by returning to an earlier quote from the Blueprint.

The goal of the Blueprint reforms is ‘to transform the APS into a strategic, forward looking organisation, with an intrinsic culture of evaluation and innovation’.

Evaluation is a significant theme in the Blueprint.

It made particular recommendations to strengthen the use of evidence and evaluation in respect of programs.

But we aim to take that approach one step further and we aim to apply evaluation techniques more consistently in the broader management arena as well.

We will need to think carefully about our approach as we grapple with concepts like Human Capital and Organisational Capability. 

And we will experiment with new tools, like Human Capital Benchmarking and Capability Reviews.

In doing so I expect that we will push the boundaries of evaluation theory and practice – not for the sake of doing so but in order to bring powerful new tools to bear on improving the effectiveness of the APS and its capacity to develop and deliver policy, regulations, services and programs that best meet the evolving needs of contemporary Australia.

I look forward to working with you, and organisations like this, as the Blueprint reforms are implemented.

Thank you.