PROGRAM EVALUATION AND PERFORMANCE AUDITING: BIRDS OF A FEATHER?

Presentation to
Australian Evaluation Society

By Wayne Cameron,
Auditor-General of Victoria

16 November 2000

Brief background to development of performance audit and program evaluation

The traditional role of a public sector auditor was one which primarily involved the conduct of a financial audit of an agency. In recent times, these agencies have been required to prepare annual financial statements which led to an expression of an opinion on the financial statements.

The role has expanded since the mid-1970s to meet the additional information needs of Parliaments and the community. These needs have increasingly demanded answers to questions directed at the legislative auditor and asked whether government programs are administered in an economic, efficient and effective manner. In the early years, these questions were examined in the context of waste, probity and compliance auditing. But the specific emphasis called for a wider examination about program effectiveness as well as efficiency. This led to the development of value-for-money auditing, comprehensive auditing (Canada) or performance auditing as it is now known. Note, however, that although it’s a pretty well accepted notion in many Westminster jurisdictions, legislative enthusiasm has not followed in all cases, e.g. Queensland - no performance auditing - the fear is said to be the potential to criticise policy.

Developments in the public sector’s operating environment such as the introduction of accrual accounting, focus on output and outcomes (reporting achievements in non- financial, qualitative as well as quantitative terms, and emphasis on service performance, management and measurement, has impacted on the quality of government reporting. With considerably more (and hopefully credible) information at our disposal than in the past about how organisations perform, the public sector auditor can focus on areas of public interest and risk. Public sector initiatives for management to report the results achieved by programs and services are reflected in the public sector auditors’ interest in focusing more on the results achieved by entities.

Program evaluation

The roots of program evaluation are in the area of applied social science which asks questions about “what if” and “why”, and relies on theories for causal explanation within the data collected and analysed.

Compared with performance audit, program evaluation has an earlier history. Studies resembling program evaluation can be traced as far back to the late 1800s to the early 1900s where studies in educational achievement and agricultural research were conducted in England. In the USA, however, large-scale evaluations of government policies and programs were not commonly performed until the mid-1950s and featured in areas of defence and social programs such as education, health, crime and delinquency.

Evaluations were, at this early time, focused on resource allocations to programs/activities from both a retrospective and a future effects perspective. The analytical techniques utilised were developed largely by economists and included policy analysis, cost-benefit analysis, systems analysis, operations research, surveys and computer-assisted statistical analysis.

Development and increasing use of program evaluation over the past 50 years has, like performance audit, responded to the increasing accountability demands on public sector management. The expectations of specific stakeholders like funders and policy makers have undergone significant shifts influenced by the dynamic socio-economic and political environment in which they operate. Increasingly, agencies are undertaking reviews of the way in which they deliver programs/services; the effectiveness of their operational strategies and the way they allocate resources.

Key characteristics and differences

What is a performance audit?

Performance audits can be broadly defined as audits that evaluate whether organisations use their resources economically and efficiently in order to effectively meet program objectives.

Performance audits may examine a particular program or the totality of an organisation’s program management, including its internal audit and evaluation activity. It may extend to an examination of similar programs across numerous public sector entities.

Performance audits are conducted in accordance with Auditing and Accounting Standards (specifically, AUS 806 and AUS 808 are concerned with performance audits), and aim to elicit unassailable evidence that will lead to an assessment by the Auditor-General about the performance of specific government programs or activity areas.

What is program evaluation?

Program evaluation is a disciplined assessment of government programs and activities. It is based on independent, systematic measurement and analysis, carried out to meet expectations set in policy and standards.

Evaluation concerns itself predominantly with measuring the effects, both intended and unintended, of policies and programs, and assessing the degree to which the desired objectives have been achieved. Evaluations may question the effectiveness of policies and programs in achieving the Government’s objectives, and whether allocations of resources are in accord with government priorities and objectives.

Evaluations may be directed by management, or externally by the program funding source (sponsor). Evaluation is an important tool to assist program managers to identify shortcomings in its administration. Evaluation may result in changes in program delivery methods or refinements or amendments to government policy. It is also an accountability mechanism which provides assurance of the efficient use of resources in effecting government policies.

An evaluation involves judging merit or value of an activity based on formal, agreed-upon procedures grounded in the principles of social science. Evaluations deal with descriptions and explanations of program effects.

What is the difference between an audit and an evaluation?

This question is discussed in a lot in the literature and there are numerous opinions on the subject. Although there is not as much difference to the methods employed, there are some important operational differences which explain why the outcomes can differ. In a practical sense, performance audits are very much like evaluations. Auditors use evaluation tools and techniques, gather evidence, and focus on the results achieved by managers and programs. They seek to measure the quality of program outcomes in their broadest sense. Differences between the 2 disciplines are as follows.

Primary client

While stakeholders in a performance audit environment are numerous (Parliament, agency, community and interest groups), the primary client for an Auditor-General is Parliament.

Evaluators would generally consider the body or person commissioning the evaluation to be the primary client. In many cases, this would be management or the sponsor/source funding the program.

Both evaluators and performance auditors would, however, interact with similar stakeholders to consult in undertaking audits/evaluations.

Initiation of a performance audit

The Auditor-General in Victoria is free to select the activity/program to be audited. The Office has invested heavily in ensuring we do this task well via the establishment of the Sector Planning and Sector Leadership function within the Office’s organisation structure. This group is charged with undertaking strategic audit planning processes (encompassing risk assessment; professional judgement, research and discussions with Parliament and public sector agencies) to identify areas meriting audit attention.

By comparison, evaluators respond to requests of management/sponsors who commission evaluations to address particular needs (e.g. policy requirements; addressing concerns; identifying improvements; the need to assess success).

Therefore, the chief difference is in the way in which audit topics are selected. In our case, we are influenced by the responsibility to provide Parliament with independent assurance about governmental activities. This may be different from the freedom/discretion evaluators have in the choice of area to evaluate.

This independence feature to the relationship is likely to manifest itself in the way in which timing and resource considerations bear on an audit/evaluation. You have an immediate customer - we don’t! Generally, Parliament will not be aware that a performance audit is being undertaken - although steps have recently been taken to bridge this gap in knowledge and expectation through the consultative process with the Public Accounts and Estimates Committee (PAEC) and in the tabling of my Annual Plan in the Parliament which shows the indicative areas where performance audits are being considered.

Reporting differences

The Auditor-General is an independent officer of the Parliament and has a legislative mandate to conduct performance audits and report the results to the Parliament. While input and feedback on the report is sought from the agencies subject to review, reporting direct to the Parliament on the results of an audit reinforces the nature of the difference constitutional relationships between the Executive (Government) and the Parliament, and the role that the Auditor-General has in providing independent and competent advice to the Parliament in order that Parliament can more effectively hold the Executive to account. Once tabled, the report assumes the status of a public document.

Evaluations are at the direction of the commissioning body that seeks the report and decides on the nature of the consequential action. The decision of whether or not the report is publicly released rests with the commissioning body. (The potentially “internal” nature of the report’s conclusions, therefore, may be seen as not without management influence. This is not to say that either review is better or more desirable than each other - each has its use.)

The Office’s independent status (need to be seen to be independent) can often impact on our relationships with the agencies we audit.

Commenting on government policy

The Auditor-General is not permitted, under legislation, to express any opinion or make any comments on the merits of government policy. A performance audit can examine and comment on the quality of policy advice and on the implementation of the policy but not on the right for an elected representative to make the policy decision. To do so would undermine the accountability responsibilities of elected representatives and the Executive.

Evaluators do not face such restrictions. In fact, evaluations may set out to deliberately question the effectiveness and merits of government policies and programs including whether resources are being allocated in accord with government’s developing and intended priorities and objectives.

Reporting results

Both evaluators and auditors report the objectives, scope, evidence and conclusions of the study to their respective clients. The challenge in the performance audit area is reporting on what might, at times, be a very technical, complex and sensitive (when made public) subject, to a generic audience (Parliament). We seek to write in a manner which conveys a clear, balanced message to the broad public, through Parliament. A key factor in achieving success in this area is to comprehensively brief agencies on the report and its findings. Indeed, there are statutory clearance processes in place to ensure our reports are both correct and fair.

Systematic processes

Common systematic processes are adopted by both disciplines in the conduct of the work. Generally, the methodological approach involves specific phases including planning, research; data collection and analysis and reporting. Common methodology is employed by auditors and evaluators to undertake their assignments, including discussions with stakeholders; review of documentation; observation; conducting surveys and interviews, systems analysis, statistical sampling, and cost-benefit analysis. Both disciplines appear to use very similar methodological approaches.

Utilisation of findings

While the objective of undertaking reviews and reporting is to provide a competent assessment which may lead to change, where necessary, in the public sector, reports of findings by both auditors and evaluators do not guarantee that changes will, however, be made. The Auditor-General, independent of the government’s decision-making and policy implementation process, and evaluators reporting to management, are unable to demand that changes be made to operations/policies. Agencies may not utilise audit’s findings because they disagree with audit’s opinion; perceive the recommendations are not feasible; changing circumstances have made the issue irrelevant or they don’t have the power to make the necessary adjustments themselves.

Evaluators may be in a better position of having their recommendations implemented given that they are in, the first instance, appointed to undertake an evaluation. The fact that management or the sponsor is involved in setting the terms of reference etc. may result in both a greater level of co-operation afforded to the evaluator and in the end, a high degree of management agreement and acceptance of the evaluator’s findings (compared with auditors who, at times, experience difficulties in gaining management’s acceptance of the findings. This has a lot to do with the fact that reporting is to the Parliament, and so public).

At the operational level

Building teams: Resources to undertake performance audits or evaluations focuses upon building teams with the appropriate skills and knowledge relevant to the topic to achieve a credible and comprehensive review outcome.

Evaluation teams do have the benefit of greater flexibility as to who can participate in the team. For instance, evaluation teams may include a representative from the agency subject to the evaluation to participate as a team member. Audit Offices are unlikely to second staff from the client agency onto the audit team.

Focus on management systems: Auditors are more likely to focus on client agencies’ management systems, controls and procedures, and are perhaps more concerned with legal and procedural compliance issues than evaluators. This arises out of the public sector auditors’ broader accountability and assurance interest in an issue.

Use of theory: Evaluators tend more to consider theoretical applications in the work they undertake (social sciences background) whereas the auditor tends more to develop a set of audit criteria - frequently in the form of expectations of what should take place then set about confirming or otherwise the factual evidence.

Access to information: Auditors have statutory powers to gain access to information required for a performance audit, evaluators often have to employ a range of techniques to ensure access to, and provision of information from, a wide range of sources.

Auditing guidance and standards: Auditing practice is guided by auditing standards to ensure a high quality of work is achieved (covers sufficiency and appropriateness of evidence, supervision and review of work, documentation etc.).

Impact of changing environment on both disciplines

Increased demands for transparency and accountability, reporting developments and changing client expectations have, and will continue to have, an impact on the future directions for both performance auditing and program evaluation.

Already, we have seen:

• Creation of program evaluation units within agencies to assist with monitoring grant programs. (A recent review by audit has observed the effectiveness of the role these units play in strengthening public sector accountability.);

• Requirements established for the progressive evaluation (in the Victorian public service) of the efficiency and effectiveness of all departmental outputs in achieving government outcomes from 2002;

• Greater liaison with the PAEC on the Office’s Annual Work Plan and specifically on the objectives of individual performance audits to ensure we maximise the value of the work from the Committee’s point of view as the party specifically charged by Parliament to scrutinise the performance of the Government; and

• Reviews by Audit Offices on the extent and importance of evaluative practices in the public sector.

Specifically,

1.Report of the NZ Audit Office on Impact Evaluation (2000).

Some key points:

• The purpose of Impact Evaluation (IE) is to determine actual outcomes from the implementation of a policy. Comparing those outcomes with the desired outcomes (when the policy was formulated) and establishing or confirming the causal link between the means by which the policy was implemented and the actual outcomes.

• IE is a practical tool to enhance the quality of decision-making by the Government and Parliament. Information generated through IE will inform decisions about the design, operation and retention of existing policies and about the nature and design of new policies. IE findings should identify significant issues such as ineffective policy actions requiring modification and the probability that a new policy proposal will be successful. The report considers that major benefits are to be gained by undertaking IE during policy formulation. Evaluation findings would be used to “give shape to policy actions and assist decision-makers to determine the right things to do to address particular policy problems”.

• A survey of NZ departments indicated that while useful evaluative activity is being undertaken (customer surveys, operational and performance auditing, routine measurement of aspects of output delivery) very little IE is occurring and departments do not have a clear understanding of IE.

2.ANAO Report. Program Evaluation in the Australian Public Service (1997).

Some key points:

• Program evaluation is clearly a critical tool facilitating the assessment of program performance. Since the 1980s, there have been a number of mandatory requirements in place for Australian public service agencies to follow in relation to evaluation planning, and the publication and use of evaluation results.

• The performance audit concluded that while there was scope for improvements generally; (agencies had developed overall evaluation plans taking account of issues of strategic importance; mechanisms were in place aimed at assisting a high quality outcome in the conduct of individual evaluations); and agencies had developed strategies to ensure evaluations had impact, i.e. agencies endorsed and monitored implementation of evaluation recommendations.

Do the differences between the 2 disciplines really matter?

Each of the 2 disciplines has come to program assessment via completely different disciplinary development and orientation. Over time, the 2 paths of evaluative inquiry have become less distinct and it is not uncommon to find a mixture of techniques from both streams used in a single performance audit or study.

General consensus is that there are some key similarities (use of multi-skilled teams, methodology, tools and techniques) and differences (role; responsibility and clients) between the 2 disciplines. When it comes to “doing the business” the 2 points of focus are essentially the same - both use any approaches, methodologies and contacts that will get the job done.

Both audit and evaluation have important roles to play in the review of government performance. Both endeavour to evaluate and assess government programs and initiatives so as to inform government policies and contribute to more effective and efficient resource allocation and management.

Over the past 5 or so years, the Victorian Auditor-General’s Office and the Australian Evaluation Society have established a close professional relationship based upon the similarities of the endeavours of the Office, its staff and program evaluators. This has assisted the transfer of knowledge, and awareness, of what is done by the 2 disciplines.

The Office has seen an increasing prevalence of program evaluation within the public sector. The work in grant evaluation referred to earlier being an example. The results of these evaluations will be used for such things as program planning and decision-making, resource allocation and the development of best practice. In undertaking performance audits, we are increasingly seeing evaluation in action, particularly within the major portfolios of education and health. In our work as performance auditors, we have utilised the work of evaluators, assessed the framework by which evaluators perform their evaluations and have sought the specialist assistance of program evaluator when available.

Where to from here?

The relationship between performance audit and evaluation is still evolving. To date, perhaps performance auditors have learnt more from evaluators than vice-versa and this is exhibited in the increased use made of certain evaluation “tools” over recent years to assist the performance audit process, particularly in the area of assessing outcomes/effectiveness of programs. For example, use of focus groups and conduct of customer satisfaction surveys. I would be interested to hear from evaluators what they have learnt from the performance auditing field!

Irrespective of our different roots, we can both learn from one another. Your kind invitation for me to share lunch with you today is evidence in action of an agreed desire to work in the public interest to achieve better outcomes for the society in which we live.