PERFORMANCE AUDIT OF GOVERNMENT-FUNDED CAPITAL PROJECTS

Presentation to
Beijing Audit Bureau
15 May 2007

By Des Pearson
Auditor-General of Victoria

Outline of presentation

• History and background

• Role of the Auditor-General

• Audit coverage and client base

• Auditor-General’s mandate and reports

• Audit of government-funded capital projects

• Key elements of project management – audit criteria for reviewing government-funded projects

• Audit of capital project contracting in Victoria

• Public-private partnerships (PPPs)

• Lessons learnt

• Audit of Melbourne 2006 Commonwealth Games

History and background

• Position of Auditor General first established in Victoria in 1851.

• Auditor General – auditing in the public interest.

• Principal aim: To conduct quality financial and performance audits of public sector organisations and comprehensively report to Parliament.

Where are we located?

Our accountability framework

• Under the Westminster parliamentary system, power is vested in government and ministers.

• Executive Government decides on the direction and management of State resources, and must account to the Parliament for its actions.

• Parliament reviews performance through - debates and joint parliamentary committees.

• Public as against private sector accountability.

Role of the Auditor-General

Role of the Auditor-General

• Auditing in the public interest since 1851.

• A key link in the accountability process.

• Constitutional safeguard to serve interests of Parliament.

• Arguably the most “independent” powers for an Auditor-General in Australia.

• Formal relationship with Public Accounts and Estimates Committee (PAEC).

Audit coverage and client base

Client agencies include:

• government departments and other budget sector agencies

• companies, trusts and joint ventures

• public bodies

• public hospitals and ambulance services

• local councils and regional library corporations

• water authorities

• police, emergency services and courts

• universities and technical and further education institutes (TAFEs)

• financial institutions and insurance bodies

• superannuation schemes.

Auditor-General’s mandate and reports

• Financial audit.

• Performance audit

• regard to whether any wastage of public resources or lack of probity or financial prudence in the management or application of public resources

• unable to question the merits of policy objectives.

Financial audit

• Opinion on whether entity financial statements:

• give a true and fair view

• comply with Australian Accounting Standards.

• Report on the State’s:

• Estimated Financial Statements

• Annual Financial Report.

Performance audit

• Determine whether an authority is:

• achieving objectives economically, efficiently and effectively

• operations or activities are being performed effectively, economically, and efficiently in compliance with all relevant Acts.

• Performance indicators.

• Financial benefits given by State or authorities.

• Acting as auditor under the Corporations Act.

• Other auditing services.

Reports to Parliament

• Results of Financial Statement Audits (x2) - 30 June and 31 December.

• Annual Report on State’s Finances (x1).

• Omnibus (multi-topic) reports (x3)

• Smaller audits

• Follow-up audits.

• Single-topic reports (x10).

Audit of government-funded capital projects

• Capital projects – general observations.

• Audit criteria for assessing project management of government funded projects.

Today, I would like to make some brief general comments on capital projects.

I would then like to briefly outline the audit criteria that my Office uses when assessing the adequacy of the management of a government-funded capital project.

Our audit criteria are based, in part, on the Project Management Body of Knowledge (PMBOK). PMBOK is a guide to project management published by an American organisation – The Project Management Institute, and is meant to bring together the shared knowledge and experience of project managers from across the world.

Capital projects – General observations

• Adequate information for decision-making.

• Optimism bias on capital projects (UK study).

• Capital expenditure underestimated by 47 per cent.

• Operating expenditure underestimated by 41 per cent.

• Observe proper process.

• Identify and manage risks.

Adequacy of information provided to senior management and government for decision-making on capital projects

• It is imperative for decision-making on significant projects or transactions to be informed by rigorous information, analysis and advice.

Optimism bias on capital projects

• Mott MacDonald study in the UK of optimism bias on 50 large public procurement projects over 20 years.

• Findings published in 2002.

• On average, Capex was underestimated by 47 per cent, and Opex was underestimated by 41 per cent.

• Highest underestimation occurred for non-standard buildings and non-standard civil engineering.

Observe proper process

• Importance of due diligence investigation into other parties involved in high profile projects, or transactions.

• The fact that a project is large, important and unusual provides more and not less justification for it to be subject to normal decision-making, tendering and contract management processes. To make exceptions is to invite significant risk.

• Make sure that the people on your side of the transaction or project have the skills and capacity to manage it effectively.

• Ensure proper documentation is established and maintained.

Identify and manage risks.

Key elements of project management

• Planning

• Scope

• Economic assessment

• Risk assessment

• Cost

• Time

• Quality

• Communications

• Management

• Human resources

• Procurement

• Probity

• Accountability

• Post-project appraisal

Planning

When examining the planning phase of a project, we apply the following criteria:

• Documented plans should be established for the development, delivery and management of the project.

• Plans should address objectives: leadership, scope, cost, time, -

• Plans should address:

• quality

• procurement

• human resources

• risk management

• stakeholder relationships

• internal/external accountability.

Plans should clearly define the objectives of the project and the relationship to broader government and organisational strategies, objectives, policies and legislation.

Plans should clearly identify and assign responsibility for project leadership and delivery

Plans should clearly identify and assign responsibility for project leadership and delivery

Plans should be approved by appropriate party(s) within government

Scope

When examining the scoping phase of a project, we apply the following criteria:

• The scope of the project should be clearly defined and supported by a detailed statement.

• All available options to address the required service need should be documented

• Variations to the scope should be clearly documented and adequately assessed.

• Procedures should be in place to minimise scope creep – poor control in this area can result in significant cost overruns

Economic assessment

When examining the economic assessment of a project, we apply the following criteria.

The business case should:

• include justification of the project from a social, economic and/or environmental perspective

• show whether the project has a net benefit

• include an appraisal for each option and the assumptions/forecasts used to determine benefits and non-benefits

• undertake a sensitivity analysis on the key benefits.

Risk assessment

When examining risk assessment processes for a project, we apply the following criteria:

• Material potential risks to the Government agency, the State and the community should be identified initially and continually throughout the project – risk assessment is not a once-only activity.

• Impacts of identified risks should be analysed and ranked in terms of importance.

• Reasonable and practical risk management and mitigation strategies should be developed and implemented

• Adequate risk management monitoring and reporting mechanisms should be established, including risk management plan, risk register and documented mitigation strategies.

Typical risks for a significant capital project fall under the following headings:

• Site risk

• Design, construction and commissioning risks

• Market risks (particularly for PPPs)

• Operating risk

• Financial risks

• Network risks

Cost

When examining the costing and budgeting for a capital project, we apply the following criteria:

• Cost estimates should identify total resources required for the project, be realistic and include contingencies.

• Cost estimates should be revisited regularly to ensure currency.

Processes should be in place to identify and manage cost variations

Any capital project creates an ongoing cost commitment in terms of the operation and maintenance of the asset over its life. For this reason, detailed data should be available during the planning phase for a capital project on lifecycle costs, key assumptions and risk adjustments.

Information needs to be known about the lifecycle costs of an asset during its planning phase because such information may be useful in informing decisions about the choice of design and construction materials. For example, when building or renewing a railway line, the choice about whether to use timber or concrete sleepers should be informed by the knowledge that concrete sleepers are considered to have lower maintenance costs than timber sleepers over their respective lifecycles.

Time

When examining the timeline management for a project, we apply the following criteria:

• Project timelines and milestones should be realistic and appropriate.

• Processes should be in place to identify and manage timeline variations.

Quality

When examining the quality management aspects of a capital project, we apply the following criteria:

• Effective quality assurance processes should be in place to ensure that target quality levels and standards are met

For a capital project, this would include an ongoing site inspection program by government representatives or an appointed project superintendent to ensure that the specified and paid-for materials are being used. We would also expect to see ongoing sign-offs by the constructor as to the quality of materials and standards of construction and finish.

Communications

When examining the management of communications for a capital project, we apply the following criteria:

• Mechanisms should be in place to monitor and report on the status and progress of the project to all relevant stakeholders.

• Strategies should be in place to effectively manage stakeholder relationships.

Stakeholders for a capital project would typically include:

• the government agency charged with overseeing the planning and delivery of the project.

• Members of any governance group established within government to oversight the project.

• Relevant senior ministers.

• Central agencies such as the Department of Treasury and Finance (it will want to monitor any emerging cost pressures associated with the project)

• The community - targeted communication may be warranted for communities directly affected by the project.

Management

When examining the overall management of a capital project, we apply the following criteria:

• There should be clear evidence of a coherent approach to the management of the deployment of human, financial and other resources in pursuit of the stated aims and objectives of the project.

• Management of the deployment of resources should focus on activity definition (what has to be done?), sequencing (when does it have to be done?) and duration (how long will it take?) to ensure an achievable project schedule is established. Note, this applies whether the project is to be constructed in-house or contracted.

• There should be evidence of ongoing management and monitoring of the various facets of the project.

Human resources

When examining the human resources aspects of a capital project, we assess whether appropriately qualified and experienced staff have been recruited and retained for the project.

There should be a documented needs analysis in place which identifies the quantum of human resources and skills mix required to ensure adequate management, monitoring and contract management for the project.

Procurement

When examining the procurement phase of a project, we apply the following criteria:

• Procurement activities should be conducted in accordance with legislative and other applicable requirements and guidelines (e.g. Victorian Government Purchasing Board).

• Tender specification and evaluation criteria should be clearly linked to project objectives and expected outcomes.

• The evaluation of bids should be rigorous and transparent (probity).

• Final contract negotiations with the preferred tenderer should be in accordance with the tender objectives and the evaluation criteria.

• Adequate contract management arrangements should be established.

Probity

What do we mean by “probity”?

• Procedural integrity.

• Transparency of process.

• Objective and consistent assessment.

• Address any potential, perceived or actual conflicts of interest.

• Decision-making consistent with key principles:

• Only take into account relevant matters.

• Not apply an inflexible policy or rule.

• Not act under the dictation of another person.

• Or exercise the power for an improper purpose (such as personal benefit).

Why should we be concerned with probity:

• Increased concern with ethics in public life and accountability.

• Need to maintain bidder and market confidence.

• Need to maintain public confidence.

• Defensibility in terms of legal challenges.

Under the audit legislation in Victoria, the Auditor-General can refer matters to the police for investigation if he suspects fraud or corruption, or other serious maladministration.

Accountability

Why should we be concerned with probity?

• Increased concern with ethics in public life and accountability.

• Need to maintain bidder and market confidence.

• Need to maintain public confidence.

• Defensibility in terms of legal challenges.

Under the audit legislation in Victoria, the Auditor-General can refer matters to the police for investigation if he suspects fraud or corruption or other serious maladministration.

Post-project appraisal

In terms of post-project appraisal for a capital project, we expect to see:

• Effective review of the project outcomes against objectives.

• Review of the achievement of the projected benefits and contractors' performance.

• Identification of lessons learnt for future use.

Post-project appraisals should provide valuable data about the long-term viability of the project and inform future decision-making for similar initiatives

Audit of capital project contracting in Victoria

Victoria has engaged in a range of contracting approaches for major capital projects over the last 20 or so years:

• Traditional design and construct contracts

• BOOs (Build, Own and Operate), BOOTs (Build, Own, Operate and Transfer) where the private sector is engaged to build and sometimes operate publicly-funded facilities

• Public-private partnerships (PPPs) such as toll roads where the private sector funds the construction of major capital projects and then operate those assets for an extended period on a concession basis where the public is charged for use and this revenue stream is used to repay debt associated with the construction cost and to deliver a return on investment to the private parties involved.

• Alliance contracts - The State or a government entity acts as “owner” and collaborates with one or more private sector service providers or “non-owner participant” (NOP) for the delivery of the capital phase of a project. Alliances are typically characterised by:

• Collective sharing of all project risks.

• No fault, no blame and no dispute between the alliance participants (except in very limited cases of extreme default).

• NOPs are paid for their services under a “3-limb” compensation model comprising: reimbursement of NOPs’ actual project costs on 100 per cent open book basis; a fee to cover corporate overheads and normal profit, and a gain-share/pain-share regime whereby the rewards of outstanding performance and the pain of poor performance are shared equitably among all alliance participants.

• Unanimous principle-based, decision-making on all key project issues.

• An integrated project team selected on the basis of best person for each position.

Matters reported to Parliament by our Office on contracting processes:

• Absence of cost-benefit analysis. For major contracts in the past, detailed cost-benefit analyses were frequently not undertaken as an important pre-condition to a project proceeding.

• Identification of conflicts of interest. In some cases, no evidence existed to show that departments have requested details of current or potential conflicts of interest where consultants have been engaged.

• Competitive and public tendering processes. Competitive and public tendering processes have not always been applied when procuring consultancy and contractor services, although justification for not doing so has usually been documented.

• Use of a probity auditor. In many cases, the tendering process has not been subject to an independent probity audit in order to attain independent assurance over the fairness and equity of the tendering arrangements.

• Evaluation criteria. Weightings not always assigned to evaluation criteria, and cost often given emphasis over other selection criteria.

• Clear lines of responsibility. Poorly structured contracts with blurred lines of responsibility have led to protracted and costly negotiations and disputes.

• Performance measures. Performance measures are often not defined in terms of business objectives, i.e. they do not cover key elements of the contractor's performance and are not based on reliable data.

• Communication with tenderers. Instances have been identified where communication, including verbal discussions, with tenderers has not been formally or adequately documented to support decisions made in the tender process.

Public-private partnerships

The involvement of the private sector in the funding of public infrastructure emerged as a new development in Victoria in the late 1980s and early 1990s, within a context of substantial budget difficulties and Loan Council borrowing limits.

The focus of early private-public partnership (PPP) arrangements was to establish new lines of financing for much needed infrastructure, while not impacting on the State’s borrowing requirements which were restricted under inter-jurisdictional Loan Council agreements

A key element of the work undertaken by our Office over the past 15 years has been to examine the major PPPs established by the Government with the view to assessing and reporting to the Parliament on the:

• impact of such arrangements on the State’s financial obligations and future financial commitments

• key risks associated with the arrangements

• the integrity of processes followed in relation to project evaluation and approval, and contractor selection, engagement and management.

The nature of these arrangements, including the risk allocation between the public and private sectors, has substantially evolved over the past decade. This reflects the increasing development and maturity of private investors, and their willingness to accept certain levels of risk. Investors are now more willing to accept operational and demand risks than at any time in the past.

The experience of the Victorian public sector in relation to PPPs has been mixed, in part reflecting the key drivers behind the specific projects established.

The first generation of PPPs involved the allocation of substantially less risk to the private sector than later arrangements, as reflected in the provision of substantial guarantees and indemnities by the Government to the private sector, which had the substantive effect of reverting the major financing risks back to the State

Given the Government’s core and ongoing responsibility for public sector service provision, the emergence of PPPs which are generally long-term in nature, pose particular issues for the public sector, including ensuring:

• adequate availability of the infrastructure

• achievement of minimum performance standards

• adequate public safety etc.

In this context, a key challenge of these arrangements is to establish appropriate risk allocation, effective management and adequate oversight, which ensure that the public’s interest is protected.

In the establishment of such models, it needs to be recognised that, irrespective of the model of infrastructure provision/funding, certain risks associated with public infrastructure are difficult and (in some cases) inappropriate to transfer to the private sector (such as a government’s duty of care) - reflecting the public sector’s underlying responsibilities associated with public sector service provision

The risk allocation principles embodied in recent Victorian frameworks acknowledge this important issue and promote the allocation of risks to the parties best able to manage them (at the least cost).

Given the differing characteristics and profile of each PPP project, assessments of the benefits and costs can only be undertaken on an individual basis.

PPPs, such as the Latrobe Public Hospital and the Metropolitan Women’s Correctional Centre, were established in a highly competitive environment, with tenderers submitting low cost proposals incorporating aggressive assumptions regarding available cost efficiencies etc. Subsequent financial difficulties by the private sector operators forced the return of service provision and the associated infrastructure to the public sector. While the original arrangements placed considerable risk with the private sector and allowed the Government to “walk away” from the arrangement in the case of poor performance by the private sector provider, in each case the Government decided to take over the facilities, principally due to its underlying obligation to provide the related public services.

There are a number of later PPPs that incorporate greater risk allocation to the private sector and overall appear to be operating quite successfully, such as the Melbourne CityLink and EastLink. Nevertheless, it needs to be understood that given the long-term nature of these arrangements, the overall benefits to the community may not become clear for some time.

In addition, given changing technology, and community needs and expectations, such arrangements can:

• commit the State to specific delivery models over a long-period; and

• restrict future policy options.

The PAEC in its October 2006 report on PPPs could not conclude on whether the PPP policy is delivering value-for-money over the life of projects compared with traditional public sector procurement methods because of a lack of publicly available information on PPPs and their performance. Our Office will continue to closely examine significant PPP arrangements.

Gateway reviews

In March 2003, the Victorian Government introduced the Gateway Review Process for major construction and systems development projects, based on the process developed by the UK Treasury Department.

Gateway involves the conduct of independent, structured and high-level reviews of medium and high-risk projects, at critical stages of a project. The Gateway Review Process has 6 “gates”: strategic assessment, business case, procurement strategy, tender decision, readiness for service and benefits’ evaluation.

Lessons learnt

In February 2006, the Victorian Gateway unit issued a lessons learnt document gleaned from over 100 Gateway reviews. The primary causes of project failure included:

• Lack of clear and precise links between the project and the organisation’s strategic priorities and objectives.

• Lack of clear responsibility, leadership and ownership of programs and projects by senior management.

• Lack of effective engagement and management of stakeholders and their expectations.

The primary causes of project failure also included:

• Lack of skills and consistent approach to:

• risk management

• project management

• project planning.

• Evaluation of proposal is driven by initial price rather than long-term value-for-money.

• Lack of understanding of, and contact with, the supply industry at senior levels in the organisation.

If you refer back to our audit criteria which I have outlined earlier, you will see that the primary causes of project failure identified by the Gateway review process are covered by our criteria. So we would be examining the potential for such weaknesses in major capital projects that we audit.

Audit of Melbourne 2006 Commonwealth Games

• October 1999 – Melbourne selected to host the 2006 Commonwealth Games

• From 1999 onwards we reported to Parliament regularly on government activities and costs associated with the Games bid and plans for staging the event

In March 2003, the Premier announced that the “net cost” of the games to the State would be capped at $697 million. Total operating and capital “costs” were expected to be $1.142 billion, and operating revenues were expected to be around $400 million.

Our planning for the audit of the games began in 2003 - 3 years prior to the staging of the event.

In planning for the audit of the games we:

• made contact with the NSW Auditor-General and discussed their approach to the audit of the Sydney 2000 Olympics

• attempted to identify key accounting issues and risks which would arise in terms of measurement and reporting of costs and income associated with the games, and we then engaged with the government agencies responsible for the planning, management and staging of the games

• identified key capital projects and determined whether they required specific audit coverage – this resulted in a separate report to Parliament on the Commonwealth Games Athletes Village

• discussed with the relevant government agencies how accountability for the financial and other performance of the games would be discharged.

The State Government determined to prepare a special purpose financial report as the accountable document by which it would provide information regarding the financial aspects of the Commonwealth Games.

Our audit strategy for the Commonwealth Games identified:

• key overall risks associated with the audit (e.g. completeness – risk of omission of costs – Government was motivated to keep the costs of the games capped. This could be achieved by shifting costs to non-games entities where they would not be captured and reported)

• material revenue and expenditure items

• key risk factors and accounting policy issues for each item

• internal audit activity

• planned audit approach for each item.

To address the risk of deliberate omission of costs, we tasked the audit teams at a large number of government agencies who we considered may be involved in providing direct or indirect support or services for the games with undertaking specific testing to identify any expenditure or use of resources for games-related purposes.

Key revenue items included:

• broadcast rights – domestic and global

• sponsorship

• ticket sales

• merchandise

Key expenditure items included:

• ceremonies and events

• security

• operating costs

• marketing and communications

Audit strategy document has further detail.

Our Office issued a report to Parliament in October 2006 on the games’ financial performance.

The State’s special purpose financial report had 2 primary statements: a Statement of Net Outlays by the Victorian Government and a Statement of Net Operating Resources Contributed by the Victorian Government.

The statement of net outlays measured how much extra the State had to appropriate and spend on the games over and above its normal annual operating costs. It also measured the amount of new “capital” spending required on games’ facilities and infrastructure over and above that already announced. This “additional” expenditure is offset by actual operating revenues received and both actual and anticipated “capital” revenues from the disposal of assets; to arrive at an overall net “cost” of the games.

The statement of net operating resources was prepared as though it were an operating statement under Australian accounting standards. This statement differs from the outlays statement in 3 material respects:

• It includes financial resources related to the staging of the games that were funded by agencies from within their existing annual budget allocations

• It excludes “capital” outlays and replaces these with a notional venue hire charge, as a large proportion of the amounts spent on assets remain reflected in their value after the games

• It includes in-kind contributions from other levels of government.

The net operating statement allows users to better understand the full resources consumed in delivering the games. It also disclosed the material resources provided free of charge to the State from other levels of government. However, accounting standards require the value of such resources to also be shown as an offsetting revenue item – so there is no net effect on the State’s bottom line.

This table summarises the key original and revised budget targets, and sets out the final costs as disclosed in the State’s special purpose financial report

(Refer to Part 4 of our October 2006 Annual Financial Report of the State of Victoria.)

Result compared well with budget. In our report, we comment on major outflows and major inflows including resources provided free of charge by other levels of government ($119.7 million)

We also commented that the statement of net outlays includes the current best estimate of the present value of proceeds from the sale of land and housing in the games village development of $81 million. We reported separately to Parliament in December 2005 on the management of the Commonwealth Games Athletes Village project.