 |
|||
 |
|||
  |
|||||
 |
 |
 |
 |
 |
 |
 |
|
| Reports & Publications |  |
|
Home > Reports & Publications > Performance audit reports |
 |
|
Electronic procurement in the Victorian government PART 3: PROGRESS AND ACHIEVEMENTS OF THE EC4P PROJECT WHAT WERE THE EXPECTATIONS OF THE EC4P PROJECT AND WHAT HAS IT ACHIEVED? 3.1 Policy relating to e-procurement, particularly during the early years of its development, has largely been couched within government policy pronouncements relating to electronic commerce or e-government. Specific activity relating to e-procurement began in 1997 and the Electronic Commerce for Procurement (EC4P) Project was established in 1998. Project expectations 3.2 The expectations of the scope, savings and timelines for implementing e-procurement changed over time. Public announcements and information about the expectations of the Project were constantly changing. Table 3A identifies some key announcements. TABLE 3A  Project achievements 3.3 Table 3B, shows the achievements of the EC4P Project at February 2003, compared with the last announced expectations, i.e. those announced in 2002. TABLE 3B  3.4 Table 3B shows that at the end of February 2003, the revised expectations of the EC4P Project have been partially met. Progress of implementation within agencies is substantially behind the announced expectations. Table 3C summarises this progress. TABLE 3C  RESPONSE by Secretary, Department of Infrastructure Table 3C states that the Department of Infrastructure has not rolled-out its e-procurement application and workflow. The Department believes that it has indeed rolled this out as all divisions are using Oracle Internet Procurement V5 (iP5) to raise purchase orders. While a few users in one division were yet to be trained at the time of the survey, the Department believes that it is definitely past the "piloting" stage as defined in the table. Note (b) in Table 3C states that “The e-procurement activities of the newly created Department of Victorian Communities are currently managed by the Department of Infrastructure”. This should be amended to state that the Department of Infrastructure is only looking after the Local Government Division of the Department of Victorian Communities as an interim arrangement. Quality of information for decision-making 3.5 In July 2000, the Minister for Finance announced that savings of between $60 million and $240 million per annum (5 to 20 per cent of materials, repairs and operations purchasing) were expected from the introduction of e-procurement to budget sector departments and Victoria Police. At the same time, the Minister announced that conservatively, savings would be at least $12 million per annum. Evidence to substantiate the size of the announced savings was unable to be identified. 3.6 An important aspect of the audit was to identify whether the expectations of the Project were supported by sound advice. However, advice provided at some key stages could not be substantiated. Advice to ministers 3.7 In May 2000, the Department of Treasury and Finance provided a Ministerial Brief to the Treasurer and the Minister for Finance about the EC4P Project. The Brief sought approval of funding for the Project. The Brief estimated that introducing e-procurement would save $150 million per annum1. The Brief was accompanied by the Project’s whole-of-government business case2, which indicated that e-procurement would save the much lower amount of $65 million over 5 years. No supporting evidence was available to substantiate the savings of $150 million. The Department of Treasury and Finance was unable to explain why its advice to the Treasurer and the Minister differed from the result of the business case. 3.8 To help determine how well-informed the decision-makers were, the business cases developed for the Project were reviewed against Department of Treasury and Finance guidelines3 and accepted practice in financial analysis. 3.9 Two major financial analyses of the Project were examined, i.e.: • The whole-of-government business case (referred to earlier); and • Final business cases prepared by agencies in 2002. These were based on a template and instructions provided by the Department of Treasury and Finance. 3.10 We also examined the consolidated summary of agencies’ final business cases prepared by the Department of Treasury and Finance, for the Minister for Finance4. We did not examine preliminary business cases prepared by agencies. 3.11 The business cases examined were not consistent with some principles of the Department of Treasury and Finance’s policy and guidelines, or accepted practice, and the templates and instructions provided by the Department of Treasury and Finance contained conceptual errors. Agencies used the templates to prepare the financial analyses in their final business cases and the business cases were inadequate. In particular, the business cases: • Included unexplained assumptions about agencies’ purchasing patterns, the proposed extent of use of e-procurement, the speed at which e-procurement would be taken up in the agency, the extent of improvement in data quality and the reduction in the need for maintenance. Each of these factors had a material effect on the financial analyses. The Department of Treasury and Finance explained that some assumptions were derived from anecdotal evidence from businesses in the information technology industry, and that each agency reviewed its financial systems and made informed decisions on what could be addressed through the catalogue or workflow improvements; • Treated productivity savings inconsistently5, e.g.: • In the summary of the financial analysis, the template treated non-cash productivity savings as if they were cash. This is a misuse of the analytical technique employed, which is designed to analyse cash; and • Some business cases provided the results of financial analyses, but went on to explain that the savings could not be realised; • Contained inconsistencies across agencies, e.g.: • Some agencies included costs for training users, others did not; • The estimated cost of processing a transaction varied from $7 at one agency to $38 at another. While some variability is to be expected, this large difference was not explained; and • The proposed extent of the use of e-procurement after 5 years varied from 8 per cent of maintenance, repairs and operational purchases at one agency to 100 per cent at another. Again, while some variability is to be expected, this large difference was not explained; • Made basic omissions, e.g.: • In the case of one agency, adjustments were made to its business case following its decision not to proceed with the electronic catalogue. The adjustments removed all consultants’ costs for implementation and all ongoing software licensing and maintenance costs, even though the e-procurement application and workflow were still to be implemented; • For most agencies, the business cases did not record cash outflows at the time they were to occur, but instead allocated them to future years, contrary to accepted practice and The Investment Evaluation Policy and Guidelines of the Department of Treasury and Finance; • Several business cases spread costs over 6 years, but calculated the final result by using only 5 years’ data. They thus omitted approximately one-sixth of the total costs from the analysis; and • Business cases included the goods and services tax in the cash outflows, but did not show the input tax credit to be received from the Australian Taxation Office as a cash inflow; and • Overstated the effect that implementing e-procurement would have on financial operating performance of agencies in future years. The cases omitted significant relevant expenses, e.g. some agencies omitted hardware or software upgrades, software licences or ongoing training. The cases assumed that unrealised productivity improvements would be reflected in financial operating statements as a real fall in expenses. 3.12The above inadequacies are likely to mean that the calculations in the business cases resulted in an overstatement of the savings to be achieved from the introduction of e-procurement, and a material overestimation of the effect on the financial operating performance of agencies. 3.13Business cases are normally used to decide whether to proceed with implementing a particular initiative. However: • Several agencies had contracted with service providers to provide an e-procurement application or workflow solution and had proceeded with significant implementation of e-procurement before the business cases were prepared; and • Some agencies advised that they used the business cases to decide whether to take up funding available from the Department of Treasury and Finance. This confuses the investing and funding decisions, contrary to the policy of the Department of Treasury and Finance6. The business cases were, therefore, used for the wrong purpose. RESPONSE by Deputy Secretary, Department of Treasury and Finance Para. 3.4 The Auditor-General states that the progress of implementation of EC4P in departments is behind “expectations”. It does not appear that the Auditor-General’s statement is based on an assessment of each department’s progress against the timelines set in its implementation plans. Accordingly, the Department of Treasury and Finance believes the statement has not been substantiated. Paras 3.7 to 3.10 The EC4P business case templates were developed in consultation with the Budget and Financial Management Division of the Department and in accordance with all relevant Department of Treasury and Finance policy and guidelines. The report does not accurately describe the business case process. The stages were: • Whole-of-government business case – the purpose of this business case was to demonstrate the potential of the Project, to obtain approval to proceed to the next stage of assessment of its viability, and to obtain in-principle agreement that upfront funding would be made available as a loan to meet initial implementation costs. The focus at this stage was on demonstrating that the Project had a positive business case. The scale of the potential savings was canvassed (para. 3.7) but this was not relevant to the decision at this stage; • Preliminary business cases were developed by each department to test the viability of the Project in that department, and to obtain agreement to proceed to the next stage of assessment of the Project; and • Final business cases were developed by each department to establish accurate savings and costs figures so as to form the basis of each department’s decision to commit to proceed with implementation of EC4P. The final business cases were also used as the basis for the Treasurer’s approval of upfront funding where requested by a department. The key additional piece of information available for the final business case was the actual cost of workflow and catalogue solutions which were factored into the final business cases. Each of the final business cases was presented to the Minister for Finance for noting. Once all business cases had been noted, the Minister was presented with a status report on the outcomes of the EC4P Project. This report also included a summary of the final business cases. Para. 3.11 The Department of Treasury and Finance reviewed each departmental business case and sometimes challenged assumptions made by departments, or the inclusion of out of scope costs. Decisions on the contents of the final business case, the signing-off of the business case and commitment to proceed with implementation remained with the department. Any adjustments to business cases could only be made by the responsible department. Para. 3.12 The Department of Treasury and Finance believes that departments adopted a conservative approach to the key drivers of the business case, e.g. percentage of routine purchasing which would be processed under EC4P. The Department has not seen any analysis which supports the Auditor-General’s conclusion that it is “likely” that there was “an overstatement of the savings to be achieved”. RESPONSE by Secretary, Department of Education and Training The Department of Education and Training utilised the business case cost matrix spreadsheet as provided by the Whole of Government EC4P Project Team as a standard tool to be adopted by all participating departments. From the outset, the Department recognised that the direct cost-benefits suggested by the spreadsheet could not be realised through the implementation of the EC4P solution. The Department clearly indicated this in Section 2.6 of its Final Business Case (dated July 2002): “From the EC4P financial analysis, approximately half of the total gains identified are productivity gains for direct procurement activities. Under the Department of Education and Training structure, staff responsible for purchasing activities are also responsible for other corporate services and program activities. This model means there is limited capacity to identify positions dedicated solely to direct procurement activities. Consequently the realisation of these productivity gains through direct position shedding may not be viable and the productivity gains identified may not immediately translate into a direct cash benefit for the Department of Education and Training. Productivity gains are likely to be realised through targeting other value adding activities within the areas of corporate services and program area responsibility”. KEY FACTORS AFFECTING PROGRESS OF THE EC4P PROJECT 3.14 The factors that have impacted on the progress of the EC4P Project to date were also examined. As shown in Chart 2D, the EC4P Project had a complex structure comprising a Steering Committee, a central project team and 8 project teams in agencies. 3.15 The following assessment relates mainly to the whole-of-government component (i.e. the EC4P Project Steering Committee and the central project team based at the Department of Treasury and Finance) and the Project as a whole, rather than the projects and project teams within the individual agencies, i.e. the 8 budget sector departments and Victoria Police. 3.16 Assessment of the factors that affected the EC4P Project was made with reference to themes identified by the Organisation for Economic Co-operation and Development (OECD) for ensuring successful government IT projects7. In making the assessment, it is recognised that the EC4P Project was not only an IT project; it was also an exercise in change management. This is the case in most large IT projects, where success or failure depends not only on the adequacy of the technological solution but also on the level of engagement of users of the technology. In this case the users are public servants who purchase goods and services, and suppliers who make those goods and services available in the marketplace. The following discussion recognises the change management strategies used by the central project team. 3.17 The themes, and our comments about the EC4P Project are presented in Table 3D. Further comments about the user take-up and supplier engagement are made in Part 4 of this report. TABLE 3D     RESPONSE by Deputy Secretary, Department of Treasury and Finance Para. 3.17 The Department briefed the Minister fully on all EC4P matters, both verbally and in formal briefs. It was the responsibility of the Project Sponsor not the role of the EC4P Steering Committee to communicate directly to the Minister. This reporting line is shown in the project reporting structure in Chart 2D. RESPONSE by Secretary, Department of Primary Industries Para. 3.16 The Department of Primary Industries does not accept that the principles used by the performance audit, based on the OECD Policy Paper for IT projects, are those on which the project should be judged. The Department of Natural Resources and Environment’s vision was to reform business practices and implement systems to improve procurement, provide savings, improve compliance and meet business needs. This was seen as a procurement and change management project, not an IT project. The project successfully met milestones and achieved significant benefits. FURTHER AUDIT COMMENT As stated in para. 3.16, “the EC4P Project was not only an IT project, it was also an exercise in change management”. CONCLUSION AND RECOMMENDATIONS 3.18 Early public statements of the benefits from the introduction of e-procurement were overly-optimistic, materially overstated the likely cost savings and were based on inadequate business cases. In May 2000, the Treasurer and the Minister for Finance were advised by the Department of Treasury and Finance of savings in the vicinity of $150 million per annum. Subsequently, in July 2000, the Government announced potential savings of between $60 million and $240 million per annum, or at least $12 million per annum, from introducing e-procurement across the 8 departments and Victoria Police. No evidence has been located to substantiate these estimates. The only savings supported by documentation was the estimate of $65 million over 5 years identified in the problematic business case that accompanied the advice to the Ministers in May 2000. 3.19 Since that time, forecast cost savings have been significantly reduced, as have the number of agencies involved. The Project is now expected to break even after 4 years and to provide savings of $11 million per annum from the 5th year. 3.20 The 8 departments and Victoria Police each prepared an agency business case to support the introduction of e-procurement. However, the business cases, which used a format and methodology prepared for the central project team by a private consultant contained conceptual errors, treating productivity savings as if they were savings of cash and failing to explain the assumptions made. In our opinion, the business cases were likely to have materially overestimated the financial benefits to be achieved from e-procurement. 3.21 The Project, viewed in its entirety, did not adequately address many key themes for ensuring successful government IT projects, including the project was small enough to be managed, monitored and re-evaluated at key stages; using known technologies; ensuring compliance with best practice project management; managing knowledge prudently; and recruiting and retaining the staff necessary to deliver the Project. Recommendations 3.22 We recommend that the Department of Treasury and Finance ensure that future business cases: • adopt accepted business principles and comply with established guidelines; and • are completed at an early stage to enable them to be used to inform investment decisions. 3.23 We recommend that the Department of Treasury and Finance use best practice principles for large information technology projects in future and in its guidance. 3.24 We recommend that the Department of Treasury and Finance revisit the single supplier database to re-assess its relevance and whether it is likely to provide value-for-money. RESPONSE by Deputy Secretary, Department of Treasury and Finance Para. 3.22 The Department of Treasury and Finance notes both parts of the recommendation. The business cases developed as part of the EC4P Project adopted accepted business principles and complied with all relevant guidelines. The Department of Treasury and Finance agrees that this is essential for future projects. Departments used the final business cases as the basis for decisions on whether to proceed to implementation stage. Para. 3.23 The Department of Treasury and Finance notes the recommendation. The Department agrees that a strong project management methodology should be used for all projects. The Department’s central project team was trained in the Anassis project management methodology. The team used this training and the best practice principles therein to develop its Project Strategy Plan. The team used this methodology to clearly set out the Project’s scope, milestones and deliverables. A key element of the Plan was the development of a Communication Plan and Strategy which identified communication activities pertinent to each stakeholder and supported the dissemination of information and guidance material about EC4P through the purpose-built web-based information repository. Para. 3.24 The Department of Treasury and Finance does not agree with the recommendation. The report does not explain the basis for this recommendation. The Department of Treasury and Finance believes that the Supplier Database is integral to the flexibility of the EC4P model. The Supplier Database allows the government to retain core information about suppliers so that departments can more easily change to a new eCatalogue provider or providers if they wish to do so. 8 Victorian Auditor-General’s Office, derived from data presented by agencies in their final business cases of 2002. 9 The Australian Procurement and Construction Council’s Government Framework for National Cooperation on Electronic Procurement requires that a supplier of goods and services through an electronic catalogue should be able to register its details with one agency, making that information available to all government agencies in those jurisdictions. This is known as a “single sign-on”. 10 EC4P Status Report 23 March 2001, tabled at the meeting of the EC4P Steering Committee of 28 March 2001 and EC4P Status Report 12 April 2001, tabled at the meeting of the EC4P Steering Committee of 18 April 2001. 11 www.vgpb.vic.gov.au/polguid/polC9.htm accessed 11 September 2002. 1 Department of Treasury and Finance, Electronic Commerce for Procurement - Project Approval and Funding, Ministerial Brief to the Treasurer and the Minister for Finance, reference T05264, 25 May 2000. 2 Department of Treasury and Finance, Electronic Commerce for Procurement EC4P: Business Case. v. 4.3 (final), 17 May 2000. 3 Department of Treasury and Finance, The Investment Evaluation Policy and Guidelines, 1996. While these guidelines relate to infrastructure investments, i.e. investments in infrastructure and capital assets, the preface to the guidelines states that “most guidance however applies to equally to the appraisal of any expenditure decisions”. 4 Department of Treasury and Finance, Electronic Commerce For Procurement - Project Outcomes & Status, Ministerial Brief to Minister for Finance, reference B07844, 12 August 2002. 5 Productivity savings were mostly notional calculations of saved time. They would not result in a reduction in the work force, and would not result in a cash saving. Rather, productivity savings would free-up staff time. 6 Department of Treasury and Finance, The Investment Evaluation Policy and Guidelines, 1996. 7 OECD, The Hidden Threat to E-Government: Avoiding large government IT failures, OECD Public Management Policy Brief No. 8, March 2001. The themes in the policy brief are based on the experiences from participants and country reports presented at a meeting on 26-27 October 2000. Representatives of 17 countries met at OECD headquarters in Paris to share experiences on managing large public IT projects. The meeting helped to define the problems and find possible solutions. |