PART 7

MEASURING THE EFFECTIVENESS OF MENTAL HEALTH SERVICES

BACKGROUND

7.1 Given the importance of mental health services to the community and the significant level of program expenditure, the Department of Human Services requires a broad set of key performance indicators (KPIs) to measure their performance.

7.2 Appropriate KPIs can benefit an organisation by:

• making performance more transparent, allowing assessment of whether program objectives are being met;

• helping clarify government objectives and responsibilities;

• informing the wider community about government agency performance;

• encouraging ongoing performance improvement; and

• encouraging efficient service delivery.

Characteristics of good indicators

7.3 Effective performance management and reporting should support achievement of agency objectives and government outcomes. For public sector agencies, this requires:

• Government outcomes to be clearly specified;

• Outcomes to be linked to departmental objectives. This requires that departmental objectives are clearly defined and cover the key dimensions of departmental accountabilities and operations, i.e. address both the “purchaser” interest in deliverables, and the “owner” interest in organisational capability and long-term sustainability;

• The budget and resource allocation process to be integrated with the performance management and reporting system; and

• Information systems that efficiently provide data necessary to measure, monitor and manage government programs.

7.4 Sound KPIs have the following attributes:

• Relevant. The indicator has a logical and consistent relationship to the agency’s objectives and is linked to the Government’s desired outcomes;

• Appropriate. The indicator gives sufficient information to assess the extent to which the agency has achieved a pre-determined target, goal or outcome, by reference to:

• the trend in performance over time;

• performance relative to the performance of other similar agencies; and

• performance relative to pre-determined benchmarks; and

• Fairly represent performance. The information must be capable of measurement, represent what it purports to indicate, and be accurate and auditable. Auditable means that quantifiable, consistent and verifiable data are available.

Departmental performance management and reporting framework

7.5 In the 2002-03 State Budget, $588.5 million was allocated to mental health outputs. This represents 7 per cent of total recurrent funding to the Department and 11 per cent of the Health budget. The major output targets for mental health services for 2002-03 include:

• 17 400 inpatient separations provided to people with a mental illness;

• 55 000 continuing clients in clinical community care;

• 9 124 clients receiving Psychiatric Disability Support Services; and

• 19 521 training hours of clinical staff¹.

7.6 In Victoria, the performance management and reporting framework shows output measures linked to outputs funded through the annual Budget process. These output measures, which are expressed in terms of time, quantity and cost, are published in the Budget Papers. Agencies are required to:

• account to the Treasurer quarterly against these targets in order to receive the moneys appropriated; and

• report their achievements against their output targets in their annual reports to Parliament.

7.7 In addition to output measures established under the performance management and reporting framework, performance measures may be established to provide management with the information it needs to make appropriate decisions linked to the organisation’s day-to-day activities. Timely acquisition of relevant, reliable information is the key to determining whether the direction set is being followed, whether the health system is achieving the intended results and desired outcomes, and whether resources are being allocated appropriately.

Mental Health Outcomes Project

7.8 Following significant changes to public mental health services in the 1990s, in 1997 the Department commenced a study to develop outcome measures. In February 1999, the “Mental Health Outcomes Project” released a discussion paper² proposing a suite of measures for Adult Mental Health Services.

7.9 The indicators proposed for reporting consumer outcomes included:

• health status indicators, such as severity scales (HoNOS³ scale and a self-rating tool⁴);

• change in level of disability (life skills profile⁵); and

• aim of treatment intervention (focus of care rating⁶).

7.10 The indicators proposed for reporting service outcomes included:

• KPIs derived from existing minimum datasets (these were unspecified);

• services’ responsiveness to client need (listed as “under development”);

• service integration (also listed as “under development”); and

• consumer and carer satisfaction.

7.11 The consumer and service outcome indicators were piloted by 4 Area Mental Health Services (AMHS) in 2000. The consumer outcome indicators are currently being “rolled-out” Statewide.

7.12 Work on the service outcome indicators was delayed until October 2001 when the Department engaged a consultant to revise the current set of KPIs. The development work is designed to:

• focus on outputs of mental health care;

• inform judgements about agency performance; and

• contribute to an understanding of population needs.

7.13 The project will produce a set of KPIs for use in child and adolescent mental health services, adult and aged persons services. The project is expected to conclude in September 2003.

ASSESSMENT OF MENTAL HEALTH KEY PERFORMANCE INDICATORS

Current mental health indicators

7.14 In 2001-02, the Department published 2 sets of performance measures for mental health outputs or deliverables. The first set appeared in the Department’s 2000-01 Annual Report and was replaced by a second set which appeared in the 2002-03 Policy and Funding Guidelines⁷. These measures are a mix of performance indicators (which focus of objectives) and performance measures (which relate to outputs) and are shown in Table 7A.

Table 7A
Major outputs/deliverables – Mental Health Services,
Department’s Annual Report 2000-01, and 2002-03 Policy and fUnding guidelines

Performance indicator or measure	Unit of measure	2000-01 DHS Annual Report	2002-03 Policy and Funding Guidelines (a)	Whether performance indicator or measure
Acute and sub-acute services
Quantity -
Acute inpatient treatment capacity	Beds			PM
Acute inpatient separations	Number			PM
Sub-acute treatment capacity	Beds			PM
Registered clients	Number			PM
Registered clients	Contacts			PM
Quality -
Inpatients re-admitted within 28 days (unplanned)	Per cent			PI
Providers participating in Quality Incentives Strategy Projects	Per cent			PM
Community care and support
Quantity -
Residential rehabilitation	Clients			PM
Home-based outreach support	Clients			PM
Continuing clients	Clients			PM
Quality -
Agency reporting on implementation and review of Individual Program Plans	Per cent			PM
Clinical inpatient clients who have contact with clinical community care service providers during the 7 days prior to admission	Per cent			PI
Clinical inpatient clients who have contact with clinical community care service providers within 7 days of post-discharge	Per cent			PI
Supported residential care
Quantity -
Psychogeriatric supported residential care capacity	Beds			PM
Prevention and promotion
Quantity -
Mental Health Week events	Number			PM
Primary Mental Health Response contacts	Number			PM
Training, research and development
Quantity -
Mental health academic positions sponsored	Number			PM
Postgraduate nursing placements (mental health)	Number of positions			PM
Psychiatric Disability Support Services
Quantity -
Clients receiving Psychiatric Disability Support Services	Number			PM
Quality -
Individual Program Plans completed within 2 months	Per cent			PI
Mental Health Service Systems Capacity Development
Quantity -
Number of clinical staff training hours	Number			PM
Quality -
Clinical staff successfully completing courses	Per cent			PM

(a) Also in 2002-03 Budget Paper No.3.

7.15 The Department has undertaken extensive consultation in developing the current set of indicators and measures. It has also replaced the former Psychiatric Records Information Systems Manager (PRISM) data system with the Redevelopment of Acute and Psychiatric Information Directions (RAPID) system that collects a broader range of data. However, comparative data are not regularly published so Area Mental Health Services (AMHSs) cannot compare their performance.

Audit assessment

7.16 Performance indicators related to departmental objectives should address efficiency, effectiveness and the causal impact of programs, and can apply to the medium or long-term. However, output performance measures are directly related to the purchasing function and the annual resource allocation model. Because of this, there are fundamental differences in the attributes of a good performance indicator and a good performance measure, and the criteria for assessment of each will differ⁸.

7.17 In particular, the relevant and appropriate criteria (shown in paragraph 7.4) are more difficult to assess in relation to performance measures. Table 7B sets out an assessment of each indicator based on these criteria. Of the 4 performance indicators, 3 meet all the criteria; the other one does not meet the fairly represents criterion, as there are concerns regarding the accuracy of the data required to calculate this indicator.

Table 7b
audit assessment of mental health services,
2002-03 performance indicators and measures

Performance indicator or measure			Relevant	Appropriate	Fairly represents
Quantity -
Acute inpatient separations
Continuing clients in community care
Clients receiving Psychiatric Disability Support Services
Number of clinical staff training hours			No	No	Unknown
Quality -
Inpatients re-admitted within 28 days (unplanned)
Clinical inpatient clients who have contact with clinical community care service providers during the 7 days prior to admission
Clinical inpatient clients who have contact with clinical community care service providers within 7 days of post-discharge
Clinical staff successfully completing courses					Unknown
Timeliness -
Individual Service Plans completed within 2 months					No
Legend		Yes
		To some extent, some attributes not addressed.

7.18 The Department’s set of mental health KPIs and performance measures are not sufficiently comprehensive to provide management with the necessary information to measure the effectiveness of mental health services. In particular, they do not provide any indicators of service demand. In the absence of a broader set of indicators, many AMHSs have developed their own management reports. While the 4 performance indicators currently used by the Department should be retained, improved performance information could enhance the management effectiveness of the sector.

7.19 We note that 4 AMHSs have been implementing clinical outcome measurement since July 2000 in a pilot project supported by the Commonwealth Government. Such measures will be valuable when eventually rolled-out to all AMHSs.

Potential indicators

7.20 Following our assessment of the performance indicators and performance measures in place, we have developed some suggestions that could enhance these indicators and measures.

7.21 We first examined the client pathway as described in Part 3 of this report (refer Chart 3B). We believe that each major phase of the client service model is a key component of the mental health system, and requires measures and indicators so its performance can be judged.

7.22 We then assessed each potential indicator based on our criteria, namely, indicators need to be relevant, appropriate and fairly represent performance. The relevance criterion is based on linking the indicator to one of the Department’s objectives. These objectives are:

• waiting times for health, community care, disability and housing programs are at, or below, national benchmark levels;

• quality of human services improves each year;

• sustainable, well managed and efficient government and non-government services;

• reduce social dislocation and the need for secondary and tertiary service intervention through strengthening the communities, family support, early intervention and health promotion measures;

• increase the proportion of people needing the Department’s funded services who remain in supportive families and communities; and

• reduce inequalities in health status and wellbeing, and in access to services⁹.

7.23 In suggesting indicators, we have done so with 2 audiences in mind:

• External. These indicators should inform strategic decision-making at the most senior level and be published to promote public accountability; and

• Internal. These indicators are at the operational level. They need to reflect how services are being implemented or managed, and highlight any problem areas that need attention.

7.24 Table 7C shows a proposed set of KPIs for mental health services. Table 7D presents a proposed set of performance measures. In both cases, we have identified:

• the type of measure/indicator (what should be regarded as “success”);

• the performance measure/indicator;

• what the measure/indicator should be compared against;

• a suggested or potential data source on which the measure/indicator can be calculated;

• how frequently the measures/indicators should be reported; and

• to whom the measure/indicator should be reported.

Table 7C
possible KPIs for Mental Health Services

(a) Indicates is currently a Department of Human Services performance measure or indicator.

Table 7D
possible performance measures for Mental Health Services

CONCLUSIONS

7.25 The current set of mental health measures and KPIs do not provide sufficient information to management and the Victorian Government to appropriately monitor the effectiveness of the services being delivered. Most of the current measures and KPIs are not tied to departmental objectives and relate to service delivery (i.e. outputs) rather than consumer outcomes. The current set of measures and indicators is also limited in its coverage of mental health services.

7.26 We have proposed a set of measures and indicators that we believe would both inform strategic decision-making and assist line managers administer their programs. Most of the information required to calculate the measures and indicators is available from current systems. However, there may need to be some revisions to the Department’s RAPID data system and an effort made to ensure data completeness.

RECOMMENDATIONS

7.27 We recommend that, as part of the process of development and revision of its performance measures and KPIs, the Department should:

• consider the measures and indicators proposed by this audit;

• continue to consult with service providers on their appropriateness;

• continue to develop information systems and reporting mechanisms to support decision-making;

• provide training to senior managers to interpret measures and KPIs;

• publish a comparative set of area-level measures and KPIs, at least annually; and

• analyse and review the measures and KPIs every 12 months to ensure their ongoing relevance.

RESPONSE provided by Department of Human Services

The performance measures and key performance indicators are deliberately output focussed as they are used for aggregate reporting to government on output performance they are not intended to be used for service management or outcomes measurement purposes

Development of comprehensive key performance indicators has been funded by the Commonwealth under the Information Development Plan. They will be completed by June 2003. Outcome measurement will also assist in measuring effectiveness from a consumer and service perspective. The Department of Human Services will be pleased to consider audit’s proposal for key performance indicators as part of this process.

Para. 7.27

The Department is putting considerable effort into the further development of performance and outcomes measures, including:

• Extension of clinical outcome measurement to all relevant Victorian mental health services. Outcome measurement has been used in a small number of services over recent years. In 2002-03, all remaining clinical services will receive assistance and training to support them in implementing routine outcome measurement based on the use of established tools;

• Establishing mechanisms for regular consultation with service managers and senior clinicians;

• Setting key performance indicators and benchmarks for service delivery, monitor performance of services against those benchmarks, and make the results publicly available;

• Refining current performance measures and the Quality Incentive Strategy; and

• Streamlining data collection and improve information systems to support service planning, delivery and monitoring.

In this context, the Department will consider the measures proposed by audit.

RESPONSE provided by St Vincent’s Mental Health Service

Para. 7.26

The proposed key performance indicators appear reasonable. More comparative data between health services would be helpful, as long as it was accurate.

The performance of RAPID will need to be improved in order to facilitate the extraction of this data. Currently, extracting meaningful reports from RAPID is difficult and time-consuming. There are also discrepancies between data extracted locally and data extracted by the Department, which creates problems for service planning and performance management.

¹ Government of Victoria, 2002-03 Budget, Budget Paper No. 3, Statement No. 2.

² Department of Human Services, Aged Community and Mental Health Division, Health Status and Outcomes in Victoria’s Mental Health Services, February 1999.

³ Health of the Nation Outcome Scales (HoNOS) measure the health and social functioning of people with mental illness. The 12-item scale measures behaviour, impairment, symptoms and social functioning, and can be determined by a health professional following routine clinical assessments.

⁴ Two self-rating mental health indicators are suggested.

⁵ The Life Skills Profile (LSP) is a 39-item scale that was developed as a measure assessing function and disability in persons with schizophrenia in the community.

⁶ Focus of care is a measure that best describes the consumer’s primary goal of care. The result is one of: Acute (short-term reduction in symptoms), Functional gain (improvement in personal, social or occupational functioning), intensive extended (minimisation of further deterioration) or maintenance (of current status).

⁷ DHS 2002, Victoria – Public Hospitals and Mental Health Services, Policy and Funding Guidelines, 2002-03.

⁸ Victorian Auditor-General’s Office, Departmental performance management and reporting, November 2001, (p. 72).

⁹ Department of Human Services 2002, Departmental Plan, 2002-03, Output Statement, Mental Health.