Home > Evaluation methodologies

During 2003—04, the APS Commission conducted several evaluations as part of its evaluation program including on agencies’ management of the Code of Conduct, agencies’ protocols supporting employees’ interactions with Ministers’ offices and agencies’ Workplace Diversity Programs (WDPs). Information on evaluation methodologies is provided below.

EVALUATION OF AGENCIES’ MANAGEMENT OF THE CODE OF CONDUCT

Six APS agencies with varying rates of investigations into suspected breaches of the Code of Conduct and varying use of ‘high impact’ sanctions were approached to participate in the evaluation.1 An SES employee from each agency was nominated to be a member of a reference group set up by the APS Commission to facilitate and assist the evaluation in their agencies.

The objectives for the evaluation were:

1. To develop guidance on ‘good practice’ for APS agencies on managing suspected breaches of the Code of Conduct.
2. To examine the procedures involved in selected agencies in the management of suspected breaches of the Code of Conduct, including:
   1. the methods used to facilitate and encourage staff reporting of suspected breaches and their effectiveness (including whistleblower procedures)
   2. an agency’s preparedness to pursue suspected breaches of the Code of Conduct and to follow through with formal investigations
   3. the appropriateness and consistency of processes used to investigate suspected breaches (including the factors taken into account to decide when and if formal processes are used)
   4. the appropriateness and consistency of decision-making processes relating to whether or not a breach has occurred
   5. the appropriateness and consistency of decision-making processes used to determine the sanctions imposed as a result of breaches
   6. the extent to which procedures comply with the legislation (including the provisions included in agency certified agreements).
3. To identify examples of good practice in the procedures in selected agencies in the management of suspected breaches of the Code of Conduct.
4. To identify any impediments to agencies in effectively managing suspected breaches of the Code of Conduct.

1 ‘High impact’ sanctions are those that have the greatest impact on employees: termination of employment, reduction in classification and reduction in salary.

Information on the management of suspected Code of Conduct breaches in the six agencies was gathered from a number of sources, including:

• analysis of each agency’s results from the 2003 State of the Service employee and agency surveys
• structured interviews with key personnel in each agency, including:
  • employees with policy responsibility for the management of the Code of Conduct within the agency and, for very large or decentralised agencies, people with responsibility for management of the Code of Conduct within a number of regions of the agency
  • employees who have conducted investigations into suspected breaches of the Code of Conduct
  • employees who have made determinations about whether an employee has breached the Code of Conduct
  • delegates within the agency who have imposed sanctions consequent upon a determination that an employee has breached the Code of Conduct
  • representatives of employees including union representatives and members of various consultative committees (where available)
  • staff counsellors or people providing advisory services to employees (where available)
• focus groups of employees–three groups were held in each of the very large agencies and one group in each of the other four agencies
• an analysis of paper-based documentation (misconduct procedures as required under s.15 of the PS Act, whistleblower procedures as required by Public Service Regulation 2.4, training materials, etc)
• an interview with Mr Steve Ramsey, Director, Legal and Technical, Community and Public Sector Union
• an audit of a selection of each agency’s misconduct case files from 2002—03.

EVALUATION OF AGENCIES’ PROTOCOLS SUPPORTING EMPLOYEES’ INTERACTIONS WITH MINISTERS’ OFFICES

The evaluation of agencies’ protocols supporting employees’ interactions with Ministers’ offices was based on both quantitative and qualitative analysis. Initial quantitative analysis involved regression analysis of data for 20 large agencies based on responses to the 2003 agency and employee surveys and documentation of policies and protocols supplied to the Commission by the agencies.

The 20 agencies were: ABS, ASIC, ATO, ATSIS, BOM, Centrelink, CSA, Customs, DAFF, Defence, DEH, DEST, DEWR, DFAT, DIMIA, DITR, DOTARS, DVA, FaCS and Health.

The Commission then conducted qualitative analysis based on:

• focus groups conducted with 47 senior APS employees, particularly at SES Band 1 and EL2 levels, across five agencies
• interviews with departmental staff with experience as Departmental Liaison Officers
• discussions with staff working in parliamentary and ministerial liaison roles in departments, PM&C and Finance
• comments and suggestions from the Parliamentary Network, a contact group established in 2004, comprising agency staff in parliamentary and ministerial liaison roles.

EVALUATION OF AGENCIES’ WORKPLACE DIVERSITY PROGRAMS

The evaluation of APS agencies’ WDPs was aimed at assessing how effectively the statutory requirements for WDPs, as set out in Chapter 3 of the Public Service Commissioner’s Directions 1999 (the Directions), have been incorporated into written programs.

All agencies were assessed in relation to the requirements to establish and review their WDP, and to provide a copy of the WDP to the PS Commissioner (clauses 3.3 to 3.6 of the Directions). However, the degree of specificity of Chapter 3 of the Directions was such as to make it impractical, given limited resources, to evaluate every APS agency against each aspect of the requirements. Therefore, only a sample of agencies were examined against the criteria relating to the measures required to be included in written WDPs (as outlined in clause 3.3 of the Directions).

For the purpose of the evaluation agencies were regarded as ‘within scope’ if they were included in the 2004 agency survey (see Appendix 1). The sample was selected randomly from a list of agencies compiled for the 2004 agency survey, for which the Commission had a current WDP. A higher proportion of large agencies were selected as they account for the majority of the APS workforce. The sample included ten large, six medium and eight small agencies (AGD, AIR, ANAO, APS Commission, ASIC, AWM, CGC, CrimTrac, CRS, Customs, BoM, DAFF, DEST, DIMIA, DOTARS, FaCS, Federal Court, Finance, GBRMPA, Health, ITSA, NMA, PC, Treasury).

Assessment of the WDP of each agency in the sample included cross-referencing to relevant data from APSED and, in the case of large agencies, from the 2004 employee survey. This data was used as a point of comparison between the rhetoric of the written WDP and the reality of the agency’s workforce.