Appendix 2
Agency survey methodology
The scope of the agency survey was the 89 APS agencies, or semi-autonomous parts of agencies, employing 20 or more staff under the PS Act. The agencies were sent the survey in May 2003 for completion. Agencies were given six weeks to complete and submit their response. As part of their survey return, agency heads were required to ‘sign off’ their agency’s response. All 89 agencies responded to the online agency survey by the due date. This represents a 100% response rate.
This year the agency survey was an online survey rather than a paper survey. The survey design reflects the decision to develop a longitudinal data set to facilitate trend analysis of certain issues.
To assist with analysis of data from the agency survey in this year’s report, agencies have been grouped according to size. Of the 89 responding agencies, 22 were classified as large (>1000 APS employees), 26 as medium (251–1000 APS employees) and 41 as small (20–250 APS employees). These size categories are generally consistent with those used by the ANAO in Report No. 49 of 1998–99, Staffing reductions in the Australian Public Service.
Employee survey methodology
The employee survey was designed to establish the views of APS employees on a range of issues, including work-life balance, the APS Values, the Code of Conduct, job satisfaction, and learning and development. The results of the employee survey are one of the main sources of information on which the APS Commission has drawn during the preparation of this report.
The employee survey was also designed to complement the agency survey. The results of the employee survey were, in part, intended to act as a ‘reality check’ in analysing responses to the agency survey. To achieve this objective, similar questions were asked in both surveys on a range of topics. Additional questions, suitable for employees but not for agencies (such as on job satisfaction), were also included in the employee survey.
Scope and coverage
The scope of the survey was all APS employees (both ongoing and non-ongoing) in agencies with more than 100 APS employees. Employees in agencies that employed 100 or fewer APS employees were excluded on the basis that their responses could possibly identify them.
At the time the survey sample was selected, the most up-to-date data (contained in the Statistical Bulletin 2001–02) indicated that the total number of APS employees at 30 June 2002 was 123,494. The survey sample was selected from the total population of APS employees from agencies with more than 100 APS employees, which at 30 June 2002, was 122,529.
Stratification
A stratified random sample of 6279 APS employees was selected using two databases maintained by the APS Commission—the APS Employment Database (APSED) and the Senior Executive Management System (SEMS). The sample was stratified by:
- level (APS, EL and SES classification groups)
- agency size (small 100–250 APS employees; medium 251–1000 APS employees; large >1000 APS employees)
- agency (only for the 22 large agencies)
- location (ACT and non-ACT).
To enable sound statistical inferences to be made about all APS employees, the individuals were randomly selected from each of the strata. Each individual within a stratum had an equal chance of selection.
The sampling rates varied between the strata to ensure that sufficient statistical accuracy would be achieved for survey estimates from APS employees with the key characteristics captured by the stratification variables (level, location, agency and agency size). To gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate was required than for a larger population (such as APS-level employees).
The accuracy requirements varied between the demographic variables listed above, and this also led to differing sampling rates for these demographic variables.
This stratification process has not introduced a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account. See the section ‘Weighting and estimation’ for further details.
Reporting of large agency results
The survey was designed to allow the APS Commission to provide the large agencies that want them, a copy of their own employee survey results for internal management purposes, subject to the results satisfying a statistical accuracy benchmark. To enable this, the 22 agencies with more than 1000 APS employees were included separately in the stratification process (see the section ‘Stratification’ above).
Where relevant, the State of the Service report includes agency level results of large agencies that met the minimum number of weighted responses (see the section ‘Measures of accuracy and error’ below).
Privacy, anonymity and confidentiality
Maintaining confidentiality throughout the entire employee survey process was a primary concern to the APS Commission.
Privacy arrangements for APSED preclude APS Commission staff, other than those in the APSED team and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of those individuals selected in the non-SES sample from the APSED were not known to the State of the Service team, ORIMA Research staff or any other non-APSED staff involved in the survey.
Privacy arrangements for the SEMS database, however, do allow for limited access within the APS Commission on a project-by-project basis. The identity of individuals selected in the SES sample were known to the State of the Service team and relevant ORIMA Research staff. However, all responses to the employee survey were anonymous so individuals could not be identified. Respondents were not asked to provide names, positions or workplace location (other than postcode) when responding to the survey.
The need to maintain respondent anonymity affected survey design and the ability to conduct targeted follow-up.
Survey design
Initial input into the draft employee survey was sought from several central agencies, including DEWR and the NAA. The draft survey was then subjected to cognitive testing involving individuals at the APS, EL and SES levels from AQIS, Defence, DEWR, Health and the APS Commission. A further refined draft survey was then tested at a focus group session attended by individuals at the APS, EL and SES levels from Centrelink, Comcare, Customs, DIMIA, NOHSC and the APS Commission.
The majority of questions, 78 of 90, were asked of all respondents. Two questions were asked of SES-level employees only. Five questions were asked of EL and SES employees only. Five questions were asked of APS and EL-level employees but not SES.
The questions asked of employees were not dependent on location (inside or outside the ACT) or agency size (small, medium or large).
The employee survey was delivered using two methods. The main delivery method was online via a secure internet site. Employees working in agencies with routinely available desktop internet access were sent a letter from the Commissioner inviting them to participate in the online survey.
The secondary, paper-based, delivery method was developed and implemented for employees working in agencies that do not routinely have access to the internet at their workplace desktops. A list of agencies without routinely available desktop internet access is at Appendix 3. Employees selected in the random sample from these agencies received a letter from the Commissioner inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.
The 6649 invitation letters were mailed out to respondents’ workplace addresses between 22 and 27 May 2003.1 Respondents were asked to complete the survey and return it by Wednesday, 18 June 2003.
An adjustment was made to the final sample size to account for separations from the APS that were estimated to have occurred between 30 June 2002 and the time the sample was drawn (early May 2003).2 Accordingly, the final sample size was reduced by 370 to 6279.
Weighting and estimation
The survey responses were re-weighted to reflect the characteristics of the underlying population of APS employees. This was done to ensure that the overall demographic characteristics of the survey results exactly matched the demographic characteristics of all APS employees. The re-weighting process was based on five main demographic characteristics:
- level (APS, EL and SES classification group)
- location (ACT and non-ACT)
- agency size (small 100–250 APS employees; medium 251–1000 APS employees; large >1000 APS employees)
- agency (only for the 22 large agencies)
- sex.
There were therefore 288 different weights applied—level (3) multiplied by location (2) multiplied by agency size and agency (24) multiplied by sex (2). For this survey, the weights were calculated by dividing the populations of each stratum by the number of respondents to the survey in each stratum. For example, if there are 4000 EL women at medium-sized agencies in the ACT and 200 responded, the weight assigned to each ACT EL woman working in a medium-sized agency is 20. If the data were not re-weighted, some strata could be over-represented and others under-represented in the total survey results.
The weighting approach adopted assumes that respondents respond in the same way as non-respondents for the characteristics of interest. Applying the weighting method above, the responding persons are used to represent the non-responding persons. However, the re-weighting by sex has the effect of rectifying to some degree any potential non-respondent bias arising from differences in the sex of respondents.
In this survey, with anonymous responses, it was not possible to identify the non-respondents. This severely limited our ability to follow-up non-respondents. However, there would need to be a marked difference in the views of non-respondents from those of the respondents to alter or bias the overall results to any significant extent. For analysis presented in this report it was assumed that there was no significant bias between those who responded in the survey and those who did not respond.
The results are calculated under the assumption that responding persons answer in the same way as nonrespondents. This should be considered when using the data to make inferences about the APS population.
Please note that results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.
Measures of error and accuracy
Two types of error can occur in sample surveys: sampling error and non-sampling error. Sampling error arises because in a sample survey not all of the population are surveyed. Hence a measured sample statistic is not usually identical with the true population behaviour. Non-sampling errors cause bias in statistical results and can occur at any stage of a survey and can also occur with censuses (i.e. when every member of the target population is included). Sampling error can be estimated mathematically whereas estimating non-sampling error can be difficult. It is important to be aware of these errors, in particular non-sampling error, so they can be either minimised or eliminated from the survey.
Non-sampling error
The survey received a response rate of 50.7%. This response rate excludes responses that were received but were insufficiently complete to provide input into the data generated.
This response rate is very creditable for a voluntary survey.
Other non-sampling errors can result from imperfections in reporting by respondents, errors made in recording and coding of responses, and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors. However, every effort was made to reduce the non-sampling errors to a minimum by careful survey design and efficient operating procedures. In particular, the online survey design minimised the possibility of errors made in recording and coding of responses, as the respondents themselves entered the data when responding to the survey.
In addition, identifiable errors made by respondents while completing the survey were removed from the results database. For example, responses made by APS-level employees to an EL-only question have been removed to ensure the integrity of the data. Blank responses were generally coded to non-response categories. The exception to this practice was where responses were needed for demographic items for weighting purposes. In instances where this occurred, survey responses were disregarded.
Sampling error
One measure of the sampling error of an estimate is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% confidence interval.
For instance, we are 95% confident that the estimate of the population that would be satisfied that their supervisor would support the use of flexible work practices is between 81.8% and 84.4% (an estimate of 83.1% and a standard error of 1.3%).
The following table illustrates the standard errors from the sample design associated with estimates from 11 key questions in the employee survey. Generally, the higher the sample size for a question, the lower the standard error. For example, questions following a ‘filter’ question are more likely to have a slightly higher standard error because the population size responding to that question is lower than for ‘non-filtered’ questions. The standard error for the performance pay question estimate, for example, is slightly higher than for many others because only employees with access to performance-related pay were asked the question, thereby reducing the sample size.
| Question | 95% confidence interval | Estimate |
|---|---|---|
| Satisfied that supervisor would support use of flexible work practices | +/-1.3% | 83.1% |
| Consider that merit is consistently applied in engagement/promotion/ transfer decisions resulting from a competitive selection process | +/-1.7% | 59.0% |
| Familiar with the APS Values | +/-1.5% | 76.9% |
| Familiar with the APS Code of Conduct | +/-1.5% | 74.4% |
| Faced an ethical challenge in dealing with Ministers and/or Ministers’ offices | +/-2.6% | 34.7% |
| Directly involved in delivering services to the public or managing employees delivering such services | +/-1.8% | 60.5% |
| Subjected to harassment, discrimination or bullying in the workplace | +/-1.4% | 18.1% |
| Received formal individual performance feedback in previous 12 months | +/-1.5% | 78.5% |
| Agree that performance pay system operates fairly and consistently | +/-2.2% | 40.0% |
| Consider that an employee in work area consistently underperformed in 2002–03 | +/-1.8% | 50.2% |
| Consider that immediate supervisor is highly or very highly effective at managing people | +/-1.8% | 46.8% |
Please note results for questions where the number of weighted responses is fewer than 20 have not been reported. This approach has been adopted for two reasons: firstly, to eliminate the possible identification of individuals who responded to these questions; secondly, to remove less reliable results from the analysis.
This approach has not affected reporting of results at the aggregate level; however, it has limited our ability to report on disaggregated data where the sample size is small—as is sometimes the case for questions following ‘filter’ questions.
Interpretation of scales
Scales were included in any question that required a respondent to measure the strength or level of a theoretical construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.
The scales used in the surveys were generally balanced—that is, they allowed the respondents to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.
When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent to ‘very satisfied’ rather than ‘satisfied’.
Summary indexes
Summary indexes have been used to assist analysis of results of a number of survey questions comprising several parts. The indexes operate to condense a multiple response question into a single index for comparative purposes. For example, in exploring respondents’ overall level of job satisfaction, a question comprising 15 factors was summarised into a single index using a point scoring system. In this way, analysis of the 15 job satisfaction factors can be supplemented by analysis at the summary level.
1 Because of their unique operating environment involving a high proportion of APS employees posted throughout the world, all DFAT employees included in the sample were emailed the invitation letter. This was done to prevent paper copy letters arriving at employees’ workplace addresses after the close-off date.
2 The reduction in the sample was based on APSED’s annual separation rates
calculated over the previous two-year period.
The annual rate was multiplied by 0.8 to account for the ten-month period between 30 June
2002 and when the sample was drawn (early May 2003).