print this page
© Copyright Commonwealth of Australia - Australian Public Service Commission
Home page
> State of the Service reports
> State of the Service Report 2005-06 > Appendixes > State of the Service survey methodologies > Next: Evaluation methodology
‹ Previous page
Last updated: 30 November 2006
Appendix 2: State of the Service survey methodologies
Abbreviations
A list of the abbreviations used in this report is available in the Glossary
Agency survey methodology
The scope of the agency survey was the 84 APS agencies, or semi-autonomous parts of agencies, employing at least 20 staff under the Public Service Act 1999.
The 84 participating agencies were sent the online survey on 7 June 2006 for completion. Agencies were given six weeks to complete and submit their response. As part of their survey return, agency heads were required to ‘sign off ’ their agency’s response. All 84 agencies responded to the online agency survey. The results of the agency survey are one of the key sources of information which the Commission has relied on throughout the preparation of this report.
Employee survey methodology
The employee survey was designed to establish the views of APS employees on a range of issues, including work-life balance, leadership, working with external stakeholders, job satisfaction, and learning and development. A particular focus of this year’s survey was employee engagement issues. The results of the employee survey are one of the main sources of information on which the Commission has drawn during the preparation of this report.
The employee survey was also designed to complement the agency survey. The results of the employee survey were, in part, intended to act as a ‘reality check’ in analysing responses to the agency survey. To achieve this objective, similar questions were asked in both surveys on a range of topics. Additional questions, suitable for employees but not for agencies (such as those on job satisfaction and increasing individual productivity), were also included in the employee survey.
Scope and coverage
The scope of the employee survey was all APS employees (both ongoing and non-ongoing) in agencies with at least 100 APS employees. Employees in agencies that employed fewer than 100 APS employees were excluded on the basis that when tabulated their responses could possibly identify them.
The survey sample was drawn from APSED on 10 April 2006, at which time APSED indicated that the total number of APS employees was 141,796. The survey sample was selected from the total population of APS employees from agencies with at least 100 APS employees, which was 140,777. Appendix 1 provides information on agencies’ APS employee numbers as at 10 April 2006.
Stratification
A stratified random sample of 6552 APS employees was selected from APSED. The sample was stratified by:
- level (APS 1–6, EL and SES classification groups)
- agency size (small: 100–250 APS employees; medium: 251–1000 APS employees; and large: >1000 APS employees)
- agency (for the 23 large agencies, the three medium portfolio departments and the Commission)
- location (ACT and non-ACT).
To enable sound statistical inferences to be made about all APS employees, individuals were randomly selected from each of the strata. Each individual within a stratum had an equal chance of selection.
The sampling rates varied between the strata to ensure that sufficient statistical accuracy would be achieved for survey estimates from APS employees with the key characteristics captured by the stratification variables (level, location, agency and agency size). To gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate was required than for a larger population (such as APS 1–6 employees).
The accuracy requirements varied between the demographic variables listed above, and this also led to differing sampling rates for these demographic variables.
This stratification process has not introduced a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account (see the section ‘Weighting and estimation’ below for further details).
Reporting of large agency results
The survey was designed to enable the 23 large agencies, the three medium portfolio departments and the Commission to receive a copy of their own results from the employee survey for internal management purposes—subject to the results satisfying a statistical accuracy benchmark. For this to occur, these 27 agencies were included separately in the stratification process (see the section ‘Stratification’ above).
Where relevant, and to maintain consistency with previous years, the State of the Service report includes only agency level results of large agencies that met the minimum number of weighted responses (see the section ‘Measures of error and accuracy’ below). The medium portfolio departments and the Commission are not included in any agency level analysis in the report.
Privacy, anonymity and confidentiality
Maintaining confidentiality throughout the entire employee survey process was a primary concern to the Commission.
Privacy arrangements for APSED preclude Commission staff, other than those in the APSED Team, the Group Manager of the Evaluation Group, and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of those individuals selected in the sample from APSED was not available to the Commission’s State of the Service Team or any other non-APSED staff involved in the survey. A small number of ORIMA Research staff had access to the sample. All responses to the survey were anonymous so individuals could not be identified.
Each person invited to participate in the employee survey was provided with a unique password. This prevented multiple responses from individual respondents.
Survey design
The employee surveys conducted in previous years were used as the basis for this year’s survey. Some questions have been included on an annual basis, other questions have been cycled through on a two- or three-year basis, and others were included for the first time this year to address topical issues. To ensure the Commission maintains comparable time series data, any changes to questions repeated from previous years were kept to a minimum.
The draft employee survey was subjected to individual and paired cognitive testing involving individuals at the APS 1–6 and EL classifications from DAFF, NAA, DCITA, DEWR, ATO, Centrelink and Health.
The majority of questions, 73 of 83, were asked of all respondents. Three questions were asked of SES employees only. Five questions were asked of EL and SES employees only. Two questions were asked of APS 1–6 and EL employees but not of SES employees.
The employee survey was delivered using two methods. The main delivery method was online via a password-protected Internet site. The majority of the sample was sent an email from ORIMA Research on behalf of the Commissioner inviting them to participate in the online survey.
A secondary, paper-based delivery method was developed and implemented for employees working in agencies that do not have access to an individual email account or do not have (or have only limited) access to the Internet. These employees received a letter from the Commissioner inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.
The 6552 invitation emails and letters were sent out to the sample between 15 May and 19 May 2006. Respondents were asked to complete the survey and submit or return it to ORIMA Research by Friday 9 June 2006.
An adjustment was made to the final sample size to account for those out of scope of the survey (including repeatedly bounced emails, those ‘out of office’ for the entire survey period and those known to be no longer employed in the APS at the time of the survey).1 The final sample was reduced by 386 to 6166.
Weighting and estimation
The survey responses were re-weighted to reflect the characteristics of the underlying population of APS employees. This was done to ensure that the overall demographic characteristics (used for sample selection) of the survey results exactly matched the demographic characteristics of all APS employees.
The re-weighting process was based on the four demographic characteristics used for selection of the sample, namely:
- level (APS 1–6, EL and SES classification groups)
- agency size (small: 100–250 APS employees; medium: 251–1000 APS employees; and large: >1000 APS employees)
- agency (for the 23 large agencies, the three medium portfolio departments and the Commission)
- location (ACT and non-ACT).
There were, therefore, 174 different weights applied—level (3) multiplied by location (2) multiplied by agency size and agency (29). For this survey, the weights were calculated by dividing the populations of each stratum by the number of respondents to the survey in each stratum; for example, if there are 4000ELs in medium agencies in the ACT, and 200 responded, the weight assigned to each EL working in a medium agency in the ACT is 20. If the data were not re-weighted, some strata could be over-represented and others under-represented in the total survey results.
The weighting approach is based on that taken in previous years. The application of a uniform approach to sample selection and weighting will assist in the development of time series data.
The weighting approach adopted assumes that respondents respond in the same way as non-respondents for the characteristics of interest. The weighting method above assumes that the responding persons represent the non-responding persons.
In this survey, with a response rate of 64%, there would need to be a marked difference in the views of non-respondents from those of the respondents to alter or bias the overall results to any significant extent. For analysis presented in this report it was assumed that there was no significant bias between those who responded in the survey and those who did not respond.
The results are calculated under the assumption that responding persons answer in the same way as non-respondents. This should be considered when using the data to make inferences about the APS population.
Results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.
Measures of error and accuracy
Two types of error can occur in sample surveys: sampling error and non-sampling error. Sampling error arises because in a sample survey not all of the population are surveyed. Hence a measured sample statistic is not usually identical with the true population behaviour. Non-sampling errors cause bias in statistical results and can occur at any stage of a survey and can also occur with censuses (i.e. when every member of the target population is included). Sampling error can be estimated mathematically whereas estimating non-sampling error can be difficult. It is important to be aware of these errors, in particular non-sampling error, so that they can be either minimised or eliminated from the survey.
Non-sampling error
The survey received a response rate of 64%. This response rate excludes responses that were received but were insufficiently complete to provide input into the data generated. This response rate is very creditable for a voluntary survey.
Non-sampling errors can result from imperfections in reporting by respondents, errors made in recording and coding of responses, and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors. However, every effort was made to reduce the non-sampling errors to a minimum by careful survey design and efficient operating procedures. In particular, the online survey design minimised the possibility of errors made in recording and coding of responses, as the respondents themselves entered the data when responding to the survey.
In addition, identifiable errors made by respondents while completing the survey were removed from the results database; for example, responses made by APS 1–6 employees to an EL-only question have been removed to ensure the integrity of the data. Blank responses were generally coded to non-response categories. The exception to this practice arose where responses were needed for demographic items for weighting purposes. In instances where this occurred, survey responses were disregarded.
Sampling error
One measure of the sampling error of an estimate is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% Confidence Interval.
For instance, we are 95% confident that the estimate of the population who agree that their manager provides the support they need to do their job is between 73.7% and 76.7% (an estimate of 75.2% and a confidence interval of +/-1.5% based on a standard error of 0.75%).
The following table illustrates the standard errors from the sample design associated with estimates from 12 key questions in the employee survey. Generally, the higher the sample size for a question, the lower the standard error; for example, questions following a ‘filter’ question are more likely to have a slightly higher standard error because the population size responding to that question is lower than for ‘non-filtered’ questions. The standard error for the whole of government example question estimate, for example, is slightly higher than for many others because only EL and SES employees involved in multi-agency forums or structures were asked the question, thereby reducing the sample size.
| Question | 95% confidence interval | Estimate result |
|---|---|---|
| Understand how their agency’s decision-making processes operate (e.g. relevant committee structures and how committees are linked) | ±1.6% | 64.0% |
| Agree that their manager provides the support they need to do their job | ±1.5% | 75.2% |
| Agree that in their agency, the leadership is of the highest quality | ±1.6% | 37.7% |
| Agree that their agency encourages the public to participate in shaping and administering public policy | ±1.7% | 47.2% |
| Satisfied with the overall say they have in decisions that impact on their work | ±1.7% | 48.1% |
| Considering their work and life priorities, they are satisfied with the work/life balance in their current job | ±1.6% | 67.6% |
| Agree that in multi-agency forums in the last 12 months, participants were primarily focused on solving whole of government problems | ±3.5% | 54.0% |
| Agree that merit is routinely applied in engagements and promotions resulting from a competitive selection process | ±1.6% | 53.7% |
| Agree that agency actively encourages recruitment and employment of people from all cultural backgrounds | ±1.4% | 81.5% |
| Satisfied with own access to learning and development opportunities in their organisation | ±1.7% | 60.9% |
| Satisfied with own access to leadership development opportunities in their organisation | ±1.6% | 38.8% |
| Colleagues in their immediate work area act in accordance with the APS values in their everyday work | ±1.1% | 88.6% |
Results have not been reported for questions where the number of unweighted responses is fewer than 20. This approach has been adopted for two reasons: firstly, to eliminate the possible identification of individuals who responded to these questions; secondly, to remove less reliable results from the analysis.
This approach has not affected reporting of results at the aggregate level; however, it has limited our ability to report on disaggregated data where the sample size is small—as is sometimes the case for questions following ‘filter’ questions.
It should also be noted that estimates relating to disaggregated data where the sample size is small will have a higher standard error because the population size responding to that question is lower than for aggregated data or disaggregated data where the sample size is large; for example, as can be seen from the following table illustrating the standard errors associated with estimates for disaggregated data, the standard error for Indigenous employees is higher than other standard errors because the population size responding to that question is small.
| Question | 95% confidence interval | Estimate result |
|---|---|---|
| Agree that manager provides them with the support needed to do their job (women) | ±1.8% | 74.1% |
| Agree that manager provides them with the support needed to do their job (men) | ±1.9% | 76.7% |
| Agree that manager provides them with the support needed to do their job (people with disability) | ±6.4% | 72.1% |
| Agree that manager provides them with the support needed to do their job (people without disability) | ±1.3% | 75.5% |
| Agree that manager provides them with the support needed to do their job (Indigenous employees) | ±12.0% | 55.9% |
| Agree that manager provides them with the support needed to do their job (non-Indigenous employees) | ±1.3% | 75.5% |
Interpretation of scales
Scales were included in any question that required a respondent to measure the strength or level of a theoretical construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.
The scales used in the surveys were generally balanced—that is, they allowed the respondents to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.
When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent from ‘satisfied’ to ‘very satisfied’.
Summary indexes
Summary indexes have been used to assist analysis of results of a number of survey questions comprising several parts. The indexes operate to condense a multiple response question into a single index for comparative purposes; for example, in exploring respondents’ overall level of job satisfaction, a question comprising 15 factors was summarised into a single index using a point scoring system. In this way, analysis of the 15 job satisfaction factors can be supplemented by analysis at the summary level.
Coding of open-ended responses
The employee survey questionnaire provided specified response options for each question. It also included open-ended response options for some questions, which enabled respondents to provide a text response to a question. Open-ended options were commonly provided, for example, as part of a specified response question in the form of ‘other (please specify)’.
Some open-ended responses have been coded to assist analysis. Coding involved, for example, removing irrelevant and incidental comments from statistical outputs as well as counting relevant comments against the appropriate response option.
Data cleaning
Every effort is made to ensure the integrity of data from the employee survey. Where inaccuracies are discovered, or a different methodology is adopted, the historical data has been revised. For this reason, caution should be exercised when comparing data in this report with that from earlier years.
As in previous years, survey respondents were asked to choose the five workplace factors (from among 15 factors) which contribute most to their overall job satisfaction. A different approach to the cleaning of the data was used this year. Specifically, responses that did not conform with the directions in the question (e.g. those that exceeded five factors) were excluded. To retain the comparability of the data, the same approach was also applied to the job satisfaction data from previous years. While the effect of this on overall levels of job satisfaction is small (the overall level of job satisfaction in 2004–05 was reported as being 71% but using the revised data was 72%), the effect on some smaller sub-populations is larger (e.g. the level of job satisfaction for people with disability in 2004–05 was reported as being 65% but using the revised data was 70%).
The revised job satisfaction data for 2003–06 is included in the State of the Service Employee Survey Results 2005–06.
- The sample was drawn in April 2006 and this was based on the most recent data provided by agencies, which was January 2006 for the majority (over 90%) of employees.
