Appendices

The appendices provide background information to the production of the report

Appendix 2: State of the Service survey methodologies

Agency survey methodology

The scope of the agency survey was the 90 APS agencies, or semi-autonomous parts of agencies, employing at least 20 staff under the Public Service Act 1999.

The 90 participating agencies were sent the online survey on 4 June 2008 for completion. Agencies were given six weeks to complete and submit their response. As part of their survey return, agency heads were required to ‘sign off’ their agency’s response. All 90 agencies responded to the online agency survey. The results of the agency survey are one of the key sources of information which the Commission has relied on throughout the preparation of this report.

Employee survey methodology

The employee survey sampling methodology was developed in consultation with the Australian Bureau of Statistics. This year, the content was designed to establish the views of APS employees on a range of issues, including work-life balance, learning and development, job satisfaction, leadership, personal productivity, interactions with Ministers and the Parliament and general impressions about the APS. The results of the employee survey are one of the main sources of information on which the Commission has drawn in preparing this report.

Scope and coverage

The scope of the employee survey was all APS employees (both ongoing and non-ongoing) in agencies with at least 100 APS employees. Employees in agencies that employed fewer than 100 APS employees were excluded on the basis that their responses could possibly identify them.

The survey sample was drawn from APSED on 11 April 2008, at which time APSED indicated that the total number of APS employees was 160,450. The survey sample was selected from the total population of APS employees from agencies with at least 100 APS employees, which was 159,394. Appendix 1 provides information on agencies’ APS employee numbers as at 11 April 2008.

Stratification

A stratified random sample of 9,406 APS employees was selected from APSED. The sample was stratified by:

To enable sound statistical inferences to be made about all APS employees, individuals were randomly selected from each of the strata. Each individual within a stratum had an equal chance of selection.

The sampling rates varied between the strata to ensure that sufficient statistical accuracy would be achieved for survey estimates from APS employees with the key characteristics captured by the stratification variables (level, location, agency and agency size). To gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate was required than for a larger population (such as APS 1–6 employees).

The accuracy requirements varied between the demographic variables listed above, and this also led to differing sampling rates for these demographic variables.

This stratification process has not introduced a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account (see the section ‘Weighting and Estimation’ below for further details).

Reporting of results from agencies with at least 400 employees

The survey was designed to enable agencies with at least 400 employees, DCC, DHS, RET and the Commission to receive a copy of their own results from the employee survey for internal management purposes—subject to the results satisfying a statistical accuracy benchmark. For this to occur, these 47 agencies were included separately in the stratification process (see the section ‘Stratification’ above).

Privacy, anonymity and confidentiality

Maintaining confidentiality throughout the entire employee survey process was of primary concern to the Commission.

Privacy arrangements for APSED preclude Commission staff, other than those in the APSED Team, the Group Manager of the Evaluation Group, and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of those individuals selected in the sample from APSED was not available to the Commission’s State of the Service Team or any other non-APSED staff involved in the survey. A small number of ORIMA Research staff had access to the sample. All responses to the survey were anonymous so individuals could not be identified.

Each person invited to participate in the employee survey was provided with a unique password. This prevented multiple responses from individual respondents.

Survey design

The employee surveys conducted in previous years were used as the basis for this year’s survey. Some questions have been included on an annual basis, other questions have been cycled through on a two- or three-year basis, and others were included for the first time this year to address topical issues. To ensure the Commission maintains comparable time series data, any changes to questions repeated from previous years were kept to a minimum.

The draft employee survey was subjected to individual and paired pilot testing involving individuals at the APS 1–6 and EL classifications from Infrastructure, DAFF, Medicare Australia, ATO, WO, FaHCSIA, DEEWR, Centrelink, AEC and Customs.

The employee survey was delivered using two methods. The main delivery method was online via a password-protected Internet site. The majority of employees in the sample were sent an email from ORIMA Research on behalf of the Commissioner inviting them to participate in the online survey.

A secondary, paper-based delivery method was developed and implemented for employees working in agencies who do not have access to an individual email account or do not have (or have only limited) access to the Internet. These employees received a letter from the Commissioner inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.

The 9,406 invitation emails and letters were sent out to employees in the sample on 12 May 2008. Respondents were asked to complete the survey and submit or return it to ORIMA Research by Friday 6 June 2008.

An adjustment was made to the final sample size to account for those out of scope of the survey (including repeatedly bounced emails, those ‘out of office’ for the entire survey period and those known to be no longer employed in the APS at the time of the survey).1 As a result, the final sample was reduced by 328 to 9,078.

Weighting and estimation

The survey responses were re-weighted to reflect the characteristics of the underlying population of APS employees. This was done to ensure that the overall demographic characteristics (used for sample selection) of the survey results exactly matched the demographic characteristics of all APS employees. The re-weighting process was based on the four demographic characteristics used for selection of the sample, namely:

There were, therefore, 294 different weights applied—level (3) multiplied by location (2) multiplied by agency size and agency (49). For this survey, the weights were calculated by dividing the populations of each stratum by the number of respondents to the survey in each stratum; for example, if there are 4,000 ELs in medium agencies in the ACT, and 200 responded, the weight assigned to each EL working in a medium agency in the ACT is 20. If the data were not re-weighted, some strata could be over-represented and others underrepresented in the total survey results.

The weighting approach is based on that taken in previous years. The application of a uniform approach to sample selection and weighting continues to assist in the development of time series data.

The weighting approach adopted assumes that respondents respond in the same way as non-respondents for the characteristics of interest. The weighting method above assumes that the responding persons represent the non-responding persons.

In this survey, with a response rate of 65%, there would need to be a marked difference in the views of non-respondents from those of the respondents to alter or bias the overall results to any significant extent. For analysis presented in this report it was assumed that there was no significant bias between those who responded in the survey and those who did not respond. This should be considered when using the data to make inferences about the APS population.

Results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.

Measures of error and accuracy

Two types of error can occur in sample surveys: sampling error and non-sampling error. Sampling error arises because in a sample survey not all of the population are surveyed. Hence a measured sample statistic is not usually identical with the true population behaviour. Non-sampling errors cause bias in statistical results and can occur at any stage of a survey and can also occur with censuses (i.e. when every member of the target population is included). Sampling error can be estimated mathematically, whereas estimating non- sampling error can be difficult. It is important to be aware of these errors, in particular non-sampling error, so that they can be either minimised or eliminated from the survey.

Non-sampling error

The survey received a response rate of 65%—the highest response rate since the employee survey was first conducted six years ago. This response rate excludes responses that were received but were insufficiently complete to provide input into the data generated. This response rate is very creditable for a voluntary survey.

Non-sampling errors can result from imperfections in reporting by respondents, errors made in recording and coding of responses, and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors. However, every effort has been made to reduce the non-sampling errors to a minimum by careful survey design and efficient operating procedures. In particular, the online survey design minimised the possibility of errors being made in the recording and coding of responses, as the respondents themselves entered the data when responding to the survey.

In addition, identifiable errors made by respondents while completing the survey were removed from the results database; for example, responses made by APS 1–6 employees to an SES-only question have been removed to ensure the integrity of the data. Blank responses were generally coded to non-response categories. The exception to this practice arose where responses were needed for demographic items for weighting purposes. In instances where this occurred, survey responses were disregarded.

Sampling error

One measure of the sampling error of an estimate is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% Confidence Interval.

We are 95% confident, for instance, that the estimate of the population who agree that their manager ensures fair access to developmental opportunities for employees in their work group is between 70.9% and 73.5% (an estimate of 72.2% and a confidence interval of +/-1.3 percentage points based on a standard error of 0.65 percentage points).

The following table illustrates the standard errors from the sample design associated with estimates from 10 key questions in the employee survey.

Question 95% confidence interval Estimate result
Understand how their agency’s decision-making processes operate (e.g. relevant committee structures and how committees are linked) ±1.3pp 63.2%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group ±1.3pp 72.2%
Agree that in their agency, the leadership is of a high quality ±1.3pp 45.6%
Agree that their agency encourages the public to participate in shaping and administering public policy ±1.4pp 47.7%
Agree that their input is adequately sought and considered about decisions that directly affect them ±1.4pp 52.0%
Considering their work and life priorities, they are satisfied with the work..life balance in their current job ±1.3pp 71.2%
Agree that in their experience, their agency’s culture usually encourages a constructive approach to collaboration with other public service agencies ±1.6pp 82.7%
Agree that their agency values and manages diversity in the workplace well ±1.3pp 64.9%
Want to try new ideas, but the public service discourages risk taking ±1.3pp 38.9%

Results have not been reported for questions where the number of unweighted responses is fewer than 30. This approach has been adopted for two reasons: firstly, to eliminate the possible identification of individuals who responded to these questions; secondly, to remove less reliable results from the analysis. Results with a confidence interval of more than ± 15 percentage points have also been excluded from the analysis. This approach has not affected reporting of results at the aggregate level; however, it has limited our ability to report on disaggregated data where the sample size is small—as is sometimes the case for questions following ‘filter’ questions.

It should also be noted that estimates relating to disaggregated data where the sample size is small will have wider confidence intervals because the population size responding to that question is lower than for aggregated data or disaggregated data where the sample size is large; for example, as can be seen from the following table illustrating the confidence intervals associated with estimates for disaggregated data, the confidence interval for Indigenous employees is higher than other confidence intervals because the population size responding to that question is small.

Question 95% confidence interval Estimate result
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (women) ±1.5pp 71.1%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (men) ±1.7pp 73.7%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (people with disability) ±4.8pp 68.9%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (people without disability) ±1.2pp 72.4%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (Indigenous employees) ±7.3pp 66.9%
Agree that their manager ensures fair access to developmental opportunities for employees in their work group (non-Indigenous employees) ±1.1pp 72.3%

Interpretation of scales

Scales were included in any question that required a respondent to measure the strength or level of a theoretical construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.

The scales used in the surveys were generally balanced, that is, they allowed respondents to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.

When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale, that is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent from ‘satisfied’ to ‘very satisfied’.

Open-ended responses

The employee survey questionnaire provided specified response options for most questions. It also included open-ended response options for some questions, which enabled respondents to provide a text response to a question. Open-ended options were commonly provided, for example, as part of a specified response question in the form of ‘other (please specify)’.

Coding

Some open-ended responses have been coded to assist analysis. Coding involved, for example, removing irrelevant and incidental comments from statistical outputs as well as counting relevant comments against the appropriate response option.

Interpretation

The report draws on the actual comments employees provided through the open-ended questions to complement other information. Employees’ comments represent a rich and valuable data source; however, they do not necessarily represent the views of all employees.

Data cleaning

Every effort has been made to ensure the integrity of data from the employee and agency surveys. Where inaccuracies are discovered, or a different methodology is adopted, the historical data has been revised. For this reason, caution should be exercised when comparing data in this year’s report with that in previous reports. Time series analysis in this report incorporates the historical revisions made to previous datasets.

 

1 The sample was drawn in April 2008 and this was based on the most recent data provided by agencies, which was January 2008 for the majority (over 90%) of employees.

Previous Next: Factor analysis