Home > Survey methodologies

AGENCY SURVEY METHODOLOGY

The scope of the agency survey was the 87 APS agencies, or semi-autonomous parts of agencies, employing 20 or more staff under the PS Act. The seven agencies with fewer than 20 APS employees had the option of not participating in the agency survey. All seven agencies opted not to take part. These seven agencies had a total of 65 APS employees.

The 87 participating agencies were sent the online survey on 2 June 2004 for completion. Agencies were given six weeks to complete and submit their response. As part of their survey return, agency heads were required to ‘sign off’ their agency’s response. All 87 agencies responded to the online agency survey, although several agencies submitted late responses and three agencies submitted responses after survey data were due to be downloaded, thereby delaying analysis.

EMPLOYEE SURVEY METHODOLOGY

The employee survey was designed to establish the views of APS employees on a range of issues, including work-life balance, the APS Values, the Code of Conduct, job satisfaction, and learning and development. The results of the employee survey are one of the main sources of information on which the APS Commission has drawn during the preparation of this report.

The employee survey was also designed to complement the agency survey. The results of the employee survey were, in part, intended to act as a ‘reality check’ in analysing responses to the agency survey. To achieve this objective, similar questions were asked in both surveys on a range of topics. Additional questions, suitable for employees but not for agencies (such as on job satisfaction), were also included in the employee survey.

SCOPE AND COVERAGE

The scope of the survey was all APS employees (both ongoing and non-ongoing) in agencies with 100 or more APS employees. Employees in agencies that employed fewer than 100 APS employees were excluded on the basis that their responses could possibly identify them.

The survey sample was drawn from APSED in two stages: the first group was drawn on 7 April 2004, and was supplemented by a further group drawn early in May 2004. APSED indicated that at 31 March 2004 the total number of APS employees was 130,908. The survey sample was selected from the total population of APS employees from agencies with 100 or more APS employees, which was 129,905. Appendix 1 provides information on agencies’ APS employee numbers as at 31 March 2004.

STRATIFICATION

A stratified random sample of 6437 APS employees was selected from APSED. The sample was stratified by:

• level (APS, EL and SES classification groups)
• agency size (small: 100—250 APS employees; medium: 251—1000 APS employees; and large: >1000 APS employees)
• agency (only for the 22 large agencies)
• location (ACT and non-ACT).

To enable sound statistical inferences to be made about all APS employees, the individuals were randomly selected from each of the strata. Each individual within a stratum had an equal chance of selection.

The sampling rates varied between the strata to ensure that sufficient statistical accuracy would be achieved for survey estimates from APS employees with the key characteristics captured by the stratification variables (level, location, agency and agency size). To gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate was required than for a larger population (such as APS-level employees).

The accuracy requirements varied between the demographic variables listed above, and this also led to differing sampling rates for these demographic variables.

This stratification process has not introduced a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account. See the section ‘Weighting and estimation’ for further details.

REPORTING OF LARGE AGENCY RESULTS

The survey was designed to allow the APS Commission to provide the large agencies (if they wished) with a copy of their own results to the employee survey for internal management purposes, subject to the results satisfying a statistical accuracy benchmark. To enable this to occur, the 22 agencies with more than 1000 APS employees were included separately in the stratification process (see the section ‘Stratification’ above).

Where relevant, the State of the Service report includes agency level results of large agencies that met the minimum number of weighted responses (see the section ‘Measures of accuracy and error’ below).

PRIVACY, ANONYMITY AND CONFIDENTIALITY

Maintaining confidentiality throughout the entire employee survey process was a primary concern to the APS Commission.

Privacy arrangements for APSED preclude APS Commission staff, other than those in the APSED Team, the Group Manager, Review and Evaluation Group, and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of those individuals selected in the sample from APSED was not available to the Commission’s State of the Service Team or any other non-APSED staff involved in the survey. A small number of ORIMA Research staff had access to the sample. All responses to the survey were anonymous so individuals could not be identified.

Each person invited to participate in the employee survey was provided with a unique password. This prevented multiple responses from individual respondents.

SURVEY DESIGN

The employee survey conducted in 2003 was used as the basis of this year’s survey, although while some questions are to be asked annually, it is anticipated that a number of other questions will be asked every two or three years. Refinements were made to some survey questions in light of last year’s experience, however, the need to maintain comparable time series data meant that changes were kept to a minimum.

The draft employee survey was subjected to cognitive testing involving individuals at the APS and EL levels from Defence, FaCS and the APS Commission. A further refined draft survey was then tested at a focus group session attended by individuals at the APS and EL levels from the ATO, NAA, ABS, Centrelink, Customs, DVA, DAFF, Health and FaCS.

The majority of questions, 64 of 76, were asked of all respondents. Two questions were asked of SES level employees only. Five questions were asked of EL and SES employees only. Five questions were asked of APS and EL level employees but not SES.

The questions asked of employees were not dependent on location (inside or outside the ACT) or agency size (small, medium or large).

The employee survey was delivered using two methods. The main delivery method was online via a passwordprotected internet site. The majority of the sample was sent an email from the Commissioner inviting them to participate in the online survey.

The secondary, paper-based delivery method was developed and implemented for employees working in agencies that do not have access to an individual email account. These employees received a letter from the Commissioner inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.

The 6977 invitation emails and letters were sent out to the sample between 19 and 31 May 2004.1 Respondents were asked to complete the survey and submit or return it to ORIMA Research by Friday 11 June 2004.

An adjustment was made to the final sample size to account for those out of scope of the survey (including repeat bounced emails, those ‘out of office’ for the entire survey period and those known to be no longer employed in the APS at the time of the survey). The final sample was reduced by 540 to 6437.

WEIGHTING AND ESTIMATION

The survey responses were re-weighted to reflect the characteristics of the underlying population of APS employees. This was done to ensure that the overall demographic characteristics (used for sample selection) of the survey results exactly matched the demographic characteristics of all APS employees. The re-weighting process was based on the four demographic characteristics used for selection of the sample, namely:

• level (APS, EL and SES classification group)
• location (ACT and non-ACT)
• agency size (small: 100—250 APS employees; medium: 251—1000 APS employees; large: >1000 APS employees)
• agency (only for the 22 large agencies).

1 At the request of the Secretary of DVA, invitations to participants in the survey were sent to DVA staff on Monday 31 May 2004 so as to avoid overlap with DVA’s staff survey, which closed on Friday 28 May 2004

There were therefore 144 different weights applied–level (3) multiplied by location (2) multiplied by agency size and agency (24). For this survey, the weights were calculated by dividing the populations of each stratum by the number of respondents to the survey in each stratum. For example, if there are 4000 ELs at medium-sized agencies in the ACT and 200 responded, the weight assigned to each ACT EL working in a medium-sized agency is 20. If the data were not re-weighted, some strata could be over-represented and others under-represented in the total survey results.

The weighting approach is based on last year’s approach. The application of a uniform approach to sample selection and weighting will assist in the development of times series data.

The weighting approach adopted assumes that respondents respond in the same way as non-respondents for the characteristics of interest. The weighting method above assumes that the responding persons represent the non-responding persons.

In this survey, with a response rate of 50%, there would need to be a marked difference in the views of non-respondents from those of the respondents to alter or bias the overall results to any significant extent. For analysis presented in this report it was assumed that there was no significant bias between those who responded in the survey and those who did not respond.

The results are calculated under the assumption that responding persons answer in the same way as non-respondents. This should be considered when using the data to make inferences about the APS population.

Results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.

MEASURES OF ERROR AND ACCURACY

Two types of error can occur in sample surveys: sampling error and non-sampling error. Sampling error arises because in a sample survey not all of the population are surveyed. Hence a measured sample statistic is not usually identical with the true population behaviour. Non-sampling errors cause bias in statistical results and can occur at any stage of a survey and can also occur with censuses (i.e. when every member of the target population is included). Sampling error can be estimated mathematically whereas estimating non-sampling error can be difficult. It is important to be aware of these errors, in particular non-sampling error, so they can be either minimised or eliminated from the survey.

Non-sampling error

The survey received a response rate of 50%. This response rate excludes responses that were received but were insufficiently complete to provide input into the data generated. This response rate is very creditable for a voluntary survey.

Non-sampling errors can result from imperfections in reporting by respondents, errors made in recording and coding of responses, and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors. However, every effort was made to reduce the non-sampling errors to a minimum by careful survey design and efficient operating procedures. In particular, the online survey design minimised the possibility of errors made in recording and coding of responses, as the respondents themselves entered the data when responding to the survey.

In addition, identifiable errors made by respondents while completing the survey were removed from the results database. For example, responses made by APS level employees to an EL only question have been removed to ensure the integrity of the data. Blank responses were generally coded to non-response categories.

The exception to this practice arose where responses were needed for demographic items for weighting purposes. In instances where this occurred, survey responses were disregarded.

Sampling error

One measure of the sampling error of an estimate is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% Confidence Interval.

For instance, we are 95% confident that the estimate of the population that would be satisfied that their supervisor would support the use of flexible work practices is between 80.2% and 82.8% (an estimate of 81.5% and a confidence interval of +/-1.3% based on a standard error of 0.65%).

The following table illustrates the standard errors from the sample design associated with estimates from 11 key questions in the employee survey. Generally, the higher the sample size for a question, the lower the standard error. For example, questions following a ‘filter’ question are more likely to have a slightly higher standard error because the population size responding to that question is lower than for ‘non-filtered’ questions. The standard error for the performance pay question estimate, for example, is slightly higher than for many others because only employees with access to performance-related pay were asked the question, thereby reducing the sample size.

Results have not been reported for questions where the number of unweighted responses is fewer than 20. This approach has been adopted for two reasons: firstly, to eliminate the possible identification of individuals who responded to these questions; secondly, to remove less reliable results from the analysis. This approach has not affected reporting of results at the aggregate level; however, it has limited our ability to report on disaggregated data where the sample size is small–as is sometimes the case for questions following ‘filter’ questions.

It should also be noted that estimates relating to disaggregated data where the sample size is small will have a higher standard error because the population size responding to that question is lower than for aggregated data or disaggregated data where the sample size is large. For example, as can be seen from the following table illustrating the standard errors associated with estimates for disaggregated data, the standard error for Indigenous employees is higher than other standard errors because the population size responding to that question is small.

INTERPRETATION OF SCALES

Scales were included in any question that required a respondent to measure the strength or level of a theoretical construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.

The scales used in the surveys were generally balanced–that is, they allowed the respondents to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.

When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent to ‘very satisfied’ rather than ‘satisfied’.

SUMMARY INDEXES

Summary indexes have been used to assist analysis of results of a number of survey questions comprising several parts. The indexes operate to condense a multiple response question into a single index for comparative purposes. For example, in exploring respondents’ overall level of job satisfaction, a question comprising 16 factors was summarised into a single index using a point scoring system. In this way, analysis of the 16 job satisfaction factors can be supplemented by analysis at the summary level.

CODING OF OPEN-ENDED RESPONSES

The employee survey questionnaire provided specified response options for each question. It also included open-ended response options for some questions, which enabled respondents to provide a text response to a question. Open-ended options were commonly provided, for example, as part of a specified response question in the form of ‘other (please specify)’.

Some open-ended responses have been coded to assist analysis. Coding involved, for example, removing irrelevant and incidental comments from statistical outputs as well as counting relevant comments against the appropriate response option.