Learning evaluation in the APS
Learning across the APS is changing. Learning practitioners are embracing the vision and targets established by the APS Learning & Development Strategy. Learning initiatives are evolving to reflect a continuous learning culture and taking advantage of technology in new ways.
With so much happening across Learning & Development (L&D) in the APS, we must start capturing stories of positive change and evaluate the effectiveness and impact of this transformation. This will help us understand the strengths and ongoing opportunities available.
The APS Learning Evaluation Framework (the Framework) and guidances help learning practitioners across the APS to evaluate the learning initiatives they offer.
Why evaluate?
There are lots of reasons to evaluate learning initiatives. Evaluation is an in-depth and values-based process that helps an organisation to understand what is happening and why.
Formative evaluation helps drive continuous improvement and innovation. It helps assess how well an initiative is, and is not, working and assists in determining what changes are needed for improvement.
More than a technical process, formative learning evaluation is deeply influenced by organisational values and the drive for continuous improvement. When done well, it aligns with your agency's purpose and strategic goals and leads to improved learning outcomes.
APS Learning Evaluation Framework
The Framework emphasises the relationship between continuous and routine monitoring, evaluation and organisational insights.
Evaluating the effectiveness of APS learning is central to ensuring the quality of L&D initiatives and individual learning.
The effectiveness of L&D practice across the APS is determined by evaluating against the following criteria:
- Efficiency
- Impact
- Relevance
- Flexibility
- Accessibility
Key evaluation questions
| Key evaluation questions | |||||
|---|---|---|---|---|---|
| Effectiveness | Accessibility | Flexibility | Relevance | Impact | Efficiency |
|
Key evaluation questions - How effective was the initiative in helping the participant perform their work? - What were the unintended effects (positive and negative) of the initiative? - To what extent was the initiative successful in achieving its objectives? - How valuable is the initiative to the participant? |
Key evaluation questions - To what extent did all the targeted audiences benefit equally? - How well did the initiative contribute to building access to learning? - How successful was the initiative in reaching hard-to-reach groups? |
Key evaluation questions - How well does the initiative complement other initiatives? - Are there other initiatives which contradict or inhibit the success of this initiative? - To what extent can the initiative be adapted and scaled to different environments and contexts? |
Key evaluation questions - How relevant is the initiative in responding to the learning needs of the participants, agency and APS? - How relevant was the initiative for the participants? - How relevant was the design of the initiative to the environment and context? |
Key evaluation questions - What difference has the initiative made to participants, agencies, and the APS? - How has the initiative produced significant change? - How has the initiative benefited the participants’ careers? - How has the initiative led to other indirect changes? |
Key evaluation questions - How efficient was the implementation of the initiative? - To what extent were the initiative’s objectives met against budget and resourcing? - How did anticipated costs change? What was the result of this? - How adequate was the budget in meeting the requirements of the initiative? |
| Information sources | ||||
|---|---|---|---|---|
|
Primary sources (eg participants and supervisors) - Questionnaires - Focus groups - Case studies - One-on-one interviews - Pre/post-program participant testing - User observations |
Other sources - Participation rates -Participant demographics - Corporate and business area plans - Other initiative outlines and work plans - Initiative proposal and implementation plan - Benchmarking against comparable initiatives - Financial and resourcing reporting |
|||
Learning evaluation guidance
To accompany the Framework, the APSC has developed resources for learning practitioners to use when evaluating learning - these resources are found in the Learning Evaluation Handbook.
The Handbook is broken up into nine tasks, which are the foundations for good learning evaluation practice:
For more information, and to begin exploring the nine tasks, visit the Learning Evaluation Handbook page.