Evaluating CPD
Opening summary
As part of the Advanced Teacher Status (ATS) programme, participants are required to undertake a quality improvement project.
SET’s Professional Status team reviewed all 60+ quality improvement projects submitted as part of the October 2020 ATS cohort, shortlisting the top 12 using a scoring matrix, including the one below from Vivienne Stockill, Lecturer at Heart of Worcestershire College.
This editorial provides an overview of Vivienne’s improvement project which focused on models for better evaluating continuing professional development (CPD).
Research context
The Tyler et al report (2017) on the role of advanced practitioners (APs) identified that tracking, monitoring and measuring of outcomes from staff development interventions was often inconsistent. However, Clay (2018) states that effective professional development has a significant impact on learners’ achievement and is the educational leadership activity that has the largest impact on learner outcomes. She goes on to add that it should always be undertaken with a clear and specific need in mind, and to assess the impact of the activity, the question that should be addressed is what success would look like if the need is met. Therefore, there is a need to establish what teachers and the organisation expect to gain from the intervention, and how the benefit of the support will be measured.
Moreover, there is an Ofsted requirement that to achieve “outstanding” grading in the inspection of Leadership and Management, the organisation must evidence that leaders ensure teachers receive focused and highly effective professional development (Further Education and Skills Inspection Handbook, para 275).
At present, in this college, a simple form is used at the end of staff development sessions which asks participants if they enjoyed the session, if the aims of the session were met and if the level was appropriate to current levels of skills and knowledge. In addition, they are asked if there is anything that participants will take away from the session to try out within their own practice and whether further input is needed in terms of additional professional development or one-to-one support.
This project examines how professional development programmes delivered by advanced practitioners (and others) may be better evaluated by applying a suggested model in real situations.
Aims and objectives
To evaluate the impact of staff development activities delivered by advanced practitioners and others, with the objectives to:
- Develop a model to evaluate profession development programmes
- Apply the model in real situations to evaluate the impact of professional development programmes.
Methodology
A review of relevant literature was conducted to ascertain methodology for the evaluation of staff development.
Data was collected from a number of staff development sessions conducted between September 2020 and April 2021 to evaluate the efficacy of the intervention. The data was obtained using a variety of methods: questionnaires, observations, quizzes and interviews.
The Kirkpatrick model (LinwaysTechnologies, 2020) for evaluation of professional development was followed with each level building upon the next, as illustrated in the diagram.
- Level 1: reaction. Data was collected via questionnaires at the end of each session, with space being provided for any additional comments participants wished to make.
- Level 2: learning. Data was obtained by a variety of methods according to the subject material. For example, testing cognitive learning via a quiz, a work-based task requiring participants to apply knowledge from the sessions in an assessment activity within groups, or constructing a lesson plan using tools and techniques covered in the session.
- Level 3: behaviour. Evaluation was attempted with at least 20% of attendees chosen at random approximately six weeks after each intervention. This was not fully effective as there was resistance to observing performance in the classroom, and additional difficulties as teaching moved online due to the pandemic. However, it was possible to obtain some written responses and conduct one-to-one interviews to check if practitioners felt confident and able to put into effect what they had learned when returning to the classroom.
- Level 4: impact. Evaluation was not possible as this would need to be carried out by line managers and the effects would not be evident until after the end date for this project.
All responses to questionnaires, interviews and quizzes were collected anonymously to enable confidentiality. Participants were informed that their responses were anonymous and would only be used for the purpose of this study, and that when responses had been collated and incorporated into a report, the data would be held securely within the college’s systems and destroyed after 12 months. This enabled participants to answer questions candidly.
Project findings and recommendations
Summary of literature review:
- In order to have impact, professional development should be planned with clear objective in mind.
- It should be evidence-based and focused on the needs of the participants as well as their prior knowledge, skills and expertise and the depth of impact required.
- Goals for development activity should not rely solely on the perspective of managers, but should come from a variety of sources, including the most pressing needs of the organisation and the views of teachers and support staff.
- In order to fully understand how professional development activities have led to improved teacher practice, it is vital to carry out an evaluation of the activities over a period of time. A useful tool to inform the evaluation of generic professional development is the Kirkpatrick model.
- The diagram summarises some suggestions for sources of material for staff development activities and where the impact of those activities should be measured.
Collection of data
Data was collected from a total of seven staff development sessions and involved 131 participants in sessions.
Using the Kirkpatrick model for evaluation of professional development mentioned above, data was collected at levels one, two and three. Raw data has been redacted to ensure confidentiality but findings are summarised below.
Level 1: reaction
A number of staff commented on the fact that it was enjoyable working with other colleagues and helpful to be able to engage with staff from different departments. However, participants did not understand in many cases why they were required to attend, and no pre-training briefing was carried out.
Further comments that were made suggest that in some cases, participants felt that the material covered had been duplicated in other sessions they had attended. Given that attendance was largely compulsory, this may indicate a need for more careful consideration by line managers as to the specific development needs of staff.
Level 2: learning
There was a disappointing response to quizzes and tasks which were intended to measure learning from the session. Where this was done at the time of the development session, a 100% response was obtained, but where staff were asked to respond to quizzes and tasks at some time after the session, the response rate was poor. In fact, it was only possible to demonstrate understanding in 48% of attendees. However, of those who responded to tasks and quizzes, 93% demonstrated understanding of the concepts.
Response rates were partly impeded by the fact that some sessions took place online and this did not always lend itself to obtaining immediate feedback, but if the effectiveness of staff development is to be demonstrated, consideration needs to be given to how this may be undertaken.
Level 3: behaviour
Responses suggested that teachers take knowledge from the sessions and are keen to put it into practice. However, there was no follow up action by line managers to use opportunities to support practitioners in their new learning, and without evaluation of the impact (level 4), it is not possible to assess whether this is done effectively to improve outcomes for learners.
Conclusions
From the results above, it is difficult to see how subject matter for staff development is determined and how this is communicated to staff. Further, there appears to be no preparation for staff prior to the development session, and no de-briefing or evaluation by line managers after the event.
For evaluation to be effective, the organisation as a whole needs to see the value to be gained from it. Effective performance management in staff development is not seen or valued as a line management activity as there is little evidence of line managers actively working with team members on individual development. Further, observations which would confirm the impact of staff development are seen as an activity that is carried out by managers, rather than using the existing peer review process or allowing APs to carry out learning walks for evaluation purposes.
Recommendations
- Professional development should be linked to performance management so that staff understand why they are completing it.
- Briefings should take place prior to the activity and de-briefing after the activity.
- Participants in sessions should be encouraged to engage with activities that gather feedback and check that learning has taken place.
- Application of learning should be evaluated within the classroom – line managers, APs, peer reviews.
- A further study should be carried out in two years’ time.
Professional reading
Clay, B. (2016). Five principles to help you evaluate your CPD. [Online] Available at: https://www.sec-ed.co.uk/best-practice/five-principles-to-help-you-evaluate-your-cpd/ [Accessed 28 October 2020].
Guskey, T. (2002). Does it make a difference? Evaluating professional development. [Online] Available at: http://www.ascd.org/publications/educational-leadership/mar02/vol59/num06/Does-It-Make-a-Difference%C2%A2-Evaluating-Professional-Development.aspx [Accessed 29 October 2020].
Herbert-Smith, K. (2018). How can you measure the impact of CPD? [Online] Available at: https://blog.irisconnect.com/uk/community/blog/how-can-you-measure-the-impact-of-cpd/ [Accessed 28 October 2020].
Higgins, S., Cordingley, P., Greany, T. and Coe, R. (2014). Developing Great Teaching [Online] Available at: https://tdtrust.org/about/dgt/ [Accessed 26 October 2020]. London: Teacher Development Trust.
Joyce, B. and Showers, B. (2002). Student achievement through staff development (3rd edition). London: Longman.
Linways Team (2020). An introduction to the Kirkpatrick Model: The four levels of learning evaluation. [Online] Available at: https://stories.linways.in/an-introduction-to-the-kirkpatrick-model-ac3058a16295 [Accessed 28 October 2020].
Pandey, A. (2018). Using Kirkpatrick’s Model of Training Evaluation. [Online] Available at: https://elearningindustry.com/roi-of-elearning-using-kirkpatricks-model-training-evaluation-determining [Accessed 28 October 2020].
Teacher Development Trust (2020). CPD Processes. [Online] Available at: https://tdtrust.org/about/evidence/#CPDprocesses [Accessed 26 October 2020]. London: Teacher Development Trust.
Thedham, J. (2019). How managers can support and develop advanced practitioners – an organisational approach. [Online] Available at: https://improving-teaching.excellencegateway.org.uk/content/etf3124 [Accessed 7 December 2020]. London: The Education and Training Foundation.
Tyler, E., Marvell, R., Green, M., Martin, A., Williams, J. and Huxley, C. (2017). Understanding the Role of Advanced Practitioners in English Further Education. [Online] Available at: https://www.excellencegateway.org.uk/content/etf2801 [Accessed 7 December 2020]. London: The Education and Training Foundation.
Weston, D. and Clay, B. (2018). Unleashing Great Teaching. London: Routledge.
Content from ATS Academic Poster - Viva preparation – Vivienne Stockill – 2022
Find out more about the Advanced Teacher Status (ATS) programme.