Abstract

The study aims to evaluate the effectiveness of Punjab Education and English Language Initiative (PEELI) interventions (2018-19) on professional skills of primary school teachers in Punjab using Stufflebeam’s CIPP evaluation model (1983). PEELI is a collaborative project between the Government of the Punjab and British Council being implemented through Quaid-e-Azam Academy for Educational Development Punjab, Pakistan. In this qualitative research, data has been obtained through British Council’s Annual Reports, quarterly reports of third party, semi-structured interviews with four stakeholders, twelve trainers and thirty six primary school teachers in twelve districts of Punjab. The findings suggest that PEELI interventions have created positive impact on classroom delivery for making it child-centric. The study recommends the following strategies to achieve positive outcomes: needs analysis of primary teachers in the beginning of the project, equal opportunity of training to young and senior primary teachers, continuous professional development of teachers and developing follow up mechanism.

Key Words

Punjab Education and English Language Initiative, CIPP Model, Child-Centered Classroom, Professional 

     Development.

Introduction

Punjab Education and English Language Initiative (PEELI) project is collaboration between the Government of the Punjab and British Council Pakistan. This project was initiated in 2013 and was reinstated as an expansion of a three year project (2017-2020) with Memorandum of Understanding (MoU) signed by the Government of the Punjab and British Council Pakistan. The project aims to improve the quality of teaching in primary schools of the Punjab through professional development of the primary school teachers. The project intends to equip 200,000 primary school teachers with knowledge and skills required to adapt activity based teaching approach with the aim to improve the outcomes of the learners in classroom. 

British Council’s ‘Teaching for Success’ model offers flexible design for professional development opportunities of teachers according to needs, priorities and local context of the country all over the world. Teaching for Success uses the following steps; evaluate needs and priorities accurately, design quality projects and programmes, deliver quality projects and programmes and measure impact accurately. British Council is using the same model in Punjab with the title of Punjab Education and English Language Initiative. PEELI project has six components; training, licensing, content and material, institutional capacity development, monitoring and evaluation and policy research and development. These components are interlinked and present a complete model for professional development of teachers and trainers. British Council uses a cascade training model to achieve the target of training of 200,000 primary school teachers. British Council’s trained consultants train the Expert Trainers and these trainers deliver sessions in training of the primary school teachers.  

This study aims to evaluate the PEELI project being implemented in Punjab using Context, Input, Process and Product (CIPP) model. The source of data has been obtained through focus group interviews with three hundred sixty primary school teachers, twelve Expert Trainers from twelve Districts in the Punjab. Four stakeholders including third party were also interviewed. The comparative analysis of the semi-structured interviews and third party monitoring reports was done to evaluate the project. 

Theoretical Framework: CIPP Evaluation Model

The purpose of evaluation is to determine achievement of learning outcomes which may influence the future planning of the similar programs, state Msila and Setlhako (2013). According to Mohebbi et al (2011), CIPP evaluation model is an appropriate model to evaluate quality of a program, following context, input, process and product components. Williams (2000) and Smith and Freeman (2002) state the types of evaluation and their functions as described by Stufflebeam (1971a):

Context evaluation determines purpose and needs of the programs. The focus is on needs analysis, underlying problems and finding reasons for unexploited opportunities. 

Input evaluation determines resources required for the program. It evaluates current utilization of the resources and suggests strategies to achieve unexploited resources. 

Process evaluation determines implementation of the program being evaluated. 

Product evaluation determines results and achievement of the program. 

The most fundamental principle of CIPP model is “not to prove, but to improve” (Stufflebeam & Shinkfield, 2007: 331). According to Zhang, Griffith, et al. (2009) and Zhang, Zeller, et al. (2008), CIPP evaluation model has been adopted by educationists to evaluate educational programs. Matthews and Hudson (2001) adopted the model to evaluate parent training project. CIPP model was also used to examine a faculty development project (Steinert, Cruess, Cruess, & Snell, 2005). Similarly, Chien, Lee, & Cheng (2007) assert that CIPP model supported Taiwan’s national educational indicator systems. 




Purpose and Research Questions 

The study aims to evaluate progress of PEELI project using CIPP model. The study addresses the following questions: 

1. Are the needs of the Expert Trainers and Primary School Teachers addressed in the PEELI project? 

2. Are the required resources being utilized for successful implementation of the project?

3. How well the project is being implemented? 

4. What are the effects of the project on teachers’ capacity building? 

Research Setting and Methods

This is a descriptive and analytical study.  To evaluate the project, we used qualitative methods with the following tools:

·         Focus group interviews with twelve classes consisting of thirty male and female primary school teachers each in twelve districts of Punjab

·         Face to face semi-structured interviews with six male and six female Expert Trainers from twelve districts

·         Face to face semi-structured interviews with four stakeholders

·         Document analysis of eight quarterly monitoring reports by third party and two yearly reports published by the British Council Pakistan

·         Review of the training material

 

 

 

 

Table 1. The Detail Regarding Districts, Number of Primary Teachers and Trainers is given Below

Region	District	Number of Teachers Per District	Expert Trainers
North	Kasur	30	1
	Faisalabad	30	1
	Rawalpindi	30	1
	Mianwali	30	1
Total	4	120	4
South	Bhakkar	30	1
	Khanewal	30	1
	Dera Ghazi Khan	30	1
	Rahim Yar Khan	30	1
Total	4	120	4
Centre	Lahore	30	1
	Sheikhupura	30	1
	TobaTek Singh	30	1
	Jhang	30	1
Total	4	120	4
Total for 12 Districts	12	360 teachers	12 trainers

The interviews and documents were analyzed through interpretation and cross references. 

Analysis

The project has been evaluated using the CIPP model (1970) and each type of evaluation is discussed in detail below:




Context Evaluation

Punjab Education and English Language Initiative (PEELI) project is based on British Council’s continuing professional development (CPD) frameworks for Teachers and Teacher Educators. The rationale behind this approach is to enable teachers of all subjects to understand and plan their own professional development. This was explained by one of the stakeholders from British Council who further explained that this framework had four stages of development, seven enabling skills, five self-awareness features and ten professional practices. The purpose was to develop awareness and encourage practicing child-centered approach promoting active learning in the classroom. Activity based approach encourages discovery learning and develops critical thinking skills of learners. Happy classroom, motivated children and teachers’ understanding of self-professional development, peer competition were some of the objectives of the project. Teachers’ WhatsApp groups, use of social media as a platform for learning were examples of the key objectives of this project. Another stakeholder from Quaid-e-Azam academy for Educational Development, Punjab told that PEELI aimed to enhance quality of classroom teaching at the primary level by enabling the primary teachers to teach effectively. In response to a question, stakeholders from British Council stated that they began with developing training skills of trainers and then trained them on the modules developed for primary teachers. However, need assessment of primary school teachers was done on the basis of feedback of the teachers during training collected by British Council, recommendations submitted by third party in monitoring reports, classroom observations by the British Council team and also feedback received by teachers engaged in longitudinal study. When British Council was asked about the process of selection of trainers, they reported that QAED selected the trainers and British Council validated those trainers through training consultants. 

Third party quarterly monitoring reports published in both years were also reviewed. The findings of the report suggest that PEELI was behind in achieving its numeric targets i.e. target of 200,000 PSTs’ training each year.  116,500 Primary school teachers received 05 days’ training in FY-I (2017-18) and only 95,000 PSTs out of 200,000 received 5 days’ training in FY-II (2018-19). The reasons of not achieving the target include lack of communication between District Education Authorities and District QAEDs, rescheduling/ delays of planned events including trainings after cancellation and intimation to teachers. As far as quality of training was concerned, this may safely be rated as ‘satisfactory’.  The findings of the reports also approved the trainers’ demonstration techniques, their perceptions and practices during training and relevancy of the training content to the trainers and primary teachers’ needs. As far as the other components are concerned, PEELI project achieved its objectives of training of Expert Trainers, Expert English Trainers, developing the content for all trainings, internationally recognized courses i.e. Certificate in English Language Teaching for Adults (CELTA), Professional Award for Teacher Development (PATD), Teaching Knowledge Test (TKT), need-based Institutional Capacity Building (ICB) of QAED staff, monitoring and evaluation of all trainings/ activities and policy and research. In response to a question regarding needs of trainers and teachers, reports as well as analysis of the interviews showed that training catered for needs of both trainers and primary teachers. Third party analyzed the needs of trainers and primary teachers and offered recommendations based on data collected during monitoring of trainings. QAED and British Council developed new modules according to needs of trainers and teachers recommended by the third party. 

According to third party quarterly monitoring reports and British Council’s Annual reports, trainers, by and large, achieved the ‘engagement level’ which was the third performance indicator for trainers. They developed their knowledge and skills through professional learning and practical experience. During interviews, trainers’ expressed their experience of being part of the project saying that PEELI training improved their pedagogical and communication skills. Similarly, almost all primary school teachers agreed that the training improved their pedagogical skills by introducing activity based and student-centered teaching methodology. By adopting activity based teaching strategy, their traditional teaching approach had been changed, that brought a change in their teaching behavior also. Only some of the teachers from rural schools stated that there was no remarkable change in their teaching behavior and teaching methodology because of the environment of their schools and colleagues. It was difficult for them to adopt activity based teaching approach. They further expressed that either intrinsic or extrinsic motivation was required for adopting this teaching approach. Majority of the teachers stated that training improved their communication skills and raised their confidence level for speaking English in class. This also helped them improving their vocabulary. However, a few teachers had a view that it was not possible to continue these practices for them while serving in rural schools. Almost all of the teachers agreed that training changed their teaching behavior. However, some of them explained that continuous professional development could bring more positive results as compared to these short courses once a year. 

Trainers stated that PEELI training improved their planning and management skills which they experienced during the training sessions conducted by them. They practically applied the skills learnt during training. Some of the trainers stated that training boosted their confidence. Moreover, they learnt how to communicate well. Similarly, trainers claimed that their approach towards teaching changed after receiving PEELI trainings. They learnt about child-centered approach and Activity based teaching. 

“Change in direction developed a new   understanding of teaching and learning” [female trainer from Sargodha]

Some trainers stated that training helped them to prepare themselves for plan-A & plan-B. This planning strategy saved their time and improved their confidence for facing any challenging situation. Similarly, class observation and feedback to teachers during micro-teaching sessions enhanced their own training and pedagogical skills. Moreover, understanding of individual differences of teachers developed through training helped them to deal with teachers using different strategies, stated some trainers. 

“Sharing of problems/ issues by primary school teachers, during training, supplemented my   understanding of individual differences”                                                          

Delivering training to primary teachers was hands on experience of understanding their problems. They supported teachers through offering possible solutions of their problems, stated some of the trainers. In response to question related to lesson planning and assessment, majority of the teachers agreed that training improved their lesson planning and assessment skills. They had already learnt those skills during induction training conducted by QAED Punjab but PEELI training micro-teaching sessions gave them opportunity to implement the theory.

Input Evaluation

In interview with British Council team members, it appeared that to ensure quality in development of the content, British Council used already developed teaching for success material, contextualized it putting in visuals, reviewed by Pakistani consultants who have teaching as well as training experience. However, British Council was not satisfied with the copy of print provided to the teachers during training. She further stated that neither the quality of print material was good nor it was properly bound. Whereas stakeholders from QAED, Punjab hold the opinion that cost for printing of material was insufficient in PC-I of the project. 

Third party was asked question about resources and their utilization to implement training in the field. They reported that required resources e.g. training content training rooms, seating arrangement and refreshment were satisfactory during expert Trainers’ training. District QAEDs were venue of training of trainers. Whereas training of primary school teachers was managed at schools which had no proper facilities i.e. training rooms. Thus seating arrangement and refreshment for primary teachers during 5 days’ training was not satisfactory. However training content was provided to the trainees during both trainers and primary teachers’ training. Similarly, trainers had a view that training rooms were equipped with required facilities at District QAEDs but these facilities were not available in training rooms at schools. Training rooms at schools were neither spacious nor suitable for activities, stated both trainers as well as teachers. Majority of trainers were satisfied with the refreshment provided to them during their training at District QAEDs. But both trainers as well as teachers showed their dissatisfaction over the provision of refreshment to the PSTs. Majority of the teachers approved that the trainer’s delivery was very effective and the content was well explained whereas some of the teachers said that content delivery was less effective because the trainers used English language in the class. However, they agreed that the trainers used appropriate strategies to monitor teachers’ performance during micro-teaching sessions, provided them feedback and kept guiding them for clarity of the difficult concepts. 

According to the quarterly monitoring reports, primary teachers expressed their full confidence in their respective trainers. They commented that the trainers were well planned and equipped to deliver sessions by involving all the trainees in a very interesting manner. According to British Council’s Annual Reports, the trainers’ delivery was assessed through training class observation on the five key performance indicators against four levels of Trainers’ Professional Development: a) induction, b) foundation, c) engagement, and d) integration. Expert Trainers’ skills to plan, manage and adapt activities improved significantly. The report shows that all Expert Trainers have achieved the ‘Engagement Level’. When Expert trainers were asked about their trainers’ delivery of content during training they received, they showed satisfaction.  Rather they were impressed by the training skills of the trainers. British Council’s trainers’ training and mentoring skills developed Expert Trainers’ training and mentoring skills. According to majority of trainers, content of training courses was aligned with the training outcomes and the content was properly graded according to level of primary school teachers. However, language of the course needed simplification. It was difficult for PSTs to comprehend language of the content. However, trainers explained the content using Urdu language.  In response to the same questions, all the teachers agreed that the content of training courses was well aligned with training outcomes. Similarly, majority of the teachers agreed that the language and course content was very easy and according to the level of PSTs whereas some teachers found training content and language difficult. They suggested that the training content should be more simplified. The content was reviewed to validate responses of the teachers. The content was well designed and developed. However, language of the content was difficult for senior primary teachers. 




Process Evaluation

British Council assessed teachers’ learning through self-reflection by teachers and class observation during training. Whereas some of the British Council’s team members, who observed teachers teaching in fifteen schools, showed satisfaction over the impact of the PEELI training. PEELI trained teachers engaged their students in classrooms. Moreover, the monitoring reports submitted by third party also presents robust data that showed positive change in teachers’ teaching methodology and classroom practices stated team members from the British Council. Thus according to British Council team members, feedback of training consultants who observed Expert Trainers conducting training and primary teachers’ self-reflection and micro-teaching sessions ensured quality of PEELI training.  In response to the same question, stakeholders from QAED, Punjab showed confidence on the monitoring reports and results of longitudinal study and found the reports a solid evidence of change in teaching practices in primary schools.  

Third party submitted suggestions/ recommendations in monitoring quarterly reports. QAED and British Council developed training content for primary teachers in year-II on the topics proposed by primary teachers mentioned in the reports. However, some of the recurrent suggestions presented in each report were totally ignored by QAED and British Council:

• Policy decision by SED regarding release of the teachers/ trainers for training 

• Annual PEELI Training Calendar

• Expert Trainers’ engagement with other activities/ trainings other than PEELI

• Coordination between DEA and District QAEDs

• Quality of training material and timely distribution at the training venue

• Selection of Training Centers

• Post training Follow up support 

• Primary school teachers offered the following recommendations: 

• ground realities need to be considered 

• training kits should be provided to all PSTs

• proper follow up system for PSTs




Product Evaluation

British Council team was asked about achievement of project outcomes. They expressed that there were two perspectives of looking at the achievement of the project outcomes; qualitative and quantitative. 

“Obviously we are behind achieving our target of training of 200,000 PSTs. Training was postponed in year-II. Even Expert Trainers’ training is pending”

[Team member, British Council] 

The delays in training de-motivated trainers as well as teachers.  In contrary to comments from British Council, team members from QAED, Punjab said that budget allocation for training component was not judicial. Insufficient budget caused delays in planning and implementation of the project. They, however, affirmed the qualitative effects of the project on teachers and trainers’ professional growth. 

British Council observed effects of PEELI training on professional development of Expert Trainers and active learning in trained PSTs’ class. Even the principals felt proud on the change in teachers’ teaching behavior and pedagogy. Primary teachers’ were very keen to join digital platform for sharing teaching practices and peer learning. A robust digital system is required to link all the primary teachers with each other and to develop a system for learning from each other. Moreover, British Council team supported this idea saying that there was a need to see how primary teachers moved one step ahead. The change started with level of awareness and change in teaching attitude. Now they were influencing each other by sharing their teaching practices.  

According to quarterly monitoring reports, PEELI initiated a longitudinal study in the first year of the project. Eight primary teachers from Kasur and Lahore were contacted fortnightly. The frequency of interaction was increased for follow ups and feedback to help them keep track of their professional growth. Analysis of reflective writings and peer observation and class observation by third party suggests that change in teaching methodology from teacher-centered to student-centered helped the teachers in attaining student learning outcomes effectively. Reflective practices and peer observation enabled them to enhance their teaching skills by learning from good practices and to eradicate the weak areas. Apart from change in teaching methodology, PEELI training developed some other positive changes i.e. understanding constructive feedback, active learning and group work. 

Almost all trainers claimed that training changed their teaching approach. They had no experience of teaching using activity-based and student-centered approach. They used new teaching strategies during training and later applied the same in their classrooms as teachers. As far as primary teachers are concerned, trainers observed teachers using activity-based approach during micro-teaching session. Some trainers expressed that they were in contact with their trainees who keep sharing their teaching experience and improvement in their performance. Some of the PSTs are used to share pictures of activities they performed in classroom with the trainers. However, majority of the trainers had a view that their passion and commitment to change disappears with the passage of time due to lack of follow up mechanism in education system. To ensure continuous improvement in PSTs’ improvement, continuous professional development of PSTs and follow up mechanism is suggested by most of the trainers and primary teachers.  Majority of primary school teachers agreed that the training improved results of their students whereas some of the teachers found no significant improvement in result of their students.

Findings

Findings based on context, input, process and product evaluation of PEELI project conclude that the project focused on change in teaching approach of the teachers. Training equipped the trainers with training skills i.e., critical thinking and problem solving, lesson planning and assessment of the students. Whereas training brought change in teaching approach of the primary teachers and improved their pedagogical skills. Physical and material resources utilized during trainings required improvement. However, training content and trainers’ delivery during sessions proved effective. The paper highlights some issues that need to be taken into account during further planning and implementing of training project i.e., policy decision regarding justified allocation of budget for training resources and release of teachers and trainers during training, training calendar, selection of training centers, coordination between District Education Authorities and District QAEDs and post training follow up support. Training changed teaching approach of the primary teaches as well as the trainers. They have started using child-centered approach and interactive learning in classroom. Continuous professional development of the primary teachers and follow up mechanism is recommended for successful implementation of the training project. Moreover, consistent departmental policy, systematic approach and proper internal and external monthly evaluation may lead to successful implementation of the project. 

References

• Chien, M., Lee, C., & Cheng, Y. (2007). The construction of Taiwan's educational indicator systems: Experiences and implications. Educational Research for Policy & Practice, 6(3), 249-259.
• Mohebbi, N., Akhlaghi, F., Yarmohammadian, M. H., & Khoshgam, M. (2011). Application of CIPP model for evaluating the medical records education course at master of science level at Iranian medical sciences universities. Procedia - Social and Behavioral Sciences, 15, 3286-3290. https://doi.org/10.1016/j.sbspro.2011.04.287.
• Msila, V. & Setlhako, A. (2013) Evaluation of Programs. Reading Carol H. Weiss. Universal Journal of Educational Research, 1(4), 323-327. https://doi.org/10.13189/ujer.2013.010408.
• Smith, C. L., & Freeman, R. L. (2002). Using continuous system level assessment to build school capacity. American Journal of Evaluation, 23(3), 307-319.
• Steinert, Y., Cruess, S., Cruess, R., & Snell, L. (2005). Faculty development for teaching and evaluating professionalism: From project design to curriculum change. Medical Education, 39(2), 127-136.
• Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, & applications. San Francisco, CA: Jossey-Bass.
• Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. Wiley (Ed.), The instructional use of learning objects. Retrieved from http:///reusability.org/read.
• Zhang, G., Griffith, R., Metcalf, D., Zeller, N., Misulis, K., Shea, D., & Williams, J. (2009). Assessing service and learning of a service-learning program in teacher education using mixed-methods research. Paper presented at the American Education Research Association Annual Conference, San Diego, CA.
• Zhang, G., Zeller, N., Shea, C., Griffith, R., Metcalf, D., Misulis, K., Williams, J., & Knight, S. (2008). A 360° assessment of the multidimensional effects of a service-learning program in teacher education using mixedmethods research. Paper presented at the 8th International Research Conference on Service-Learning and Community Engagement, New Orleans, LA.