August 3, 2022

Development and implementation of a formative pedagogical support program using the Inventory of pedagogical practices within the framework of a health professions program | BMC medical training

Program development

The school center for [BLINDED for REVIEW] to [BLINDED for REVIEW] was asked by the school administration to develop a standardized peer-observation process for faculty. While peer observations were already required at the School, observers used various instruments and criteria for observations. The following steps describe the process used to develop, implement and evaluate the IPC (Fig. 1). During each stage, decisions were approved by the school administration to ensure buy-in and support for the program.

Fig. 1

Process used to develop an instructional coaching program

Step 1: Establish a clear goal

A development team was established, consisting of four faculty of different streams (i.e., term, tenure-track) and ranks (i.e., assistant, associate, full ) with experience and expertise in teaching in several degree programs at the School. The first decision in developing the process was whether it should be formative, summative, or a combination of both. In other words, what were the objective and expected results of the program? The development team agreed that a formative process was needed to provide instructors with factual feedback on their teaching practices for professional growth and development. The team further agreed that combining formative and summative assessments in one program could overshadow the value of formative feedback and bias the assessment provided by observers. Therefore, the team created two separate processes: one solely dedicated to summative objectives and the other solely dedicated to formative objectives. The training process described below has been called the PCI.

Step 2: Develop a peer observation instrument

The ICP instrument was adapted from the Inventory of Teaching Practices (TPI), which was developed by Wieman and Gilbert to reduce the subjectivity typically associated with characterizing college teaching. 17 Each element of the TPI is evidence-based; in other words, each item was derived from research that demonstrated the extent to which different teaching practices were associated with student learning. 9 An item was assigned points based on the number of times the practice was used and the size of the related effect from the literature; for example, Question posed followed by small group discussion was scored “2” if the practice was used more than once and scored “0” if the practice was not used at all. 9.

Given the ability of the TPI to objectively and reliably characterize teaching practices, [9, 34,35,36] it was chosen as the basis of our coaching instrument. To focus the instrument on observable teaching practices in the classroom and enable the use of the instrument during instructional observation, we have identified 10 TPI elements that align with the school’s teaching philosophy. , which emphasizes learner-centered teaching pedagogies such as the flipped classroom model and problem-based learning [37]. Development of the instrument was led by three faculty members, who shared and endorsed the instrument with various stakeholders, including university leaders, course instructors, and education researchers. The elements of the TPI have been tested and evaluated, as described below. Feedback and edits were applied and the final ICP instrument contained the TPI elements as well as narrative feedback for the instructors, including strategies to keep, strategies to start, strategies to stop (i.e. .Keep, Start, Stop) and a summary with 1-3 priority evidence-based recommendations.

Step 3: Design a coaching program

The following program design has been established in the hopes of generating and providing feedback to instructors that promotes awareness and adoption of evidence-based teaching practices. Ideally, two to three observers (also called “coaches”) would be available for each observation, with one serving as the primary coach (eg, facilitating communication with the instructor, collecting and aggregating observation data). Prior to the observation, the instructor and coaches would review the observation process, identify the instructor’s needs/interests, and share any relevant materials. During the observation, coaches would arrive (eg, for in-person teaching) or connect virtually (eg, for online teaching) and sit towards the back of the room or turn off the video to minimize Distractions ; coaches would be instructed not to participate or intervene during an observation. Observations took place in a typical teaching setting. Following the observation, the coaches discussed their observations and created recommendations for the instructor. During a post-observation meeting with the instructor, coaches would provide feedback, recommendations, and relevant handouts with supporting documentation. At the end of the post-observation meeting, instructors would be encouraged to watch the recording of their class session and reflect on their teaching practices and the feedback provided. Since the ICP was designed to provide formative feedback, the results of the ICP would only be provided to the instructor, with the instructor permitted to share their own observation results at their discretion.

Step 4: Identify and Train Coaches

Based on the results of the instrument evaluation, it was determined that with proper training and a reliable observation instrument, any member of the School could serve as an observer (e.g., faculty, staff, trainees) for any classroom instructor. Participation as a coach was voluntary and was encouraged by recognizing effort as a service to the school during the annual review process. Coach training was a pre-training task, which involved watching a class recording and completing the observation form. During the 60-minute in-person training session, coaches discussed how they scored each item and offered suggestions for improving the instrument.

Step 5: Implement and evaluate the coaching program

Since the program was focused on growth and development, it was decided that participation would be voluntary for all instructors in the school. Program reminders were shared regularly via email and announcements at meetings. Each year, the center worked with the school administration to identify any additional recommendations for ICP participants. As described in more detail below, data from ICP observation instruments and participant surveys were collected for each session.

Step 6. Identify improvements

The key to the success and sustainability of any program is the ability to adapt and improve. Feedback was solicited from instructors and coaches with the goal of identifying opportunities for instrument and program improvement.

IPC Assessment

To develop and refine the ICP instrument, two pilot assessments were conducted. In the first round, 14 people filled out the instrument while watching a 50-minute taped biostatistics session in class. The instrument asked for frequency counts representing the number of times each teaching practice was observed. The researchers converted the frequency counts to true scores, according to the original TPI quantification scheme (Table 1). The total instrument score could range from 0 to 13. Based on the results and feedback from the first round, five items were revised with minor wording changes. In the second round, the revised instrument was provided to 11 new pilot observers, who performed the same video observation. Observer position (e.g., faculty, postdoctoral fellow, student, staff) was collected from all pilot observers to examine differences in scores based on education position. Convenience sampling was used to identify and recruit all pilot observers, and all agreed to participate. Overall inter-rater reliability was calculated using Krippendorff’s alpha [38] According to Krippendorff, [38] alpha levels above 0.67 are considered acceptable, with alpha levels above 0.80 considered ideal. To examine differences between groups, Mann Whitney U tests were applied for continuous items and chi-square tests for categorical items, with Fisher’s exact test used where necessary. Nonparametric tests were used due to small sample sizes.

Table 1 Modified second-round TPI questions and actual score values

To evaluate the ICP program, participating instructors were emailed an anonymous 3-item open-text survey regarding their experience with the program after each ICP session. Descriptive statistics were used to summarize the number of participating teachers, observations, coaches and time spent on ICP. Qualitative data collected through open-ended participant survey responses were thematically coded by a researcher. The results are presented as the median (range). This study was submitted to the institutional review board of the University of North Carolina and the study did not constitute research on human subjects as defined by United States federal regulations. [45CFR 46.102 (d or f) and 21 CFR 56.102(c)(e)(l)]. Verbal consent was obtained.