By: Kali M. VanLangen, PharmD, BCPS; Lisa A. Salvati, PharmD, BCACP
With the coronavirus (COVID-19) sweeping across the globe, many educators have embraced new delivery methods. We had previously hesitated to conduct virtual skills assessments due to the potential for technology-related issues, academic dishonesty, and losing face-to-face interaction. Despite these concerns, the pandemic forced a transition with only six days of preparation.
Developing the Virtual Teaching Objective Structured Clinical Examination (vTOSCE)
The TOSCE’s purpose was to assess pharmacy students’ verbal and written communication skills related to the Pharmacists’ Patient Care Process and provide feedback prior to the high stakes OSCE. The TOSCE involved 142 students, 19 faculty, and eight standardized patients, in five stations, three of which would be graded. In this unprecedented situation, we relied on familiar technology and past experiences with virtual interprofessional education events to design the vTOSCE.1,2
The two ungraded stations, originally planned to be timed and monitored by a P4 student, were modified to be self-monitored, and all patient case materials were posted in the learning management system. Two graded stations involving students interacting with standardized patients while a faculty grader observed, were modified to remove the standardized patient and be conducted synchronously via Blackboard Collaborate©. To accommodate potential technological issues, an additional minute was added to each station. All grading forms were deployed using ExamSoft©.
Sixteen Blackboard Collaborate© sessions were created with one faculty assigned to each session. Additionally, one faculty was assigned to grade SOAP notes; with another two faculty assisting with technology-related issues and monitoring the virtual sessions. Students were emailed links to the two separate Blackboard Collaborate© sessions they would need to enter, along with phone numbers in the event of technical problems. They were instructed to enter no more than two minutes prior to their start time to prevent one student overhearing another student’s feedback. If a student was unable to connect within the first two minutes of the scheduled interaction, they were rescheduled to the end.
vTOSCE Outcomes
A total of 141 students completed the vTOSCE. Most faculty only made one or two phone calls to students who were unable to connect. One faculty was unable to connect to Blackboard Collaborate© and needed to call all of their assigned students (n=17). Four students were unable to connect within the first two minutes and were rescheduled for later in the day.
Overall, results are similar between the Spring 2019 (S19) and Spring 2020 (S20) student cohorts (Table 1). Of note, different cases are used each year, but the grading rubric is the same.
Table 1: Average Score and Standard Deviation (SD) on each vTOSCE Station
| Patient Consultation (10 points) | Healthcare Provider (10 points) | |
| S19 TOSCE In-person with Standardized Patients (SP) | 8.06 (SD: 2.20) | 8.27 (SD: 1.92) |
| S20 vTOSCE Virtual with Faculty as SP | 8.86 (SD: 1.33) | 8.57 (SD: 1.58) |
Faculty and Student Feedback
Faculty provided positive feedback overall. The majority of faculty involved are located at least an hour from the TOSCE site and appreciated not traveling while still interacting with and providing feedback to students. The primary piece of constructive feedback was to provide additional time between students to allow for grading and connectivity issues.
Although a student evaluation of the event was not conducted all students with connectivity issues were appreciative of faculty responding to messages quickly. One student reached out and thanked the coordinator for the opportunity to still practice these skills and get valuable feedback from faculty to continue to improve.
Future of vTOSCE Assessments
As our healthcare environment continues to evolve, providing learners with telemedicine experience will be valuable.3 Literature on “teleOSCEs” in the medical education can provide a starting point for pharmacy educators.4,5 These reports indicate that teleOSCEs are a feasible and cost-effective way to assess future physicians on telemedicine skills that is generally accepted by students and able to accomplish the assessment objectives. The positive feedback and student performance results suggest vTOSCEs may become the new standard. However, these methods likely would not be useful for physical assessment and other hands-on skills.
Questions for the Academy
- What research is still needed to optimize the use of virtual skills assessments in pharmacy education? For example, what is the effect on faculty-student relationships?
- If virtual skills assessments are effective, under what circumstances should faculty still be present for face-to-face skills assessments?
- Surely we are not alone in this endeavor! What have you encountered while adapting your in-person skills assessments to virtual modalities?
References:
- Quesnelle KM, Bright DR, Salvati LA. Interprofessional education through a telehealth team based learning exercise focused on pharmacogenomics. Curr Pharm Teach Learn. 2018 Aug;10(8):1062-1069.
- McCutcheon LRM, Alzghari SK, Lee YR, Long WG, Marquez R. Interprofessional education and distance education: A review and appraisal of the current literature. Curr Pharm Teach Learn. 2017 Jul;9(4):729-736.
- DeJong C, Lucey CR, Dudley RA. Incorporating a new technology while doing no harm, virtually. JAMA 2015;314(22): 2351– 2.
- Palmer RT, Biagioli FE, Mujcic J, Schneider BN, Spires L, Dodson LG. The feasibility and acceptability of administering a telemedicine objective structured clinical exam as a solution for providing equivalent education to remote and rural learners. Rural Remote Health. 2015 Oct-Dec;15(4):3399. Epub 2015 Dec 3.
- Cantone RE, Palmer R, Dodson LG, Biagioli FE. Insomnia Telemedicine OSCE (TeleOSCE): A Simulated Standardized Patient Video-Visit Case for Clerkship Students. MedEdPORTAL. 2019 Dec 27;15:10867.
Kali M. VanLangen is a Professor of Pharmacy Practice at Ferris State University College of Pharmacy. Her research interests include APPE readiness, Entrustrable Professional Activities, and the use of electronic health records in the laboratory setting. In her free time, Kali enjoys spending time outdoors with her husband and two daughters.
Lisa A. Salvati is an Associate Professor of Pharmacy Practice and the Interprofessional Education (IPE) Coordinator at Ferris State University College of Pharmacy. Her research interests include topics related to IPE especially assessment of IPE. Outside of the workplace, Lisa likes cooking, exercising, and spending time with her family including her husband and two daughters.
Pulses is a scholarly blog supported by a team of pharmacy education scholars
Pulses 
This was a very nice posting that detailed how Drs.VanLangan and Salvati conducted a vTOSCE at their institution. APPE-readiness of P3 students is an imperative before their P4 APPEs. We did similar at my institution, with some technology very similar (e.g., using Collaborate Ultra, though wished we had used ExamSoft for scoring as well).
We also had two performance assessments involving faculty interaction, although these are part of a larger number of OSCE stations. As Dr.Sturpe had noted a time ago,1 pharmacy educators may call a performance assessment an OSCE, however a rigorous (reliable) OSCE needs many stations. In fact, I remain skeptical even with the number of OSCE stations used this year at my institution it will not be very rigorous. That said, the decisions made with it need to be softened, as the quality of this OSCE evidence is softer. From my psychometric background, I am also worried with Drs.VanLangan/Salvati and my own institution’s ability to statistically-discriminate among more and less able students (i.e., reliability of these OSCEs). Drs.VanLangan/Salvati reported narrower SD for this year’s version, and (anecdotally) I understand that my institution had few students that did not pass their first attempt. Neither scenario inspires confidence for me.
Furthermore, I would encourage the authors to use consistent and helpful acronyms. To my awareness, a team-based OSCE has been labeled a TOSCE,2 while a teaching OSCE has been labeled an OSTE.3 This blog description was neither of those. In fact, I am confused by the label therein of calling this a teaching OSCE (as the prior article3 described scoring and evaluation of an instructor’s teaching performance). However, I do like when vTOSCE was used, as this was a new acronym to me, and allowed me to review and better understand the novelty of this educational activity description. My suggestion would be to simply call this a vOSCE, or VPBA (virtual performance-based assessment) given the very small number of vOSCE stations.
Overall, let me reiterate that I greatly appreciated reading this blog and greatly thank Drs.VanLandan and Salvati for their efforts in detailing it here. It is always very encouraging to hear other experiences and to know that my own institution’s recent effort was not entirely unique, given our current COVID-related situation.
Reference:
1. Sturpe DA. Objective structured clinical examinations in doctor of pharmacy programs in the United States. Am J Pharm Educ. 2010;74(8): article 148.
2. Singleton A, Smith F, Harris T, Ross‐Harper R, Hilton S. An evaluation of the team objective structured clinical examination (TOSCE). Med Educ. 1999;33(1):34-41.
3. Sturpe DA, Schaivone KA. A primer for objective structured teaching exercises. Am J Pharm Educ. 2014;78(5): article 104.
LikeLike