Considerations for Designing Rubrics with Artificial Intelligence in Pharmacy Education

Add a comment

By: Cherie Lucas, PhD, BPharm, GradCertEduStu (Higher Ed), FPS, SFHEA; Mark Ian Jones, PhD; Helena Pacitti, PhD, BPsych (Hons.); Chris Campbell, EdD, PFHEA; Jeff Cain, EdD, MS 

Assessment in pharmacy education is pivotal for developing practice-ready graduates. Rubrics are central in ensuring assessments are transparent, consistent, and aligned with learning outcomes.1 They provide structured criteria for assessing student performance, offering transparency and actionable self-directed reflective feedback.2 They also articulate expectations for students to work towards.3 In pharmacy education, rubrics clarify expectations and guide students toward competency in areas such as clinical reasoning and ethical decision-making.1

As beneficial as they may be, many pharmacy educators are not trained in creating rubrics, making the task challenging and time-consuming. However, with generative artificial intelligence (GenAI), rubric creation can be more efficient and tailored.

Key Components of Rubrics

Key components of rubrics include:

  • Criteria: Indicators of performance, so there is an understanding of what constitutes quality work (e.g., analytical depth, communication clarity)
  • Quality Definitions: Descriptions of performance levels (e.g., high distinction, distinction, credit,  pass, fail)
  • Scoring Strategy: Metrics to quantify performance. With guidance, AI can assist with each of these components.

How can AI benefit in rubric design?

  • Efficiency: GenAI can draft rubrics quickly, allowing the educator time to refine content and align it with the Course Learning Outcomes (CLOs).
  • Comprehensiveness: GenAI prompts may sometimes identify areas that have been overlooked.  So, in a sense, the GenAI tool becomes an online 24/7 collegial friend. This means that while ideation is part of the process, verification is the critical thinking by the educator.
  • Customization: For example, prompts can be tailored to address diverse student cohort needs including those of neurodiverse backgrounds.

How to get started? 4

  1. Consider the context for the assessment. What is the assessment brief (details of the assessment task) provided to the student? 
  2. Course Learning Outcomes (CLOs): Which CLOs are aligned to that assessment task?
  3. Consider utilizing Bloom’s Taxonomy, Bloom’s Revised Taxonomy cognitive verbs, or Fink’s Taxonomy of Significant Learning and embed them in the relevant sections of the rubric. For example, if you require higher-order thinking, then adjust the rubric to include “create” (e.g. compose, design, construct, investigate) or “evaluate” (e.g. appraise, argue, defend, judge, rank), which are both derived from the revised taxonomy list of cognitive verbs. 
  4. Consider the weighting for each section of the rubric criteria (e.g., perhaps place a higher weight on the processes for critical thinking) and include relevant, appropriate citations.
  5. Prompting AI: Begin with context, alignment of CLOs, and questions. For example: “Create a rubric for a pharmacy ethics essay derived from their clinical placement experience for a diverse student cohort, emphasizing critical analysis and ethical reasoning.” Keep refining with each iteration of output. When using AI, it is often best to develop and design rubrics in an iterative fashion, embedding one prompt at a time and refining outputs as you go.
  6. Peer Review: Ensures oversight, robustness of the process and an effective way to benchmark. 
  7. Acknowledgment: Educators should model their acknowledgment of AI use for the design of the rubric showing transparency. For example, the following could be mentioned: “This Rubric was co-designed by (convenor name) and with (GenAI tool name) and peer-reviewed by (academic name(s)). 

Table Exemplar Case Prompts

Prompt 1. Provide your role. Ask your AI agent to produce a rubric for a [first] year [under] graduate course with a diverse [neurodivergent mix] student cohort of [domestic/ non-domestic, %neurodiverse, etc.] based on the CLOs addressed in the assessment task. Be as specific as you can. 
Prompt 2. Enter the assessment task description (and steps, etc) from the course outline or assessment brief. Ask for a range of grades between a High Distinction to a Fail.  
At this stage, the AI tool (AI agent) should produce a first draft rubric for your consideration. Review and ascertain if you need to refine any of your prompts. You may find that some of the criteria require tweaking, so ask your AI agent to revise. An example may be: 
Prompt 3. Please revise the criteria for the process to include reflection [or other cognitive verbs such as those derived from a taxonomy]. 
Prompt 4. For concise information- you may like to prompt, “Now make this SPARTAN.”

Table contents adapted from Lucas C et al (2024) Ethical Use of GenAI for Designing Rubrics at UNSW:   Nexus Fellows’ Perspectives.

Now that you have the basic tools to design your own rubrics, it is important to maintain ethical oversight to ensure academic integrity. 

How are you, as a faculty member, using GenAI for rubric design?  

REFERENCES 

1. Lucas C, Smith L, Lonie JM, Hough M, Rogers K, Mantzourani E. Can a reflective rubric be applied consistently with raters globally? A study across three countries. Curr Pharm Teach Learn. 2019;11(10):987-994. https://doi.org/10.1016/j.cptl.2019.06.004

2. Popham, WJ. What’s wrong- and what’s right-with rubrics. Educational Leadership, 1997; (55): 72-75

3. Brookhart SM. Appropriate Criteria: Key to Effective Rubrics. Frontiers in education (Lausanne). 2018;3.

4. Lucas C, Jones M, Campbell C, Pacitti H. Ethical Use of Gen-AI for Designing Rubrics at UNSW: Nexus Fellows’ Perspectives.  https://www.education.unsw.edu.au/news-events/news/ethical-use-gen-ai-designing-rubrics.

Author Bios: 

Dr Cherie Lucas, PhD, BPharm, Grad Cert Edu Stud (Higher Educ), FPS, SFHEA, is the Nexus Fellow (educational leadership change agent) at the School of Population Health, Faculty of Medicine and Health at the University of NSW, Sydney, Australia. She also holds an adjunct position at the University of Technology Sydney (UTS) with the Faculty of Health, UTS.  Dr Lucas is a licensed pharmacist and currently resides on the Pharmacy Council of NSW as the Deputy President of Australia. She has been awarded numerous teaching awards and recognition including being the recipient of an Australian Award for University Teaching (AAUT) Citation for the individual category.

Her educational research interests include curriculum design and instruction, developing reflective practice tools and assessment strategies to enhance student engagement, experience, and learning, and building artificial intelligence (AI) capability for educators and students. She has led interprofessional education and reflective practice teams and led the development of a  Course Convenor AI Toolkit as an academic resource. 

Dr. Mark Ian Jones is a UNSW Nexus Fellow, design historian, and Chartered Architect at the University of New South Wales Faculty of Arts, Design & Architecture (ADA), School of Art and Design. He has extensive experience in systemic approaches to curriculum design and innovation, rubric design, and the affordances of AI in the assessment and assurance of learning. Mark has extensive academic leadership experience at UNSW, including Nexus Fellow, ADA Director Education Innovation, Deputy Head of School (Design), and Program Director Bachelor of Design.  

Dr. Helena Pacitti, PhD, BPsych (Hons.) is the Nexus Fellow at the School of Psychology, Faculty of Science at the University of NSW, Sydney, Australia. 

Dr. Pacitti is an award-winning, Education-Focused Lecturer with over a decade of teaching experience. Her evidence-based teaching approach focuses on developing Self-Regulated Learning skills in undergraduates, drawing on cognitive and behavioral psychology research. Her course design innovations have improved student engagement and academic performance, while her research collaborations advance the Scholarship of Teaching and Learning through interdisciplinary partnerships. She creates inclusive learning environments that empower students’ academic success.

Dr. Chris Campbell, PFHEA, is the Nexus Fellow at UNSW Canberra, working across the city and Australian Defence Force Academy (ADFA) campuses in Australia. Chris engages with staff in innovative learning and teaching and has projects on assessment best practices and staff and student wellbeing. Since beginning at UNSW, Chris has been working with colleagues across the university on designing rubrics using AI. Her main research area is educational technologies to improve student learning and engagement, particularly online learning. As a mid-career researcher, Chris has a substantive research track record and expertise in digital technologies and pedagogies and in evaluating new and emerging technologies in the online space. Chris was the President of ASCILITE from 2019-2023, on the Executive for nine years, and became a Life Member in 2023. 

Jeff Cain, EdD, MS, is a Professor and Vice-chair in the Department of Pharmacy Practice & Science at the University of Kentucky College of Pharmacy. Jeff’s educational scholarship interests include innovative teaching, digital media, and contemporary issues in higher education. In his free time, he is a dad to a pole-vaulting daughter, an extreme trail ultramarathoner, and is president of For Those Who Would, a 501(c)(3) charity in the adventure and endurance racing communities. 

Pulses is a scholarly blog supported by a team of pharmacy education scholars

Leave a comment