Validity and Reliability of SCOPE (Structured Comprehensive Oral Problem-based Examination) using Generalizability and Decision Study
DOI:
https://doi.org/10.12669/pjms.42.3.13939Keywords:
ASSESSMENT, Assessment tool, Education, Medical, Oral examination, Generalizability study, Health professions education, Medical education, Structured vivaAbstract
Objective: To validate the SCOPE tool, determine its reliability, and identify sources of score variation using Generalizability theory, including a Decision Study to optimize its structure.
Methodology: The study was conducted at Islamic International Medical College (IIMC), Rawalpindi from March 2025 to May 2025. SCOPE (Structured Comprehensive Oral Problem-based Examination) marking sheet and CVI (Content Validity Index) form were reviewed by medical educationalists for feedback and validation. Reliability was assessed through a pilot involving 37 final-year medical students. Four trained examiners conducted SCOPE, with each student assessed by one examiner using problems derived from predefined must-know and good-to-know topic lists. Performance was scored using a structured marking sheet for Recall (five marks) and Application (20 marks) categories. Reliability was assessed using Generalizability study (G-Study) with a crossover random-effects design, followed by a Decision Study (D-Study) to estimate projected reliability coefficients across varying numbers of problems and categories.
Results: Ten medical educationalists provided qualitative feedback and rated the relevance and clarity of SCOPE sheet, demonstrating strong content validity (S-CVI/Ave = 0.92) and clarity (average CCA = 2.85). Reliability analysis showed a G-Coefficient value of 0.793, and Phi-coefficient value of 0.696, indicating good reliability for ranking students, and moderate reliability for absolute decisions. Students contributed the largest variance (48.13%), followed by student × problem × category interaction (19.8%). D-Study revealed that increasing the number of problems and categories substantially improved both relative and absolute reliability.
Conclusion: SCOPE is a feasible, reliable and innovative tool for comparative assessments in low-resource settings, with reliability further enhanced by increasing problem numbers for each student.
