+ H O S T E D BY
Available online at www.sciencedirect.com
ScienceDirect Journal of Current Ophthalmology xx (2016) 1e2 http://www.journals.elsevier.com/journal-of-current-ophthalmology
Resident competence assessment: Best practices
Keywords: Competence; Assessment; ICO-OSCAR; Ophthalmology resident
Competence can be defined as “the ability to do something well.” The goal of ophthalmology residency training is to produce competent ophthalmologists. To prove competence, appropriate assessment methods are required. These methods should follow certain principles of assessment.1 Teaching ophthalmologist's time is valuable and limited and often poorly or uncompensated. Thus any assessment should be as timeefficient as possible but still provide valid and reliable information. However, the easiest and fastest assessment methods (written tests) are not appropriate for assessment of skill. Additionally, performance is situation specific. Good performance on medical knowledge multiple choice test does not mean there will also be good performance in procedural skills or examination techniques. Thus situation specific assessments must be used. Assessment methods should be known by the residents and aligned with objectives and performance expectations. Assessments can also be a learning experience. This may not always be possible if the assessment is an examination that leads to passing or failing a course, rotation or residency. This type of assessment is known as summative feedback. On the contrary, formative feedback is assessment that is designed to improve performance and thus must be shared with the resident. The vast majority of assessments should be formative. In recent years a variety of ophthalmology-specific competency assessment tools have been developed. The International Council of Ophthalmology (ICO) offers the only internationally developed test of ophthalmic medical knowledge. The ICO examinations include both basic and advanced levels.2 The Ophthalmic Clinical Evaluation Exercise (OCEX) is designed to assess resident competence in evaluating a new patient. The teaching physician completes scoring in 33 categories that rate the residents ability to communicate effectively, perform a history and examination, and synthesize the information into a differential diagnosis and
Peer review under responsibility of the Iranian Society of Ophthalmology.
plan. The OCEX has been shown to have content validity and inter-rater reliability.3,4 Several methods of surgical skill assessment have been devised. Cremers and associates developed the “Objective Assessment of Skills in Intraocular Surgery” (OASIS), a onepage objective evaluation form to assess residents' skills in cataract surgery.5 The form is completed by an evaluator who directly observes the surgical procedure and includes objective data such as wound placement and size, phacoemulsification time, and total surgical time, etc. To complement this objective assessment the same group developed a subjective rating of surgical skills named “Global Rating Assessment of Skills in Intraocular Surgery” (GRASIS).6 This one-page form allows the evaluator to assign scores from 1 to 5 based on a behaviorally anchored rubric to domains such as pre-operative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Saleh and colleagues described an assessment tool called the “Objective Structured Assessment of Cataract Surgical Skill” (OSACSS).7 This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5-point Likert scale. The ICO modified this tool to include a modified Dreyfus model of skill acquisition (novice, beginner, advanced beginner, competent) and description of the skill required to achieve each level in each step of the surgical procedure.8 Once drafted, content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback. The product is the internationally valid and reliable ICO-Ophthalmology Surgical Competency Assessment Rubric (OSCAR) e phacoemulsification.8,9 In a similar fashion internationally applicable assessment tools for extracapsular cataract surgery (ICO-OSCAR:ECCE),8 small incision cataract surgery (ICO-OSCAR:SICS),10 lateral tarsal strip surgery (ICOOSCAR:LTS),11 and strabismus surgery (ICO-OSCAR:strabismus)12 were developed. Additional ICO-OSCARS are being developed for panretinal photocoagulation, corneal transplant, vitrectomy, trabeculectomy and pediatric cataract surgery.
http://dx.doi.org/10.1016/j.joco.2016.04.003 2452-2325/Copyright © 2016, Iranian Society of Ophthalmology. Production and hosting by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Please cite this article in press as: Golnik KC, Resident competence assessment: Best practices, Journal of Current Ophthalmology (2016), http://dx.doi.org/ 10.1016/j.joco.2016.04.003
Editorial / Journal of Current Ophthalmology xx (2016) 1e2
Medical educators have also been devising methods to teach and assess the competencies other than patient care and medical knowledge. Lee and associates described an assessment tool involving a structured assessment of journal club that leads to assessment of practice based learning and improvement.13,14 Golnik and associates developed and showed validity of a tool to assess resident on-call performance, the “On Call Assessment Tool” (OCAT).15 The OCAT is a one page checklist to be used retrospectively during random chart review of on-call consultations. Finally, a “360 ” evaluation tool can be very helpful in evaluating professionalism and communication skills. The intent is to get feedback on residents from all groups with which they interact. Thus, faculty evaluations can be considered 90 , patient surveys, ancillary staff and peer evaluations constitute the other 270 . Self-evaluations may be utilized in addition. Of course, patients and staff are not asked about the resident's medical knowledge but rather about professionalism and communication skills. Jagadeesan and associates have shown their patient satisfaction survey can discriminate levels of resident communication skill and thus may be useful to assess this competency.16 No internationally valid 360-degree assessment tool exists and thus the ICO is currently developing one in a manner similar to the ICO-OSCARs. In this issue Hassanpour and associates compared video observation of procedural skills (VOPS) to direct observation of procedural skills (DOPS) using a 10-point Likert scale. They also investigated intra-rater reliability. They found good correlation between VOPS and DOPS scores and very good intra-rater reliability. The VOPS method has the advantage of completion at a later time and may be more convenient. It also could lead to multiple assessors, and studies of inter-rater reliability should be considered. It would be interesting to apply the soon to be published ICO-OSCAR for trabeculectomy in this paradigm. Both the VOPS and DOPS methods meet the principles of good assessment described above. Utilization of tools such as these are essential in producing and assuring competence of our future ophthalmologists. References 1. Williams Ten Commandments of Principle Assessment. Available at: http://www.facs.org/education/rap/williams.html; Accessed 10.04.16.
2. The ICO examinations. Available at: http://icoph.org; Accessed April 10, 2016. 3. Golnik KC, Goldenhar LM, Gittinger Jr JW, Lustbader JM. The ophthalmic clinical evaluation exercise (OCEX). Ophthalmology. 2004;111:1271e1274. 4. Golnik KC, Goldenhar L. The ophthalmic clinical evaluation exercise (OCEX): interrater reliability determination. Ophthalmology. 2005;112:1649e1654. 5. Cremers SL, Ciolino JB, Ferrufino-Ponce ZK, Henderson BA. Objective assessment of skills in intraocular surgery. Ophthalmology. 2005;112:1236e1241. 6. Cremers SL, Lora AN, Ferrufino-Ponce ZK. Global rating assessment of skills in intraocular surgery. Ophthalmology. 2005;112:1655e1660. 7. Saleh GM, Gauba V, Mitra A, et al. Objective structured assessment of cataract surgical skill. Arch Ophthalmol. 2007;125:363e366. 8. Golnik KC, Beaver H, Gauba V, et al. Cataract surgical skill assessment. Ophthalmology. 2011;118, 427. E5. 9. Golnik KC, Beaver H, Gauba V, et al. Development of a new valid, reliable, and internationally applicable assessment tool of residents' competence in ophthalmic surgery (An American Ophthalmological Society Thesis). Trans Am Ophthalmol. 2013;111:24e33. 10. Golnik KC, Haripriya A, Beaver H, et al. Cataract surgical skill assessment. Ophthalmology. 2011;118:2094.e2. 11. Golnik KC, Gauba V, Saleh GM, et al. The ophthalmology surgical competency assessment rubric for lateral tarsal strip surgery. Ophthal Plast Reconstr Surg. 2012;28:350e354. 12. Golnik K, Motley WW, Atilla H, et al. The ophthalmology surgical competency rubric for strabismus surgery. J AAPOS. 2012;16:318e321. 13. Lee AG. Using the American Journal of Ophthalmology's website for assessing residency subcompetencies in practice-based learning. Am J Ophthalmol. 2004;137:206e207. 14. Lee AG, Boldt CH, Golnik KC, et al. Using the journal club to teach and assess competence in practice-based learning and improvement: a literature review and recommendation for implementation. Surv Ophthalmol. 2005;50:542e548. 15. Golnik KC, Lee AG, Carter K. Assessment of ophthalmology resident oncall performance. Ophthalmology. 2005;112:1242e1246. 16. Jagadeesan R, Kaylan DN, Lee P, Stinnet S, Challa P. Use of a standardized patient satisfaction questionnaire to assess the quality of care provided by ophthalmology residents. Ophthalmology. 2008;115:738e743.
Karl C. Golnik, MD, MEd Department of Ophthalmology, University of Cincinnati and the Cincinnati Eye Institute, Cincinnati, OH, USA E-mail address: [email protected]
10 April 2016
Please cite this article in press as: Golnik KC, Resident competence assessment: Best practices, Journal of Current Ophthalmology (2016), http://dx.doi.org/ 10.1016/j.joco.2016.04.003