| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
ORIGINAL RESEARCH |
From the Division of Urogynecology, Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, Good Samaritan Hospital, Cincinnati, Ohio.
 |
ABSTRACT |
|---|
TOP
 ABSTRACT MATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
METHODS: We measured baseline surgical ability and clinical knowledge of 26 residents (postgraduate year [PGY]-1 to PGY-4) using the Objective Structured Assessment of Technical Skills (OSATS) and a written examination. The OSATS consists of a global surgical skills assessment (OSATS-G), a procedure checklist (OSAT-C), and pass/fail grade. Five weeks after our baseline evaluation, a 1.5-hour workshop was administered to approximately half of the 26 residents (n=14). One week after this intervention, the residents were re-examined using the same assessment tools.
RESULTS: The OSATS demonstrated construct validity as scores on the examination increased on both the OSATS-G and the OSATS-C from PGY-1 through PGY-4 (P=.001 and P=.041, respectively). Reliability indices for the OSATS were high. Eighty-one percent of the residents failed the OSATS before intervention because of failure to identify and repair the internal anal sphincter. After educational intervention, senior residents improved on all assessments (OSATS-G, P=.041; OSATS-C, P=.004; written examination, P=.008), and all residents passed the OSATS.
CONCLUSION: A valid and reliable OSATS and written examination were developed to assess surgical skills, knowledge, and judgment necessary to properly manage fourth-degree perineal lacerations. Residents improved on the OSATS and the written examination after undergoing a structured educational workshop.
LEVEL OF EVIDENCE: II
In-training (eg, Council on Resident Education in Obstetrics and Gynecology) as well as written and oral board examinations assess cognitive knowledge and clinical judgment. Technical proficiency, however, is not easily evaluated. Case logs and faculty evaluations are imprecise and unreliable methods of evaluating physician competence. Unfortunately, most resident training programs rely on these subjective faculty assessments.8,9
The Objective Structured Assessment of Technical Skills (OSATS) is a valid and reliable method of measuring surgical skills, and it has been validated for different types of procedures in both general surgery and obstetrics and gynecology. Early in its development, it was demonstrated that OSATS measures using bench models is equivalent to those using live animals.11 Advantages of inanimate anatomic models include availability, lower cost, and portability, and use of the models obviates the ethical issues involved in the use of live animals.
The purpose of this study was to develop a valid and reliable OSATS to measure surgical skill necessary for proper repair of fourth-degree perineal laceration. We hypothesize that technical skills, knowledge, and judgment pertinent to these repairs will improve after a structured educational workshop.
|
MATERIALS AND METHODS |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
|
The residents arrived in the testing room, which contained the fourth-degree laceration anatomic replica, an adjustable light, and appropriate and multiple distracter sutures and instruments. The same clinical scenario was read to all of the residents, and then standardized instructions were given. The examiners acted as surgical assistants, did whatever was asked of them, but did not give input or assistance unless they were asked. The extent of the obstetric laceration was not stated. Four evaluators completed the OSATS score sheets for each resident.
Twenty-six obstetrics and gynecology residents from TriHealth Hospitals in Cincinnati, Ohio, participated in baseline assessment and reassessment using OSATS. Testing and retesting were separated by 6 weeks because each session lasted more than 6 hours (7 am to 1:30 pm) and interfered with clinical duties. Therefore, the chairman of the department permitted testing only during the monthly resident education day set aside for the Division of Urogynecology and Reconstructive Pelvic Surgery. In this study, we were interested in studying the short-term effects of a workshop. Thus, our educational intervention was scheduled exactly 1 week before the retesting date. Residents were re-examined with the same assessment tools. The intervention group, which consisted of about half (n=14) of the residents from each level of training, underwent a 1.5-hour workshop (Table 2). Residents at Trihealth residency program are evenly split between two hospitals that are geographically separated. To minimize interaction between the intervention and control groups, those residents doing their rotation at Good Samaritan Hospital during our study period were assigned to the intervention group.
|
Residents repaired fourth-degree laceration on a commercial anatomic replica, the Sultan Anal Sphincter (Limbs and Things Inc, Bristol, UK) (Fig. 1). This model comes with a replaceable perineal pad that allows approximately 24 repairs before replacement is necessary. Each model (skin, 1 perineal pad, base, and clamps) costs approximately $642.94. Each additional replacement perineal pad costs $109.22.
|
This study was reviewed and approved by the Good Samaritan Hospital Institutional Review Board and funded by the Hatton Institute for Research and Education. The four evaluators underwent a 30-minute examiner orientation before the study was begun. They also participated in a pilot study to demonstrate high interrater reliability on the OSATS-C and OSATS-G.
Statistical analyses were performed with SPSS 14.0 for Windows (SPSS Inc, Chicago, IL). The consistency of our tests was measured by test-retest, interrater, and internal consistency reliability (Cronbach 
) indices. Construct validity is a proxy for validity or the extent to which a test measures what it is intended to measure. Because of ordinal measures and small group–specific sample sizes, Kruskal-Wallis and Dunn tests were used to establish construct validity. Fischer exact and t tests were used to establish similarity between the control group and the intervention group. Spearman 
correlation coefficients were used to measure correlation between OSATS-G, OSATS-C, and written examination scores. P<.05 was considered statistically significant.
|
RESULTS |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
|
|
|
The control group and the intervention group did not differ in demographic characteristics. Both groups also had similar baseline composite scores on the OSATS (Table 6). The intervention group performed better than the control group on the OSATS-G, OSATS-C, and written examination (Table 7). Additionally, the PGY-2, PGY-3, PGY-4 residents improved in at least one of the assessments. The senior residents improved on all assessments after the intervention (Table 8). All residents passed the OSATS after intervention.
|
|
|
Table 9 shows the Spearman correlation between the different assessment techniques. There was moderate correlation between OSATS-G and OSATS-C before intervention. After intervention, both OSATS-C versus with OSATS-G and OSATS-C versus with written examination were strongly correlated.
|
|
DISCUSSION |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
Correlations between the OSATS-G and OSATS-C and between the OSATS-C and written examination were high after our workshop. This suggests that a written examination by itself, which is easier to administer and less time-consuming, may suffice as a technical assessment tool if administered after a highly structured workshop.
Approximately 81% of all residents, including 35% of senior residents, failed the OSATS because they did not recognize the existence of internal anal sphincter and made no attempt to repair it. We doubt that this occurred because of the inadequacies of our commercial anatomic replica because the internal anal sphincter layer is obvious and prominent (Fig. 2). To highlight the gravity of this issue, our residents are above the national average for the number of vaginal deliveries, instrument deliveries, and third- and fourth-degree laceration repairs. In 2005, one of the two TriHealth Hospitals had 4,401 vaginal deliveries, and 11% of these were forceps deliveries while 6% were vacuum deliveries. Residents at this hospital were exposed to 461 third-degree and 159 fourth-degree lacerations. Since the internal anal sphincter makes up 70–80% of the resting tone of the anal sphincter, it is very important to recognize and fix this structure properly. Thus, a nationwide educational effort is needed to address this problem.
|
The limitations of this study are as follows. First, because of the need to minimize interaction between the two groups, the intervention was not assigned by an accepted randomized method such as computer generation or random digit table. However, we doubt that this was a source of bias since residents are not assigned to their rotations in any systematic fashion. Additionally, baseline OSATS scores for the two groups were highly comparable. Another limitation is that sample size was small and residents were trained and tested by familiar faculty.
We did not study the diminution of testing effects with time or the long-term effects of our intervention. We also did not study the effects of varying our educational intervention. For example, one may study the effects of different types of intervention (didactic versus video versus self-practice).
In conclusion, we developed a valid assessment tool that was able to reliably quantify surgical skill. We were able to objectively measure improvement in surgical ability after an educational intervention.
|
Footnotes |
|---|
Corresponding author: Sam Siddighi, MD, 1138 Carney St. Unit #3, Cincinnati, OH 45202; e-mail: sam_siddighi{at}trihealth.com.
doi:10.1097/01.AOG.0000251499.18394.9f
|
REFERENCES |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
2. Morren Gl, Hallbook O, Nystrom PO, Baeten CG, Sjodahl R. Audit of anal-sphincter repair. Colorectal Dis 2001;3:17–22.[Medline]
3. Walsh CJ, Mooney EF, Upton GJ, Motson RW. Incidence of third-degree perineal tears in labour and outcome after primary repair. Br J Surg 1996;83:218–21.[Medline]
4. Karoui S, Leroi AM, Koning E, Menard JF, Michot F, Denis P. Results of sphincteroplasty in 86 patients with anal incontinence. Dis Colon Rectum 2000;43:813–20.[Medline]
5. Pinta TM, Kylanpaa ML, Salmi TK, Teramo KA, Luukkonen PS. Primary sphincter repair: are the results of the operation good enough? Dis Colon Recum 2004;47:18–23.
6. Sultan AH, Hudson CN. Are junior doctors and midwives adequately prepared to repair the perineum? J Obstet Gynaecol 1993;13:484–5.
7. Blanchard MH, Amini SB, Frank TM. Impact of work hour restrictions on resident case experience in an obstetrics and gynecology residency program. Am J Obstet Gynecol 2004;191:1746–51.[Medline]
8. Streiner Dl. Global rating scales. In: Neufeld VR, Norman GR, editors. Assessing clinical competence. New York (NY): Springer; 1985. p. 119–41.
9. Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165:358–61.[Medline]
10. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a structured technical skills assessment form. Am J Surg 1994;167:423–7.[Medline]
11. Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, et al. Objective Structured Assessment of Technical Skills (OSATS) for surgical residents. Br J Surg 1997;84:273–8.[Medline]
12. Mandel LS, Lentz GM, Goff BA. Teaching and evaluating surgical skills. Obstet Gynecol 2000;95:783–5.
13. Goff BA, Lentz GM, Lee D, Fenner D, Morris J, Mandel LS. Development of an objective structured assessment of technical skills for obstetrics and gynecology residents. Obstet Gynecol 2000;96:146–50.
14. Nielsen PE, Foglia LM, Mandel LS, Chow GE. Objective structured assessment of technical skills for episiotomy repairs. Am J Obstet Gynecol 2003;189:1257–60.[Medline]
15. VanBlaricom AL, Gogg BA, Chinn M, Icasiano MM, Nielsen P, Mandel L. A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS). Am J Obstet Gyencol 2005;193:1856–65.
16. Cunningham FG, Leveno KJ, Bloom SL, Hauth JC, Gilstrap III LC, Wenstrom KD. Williams obstetrics. 22nd ed. New York (NY): McGraw-Hill; 2005. p. 435–8.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
