Browsing by Subject "Educational Measurement"
Now showing 1 - 10 of 10
Results Per Page
Sort Options
Item Open Access A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation.(Int J Med Educ, 2016-01-11) Bradley, Kendall E; Andolsek, Kathryn MOBJECTIVE: To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education. METHODS: In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses. RESULTS: Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year. CONCLUSIONS: Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.Item Open Access Assessing intraoperative judgment using script concordance testing through the gynecology continuum of practice.(Med Teach, 2014-08) Kow, Nathan; Walters, Mark D; Karram, Mickey M; Sarsotti, Carlos J; Jelovsek, J EricOBJECTIVE: To measure surgical judgment across the Obstetrics and Gynecology (OBGYN) continuum of practice and identify factors that correlate with improved surgical judgment. METHODS: A 45-item written examination was developed using script concordance theory, which compares an examinee's responses to a series of "ill-defined" surgical scenarios to a reference panel of experts. The examination was administered to OBGYN residents, Female Pelvic Medicine and Reconstructive Surgery (FPMRS) fellows, practicing OBGYN physicians and FPMRS experts. Surgical judgment was evaluated by comparing scores against the experts. Factors related to surgical experience were measured for association with scores. RESULTS: In total, 147 participants including 11 residents, 37 fellows, 88 practicing physicians and 11 experts completed the 45-item examination. Mean scores for practicing physicians (65.2 ± 7.4) were similar to residents (67.2 ± 7.1), and worse than fellows (72.6 ± 4.2, p < 0.001) and experts (80 ± 5, p < 0.001). Positive correlations between scores and surgical experience included: annual number of vaginal hysterectomies (r = 0.32, p = <0.001), robotic hysterectomies (r = 0.17, p = 0.048), stress incontinence (r = 0.29, p < 0.001) and prolapse procedures (r = 0.37, p < 0.001). Inverse correlation was seen between test scores and years in practice. (r = -0.19, p = 0.02). CONCLUSION: Intraoperative judgment in practicing OBGYN physicians appears similar to resident physicians. Practicing physicians who perform FPMRS procedures perform poorly on this examination of surgical judgment; lower performance correlates with less surgical experience and the greater amount of time in practice.Item Open Access Establishing cutoff scores on assessments of surgical skills to determine surgical competence.(Am J Obstet Gynecol, 2010-07) Jelovsek, J Eric; Walters, Mark D; Korn, Abner; Klingele, Christopher; Zite, Nikki; Ridgeway, Beri; Barber, Matthew DOBJECTIVE: The aim of this study was to establish minimum cutoff scores on intraoperative assessments of surgical skills to determine surgical competence for vaginal hysterectomy. STUDY DESIGN: Two surgical rating scales, the Global Rating Scale of Operative Performance and the Vaginal Surgical Skills Index, were used to evaluate trainees while performing vaginal hysterectomy. Cutoff scores were determined using the Modified Angoff method. RESULTS: Two hundred twelve evaluations were analyzed on 76 surgeries performed by 27 trainees. Trainees were considered minimally competent to perform vaginal hysterectomy if total absolute scores (95% confidence interval) on Global Rating Scale = 18 (16.5-20.3) and Vaginal Surgical Skills Index = 32 (27.7-35.5). On average, trainees met new cutoffs after performing 21 and 27 vaginal hysterectomies, respectively. With the new cutoffs applied to the same cohort of fourth-year obstetrics and gynecology trainees, all residents achieved competency in performing vaginal hysterectomy by the end of their gynecology rotations. CONCLUSION: Standard-setting methods using cutoff scores may be used to establish competence in vaginal surgery.Item Open Access Getting under the hood: how and for whom does increasing course structure work?(CBE life sciences education, 2014-01) Eddy, Sarah L; Hogan, Kelly AAt the college level, the effectiveness of active-learning interventions is typically measured at the broadest scales: the achievement or retention of all students in a course. Coarse-grained measures like these cannot inform instructors about an intervention's relative effectiveness for the different student populations in their classrooms or about the proximate factors responsible for the observed changes in student achievement. In this study, we disaggregate student data by racial/ethnic groups and first-generation status to identify whether a particular intervention-increased course structure-works better for particular populations of students. We also explore possible factors that may mediate the observed changes in student achievement. We found that a "moderate-structure" intervention increased course performance for all student populations, but worked disproportionately well for black students-halving the black-white achievement gap-and first-generation students-closing the achievement gap with continuing-generation students. We also found that students consistently reported completing the assigned readings more frequently, spending more time studying for class, and feeling an increased sense of community in the moderate-structure course. These changes imply that increased course structure improves student achievement at least partially through increasing student use of distributed learning and creating a more interdependent classroom community.Item Open Access Learnings From the Pilot Implementation of Mobile Medical Milestones Application.(Journal of graduate medical education, 2016-10) Page, Cristen P; Reid, Alfred; Coe, Catherine L; Carlough, Martha; Rosenbaum, Daryl; Beste, Janalynn; Fagan, Blake; Steinbacher, Erika; Jones, Geoffrey; Newton, Warren PBackground
Implementation of the educational milestones benefits from mobile technology that facilitates ready assessments in the clinical environment. We developed a point-of-care resident evaluation tool, the Mobile Medical Milestones Application (M3App), and piloted it in 8 North Carolina family medicine residency programs.Objective
We sought to examine variations we found in the use of the tool across programs and explored the experiences of program directors, faculty, and residents to better understand the perceived benefits and challenges of implementing the new tool.Methods
Residents and faculty completed presurveys and postsurveys about the tool and the evaluation process in their program. Program directors were interviewed individually. Interviews and open-ended survey responses were analyzed and coded using the constant comparative method, and responses were tabulated under themes.Results
Common perceptions included increased data collection, enhanced efficiency, and increased perceived quality of the information gathered with the M3App. Residents appreciated the timely, high-quality feedback they received. Faculty reported becoming more comfortable with the tool over time, and a more favorable evaluation of the tool was associated with higher utilization. Program directors reported improvements in faculty knowledge of the milestones and resident satisfaction with feedback.Conclusions
Faculty and residents credited the M3App with improving the quality and efficiency of resident feedback. Residents appreciated the frequency, proximity, and specificity of feedback, and faculty reported the app improved their familiarity with the milestones. Implementation challenges included lack of a physician champion and competing demands on faculty time.Item Open Access Measuring surgical trainee perceptions to assess the operating room educational environment.(J Surg Educ, 2010-07) Diwadkar, Gouri B; Jelovsek, J EricOBJECTIVE: To determine measurable differences in the perception of learning between junior and senior residents in the operating rooms of an obstetrics and gynecology (OBGYN) residency program. DESIGN, SETTING, AND PARTICIPANTS: Using a cross-sectional design, the Operating Room Educational Environment Measure (OREEM), a 40-item educational environment inventory, was administered to 28 OBGYN residents from 1 training program, who train at 3 hospital sites. The OREEM measures a trainee's perceptions of the teaching surgeon, learning opportunities, operating room atmosphere, and workload. The primary outcome was total OREEM scores and secondary outcomes were OREEM subscale scores, global impression of education, and internal consistency and validity of the OREEM scale. Group sample sizes of 10 and 10 achieved 80% power to detect a 10% difference between group mean OREEM scores +/- 10% with a significance level of 0.05. RESULTS: Twenty-four residents including 11 junior (postgraduate years 1 and 2) and 13 senior (postgraduate years 3 and 4) residents were included in the analysis. Total OREEM scores, learning opportunities, and workload/support subscale scores were significantly lower for junior residents compared with senior residents across all sites. Perceptions of learning at a multispecialty tertiary referral hospital were lower than the community and regional hospitals. This was secondary to complexity of cases, subspecialty fellows, and decreased opportunities to first-assist in the operating room. The OREEM demonstrated acceptable reliability and construct validity. CONCLUSIONS: There are measurable differences in perception of the operating room educational environment between junior and senior OBGYN residents using the reliable and valid Operating Room Educational Environment Measure.Item Open Access Selectivity of physiotherapist programs in the United States does not differ by institutional funding source or research activity level.(J Educ Eval Health Prof, 2016) Riley, Sean P; Covington, Kyle; Landry, Michel D; McCallum, Christine; Engelhard, Chalee; Cook, Chad EPURPOSE: This study aimed to compare selectivity characteristics among institution characteristics to determine differences by institutional funding source (public vs. private) or research activity level (research vs. non-research). METHODS: This study included information provided by the Commission on Accreditation in Physical Therapy Education (CAPTE) and the Federation of State Boards of Physical Therapy. Data were extracted from all students who graduated in 2011 from accredited physical therapy programs in the United States. The public and private designations of the institutions were extracted directly from the classifications from the 'CAPTE annual accreditation report,' and high and low research activity was determined based on Carnegie classifications. The institutions were classified into four groups: public/research intensive, public/non-research intensive, private/research intensive, and private/non-research intensive. Descriptive and comparison analyses with post hoc testing were performed to determine whether there were statistically significant differences among the four groups. RESULTS: Although there were statistically significant baseline grade point average differences among the four categorized groups, there were no significant differences in licensure pass rates or for any of the selectivity variables of interest. CONCLUSION: Selectivity characteristics did not differ by institutional funding source (public vs. private) or research activity level (research vs. non-research). This suggests that the concerns about reduced selectivity among physiotherapy programs, specifically the types that are experiencing the largest proliferation, appear less warranted.Item Open Access The enhanced examination for professional practice in psychology: A viable approach?(The American psychologist, 2020-01) Callahan, Jennifer L; Bell, Debora J; Davila, Joanne; Johnson, Sheri L; Strauman, Timothy J; Yee, Cindy MHealth disciplines have increasingly required competency-based evaluations as a licensure prerequisite. In keeping with this trend, the Association of State and Provincial Psychology Boards (ASPPB) has begun to develop a second part to the Examination for Professional Practice in Psychology (EPPP). The resulting 2-part examination is collectively referred to as the Enhanced EPPP. Part 1 of the Enhanced EPPP, which consists of the current exam, is designed to be an assessment of knowledge. Part 2 of the Enhanced EPPP is newly developed and intended to address the need for a competency-based evaluation. To date, ASPPB has addressed some standard facets of validity for the EPPP Part 2, but not others. In addition, the EPPP Part 2 has yet to be subjected to a broader validation process, in which the suitability of the test for its intended purpose is evaluated. Implementation of the EPPP Part 2 before validation could have negative consequences for those seeking to enter the profession and for the general public (e.g., potential restriction of diversity in the psychology workforce). For jurisdictions implementing the EPPP Part 2, failure to gather and report the evidence required for use of a test in a forensic context may also open the door for legal challenges. We end with suggestions for feasible research that could significantly enhance the validation process for the EPPP Part 2 and offer jurisdictions concrete suggestions of features to look for in determining whether and when to implement the Enhanced EPPP. (PsycINFO Database Record (c) 2020 APA, all rights reserved).Item Open Access Variation in Oral Board Examination Accommodations Among Specialties.(JAMA network open, 2024-05) Rowe, Dana G; Charles, Antoinette J; Luo, Emily J; Arango, Alissa M; Herndon, James E; Hockenberry, Harrison; Shortell, Cynthia K; Goodwin, C Rory; Erickson, Melissa MImportance
Board certification can have broad implications for candidates' career trajectories, and prior research has found sociodemographic disparities in pass rates. Barriers in the format and administration of the oral board examinations may disproportionately affect certain candidates.Objective
To characterize oral certifying examination policies and practices of the 16 Accreditation Council for Graduate Medical Education (ACGME)-accredited specialties that require oral examinations.Design, setting, and participants
This cross-sectional study was conducted from March 1 to April 15, 2023, using data on oral examination practices and policies (examination format, dates, and setting; lactation accommodations; and accommodations for military deployment, family emergency, or medical leave) as well as the gender composition of the specialties' boards of directors obtained from websites, telephone calls and email correspondence with certifying specialists. The percentages of female residents and residents of racial and ethnic backgrounds who are historically underrepresented in medicine (URM) in each specialty as of December 31, 2021, were obtained from the Graduate Medical Education 2021 to 2022 report.Main outcome and measures
For each specialty, accommodation scores were measured by a modified objective scoring system (score range: 1-13, with higher scores indicating more accommodations). Poisson regression was used to assess the association between accommodation score and the diversity of residents in that specialty, as measured by the percentages of female and URM residents. Linear regression was used to assess whether gender diversity of a specialty's board of directors was associated with accommodation scores.Results
Included in the analysis were 16 specialties with a total of 46 027 residents (26 533 males [57.6%]) and 233 members of boards of directors (152 males [65.2%]). The mean (SD) total accommodation score was 8.28 (3.79), and the median (IQR) score was 9.25 (5.00-12.00). No association was found between test accommodation score and the percentage of female or URM residents. However, for each 1-point increase in the test accommodation score, the relative risk that a resident was female was 1.05 (95% CI, 0.96-1.16), and the relative risk that an individual was a URM resident was 1.04 (95% CI, 1.00-1.07). An association was found between the percentage of female board members and the accommodation score: for each 10% increase in the percentage of board members who were female, the accommodation score increased by 1.20 points (95% CI, 0.23-2.16 points; P = .03).Conclusions and relevance
This cross-sectional study found considerable variability in oral board examination accommodations among ACGME-accredited specialties, highlighting opportunities for improvement and standardization. Promoting diversity in leadership bodies may lead to greater accommodations for examinees in extenuating circumstances.Item Open Access WriteSim TCExam--an open source text simulation environment for training novice researchers in scientific writing.(BMC Med Educ, 2010-05-28) Shah, Jatin; Rajgor, Dimple; Vaghasia, Meenakshi; Phadtare, Amruta; Pradhan, Shreyasee; Carvalho, Elias; Pietrobon, RicardoBACKGROUND: The ability to write clearly and effectively is of central importance to the scientific enterprise. Encouraged by the success of simulation environments in other biomedical sciences, we developed WriteSim TCExam, an open-source, Web-based, textual simulation environment for teaching effective writing techniques to novice researchers. We shortlisted and modified an existing open source application - TCExam to serve as a textual simulation environment. After testing usability internally in our team, we conducted formal field usability studies with novice researchers. These were followed by formal surveys with researchers fitting the role of administrators and users (novice researchers) RESULTS: The development process was guided by feedback from usability tests within our research team. Online surveys and formal studies, involving members of the Research on Research group and selected novice researchers, show that the application is user-friendly. Additionally it has been used to train 25 novice researchers in scientific writing to date and has generated encouraging results. CONCLUSION: WriteSim TCExam is the first Web-based, open-source textual simulation environment designed to complement traditional scientific writing instruction. While initial reviews by students and educators have been positive, a formal study is needed to measure its benefits in comparison to standard instructional methods.