A Case for High-stakes Summative Examinations?

The role of examinations like the USMLE Step 1 has been called into question due to:

1. Tests only medical knowledge - not the other competencies that go towards making a "good physician"
2. Forces students to memorize content just to regurgitate it at exam time
3. Most physicians may not need to use the content memorized for this test in later life.
4. Biomedical science information is growing so fast that there is no way to cover all this content within the span of an average medical school curriculum

Medical Education recently published a very interesting study [pdf; abstract] regarding year 4 medical students in Germany learning EKG interpretation.

Summary of the study:

• Cohort 1 - winter 2008-09 - Summative assessment at end of EKG course - randomized to:
• Traditional lectures
• Small group peer teaching
• Cohort 2 - summer 2009 - formative assessment at end of EKG course - randomized to:
• Traditional lectures
• Small group peer teaching

In addition to a test before and after the EKG course, each cohort had an unannounced test 8 weeks after the EKG course.

Thus the study was designed to look at the impact of 2 types of learning (traditional lecture vs. small group peer learning) and the impact of a high stakes summative examination on the learning.

They found that in the Cohort 2 where there was only a formative assessment at the end of the course (performance did not count towards grades) the small group peer learning group did better at the immediate post test and also retained the information better at the 8 week surprise test.
On the other hand in Cohort 1, the high stakes summative test seemed to eliminate all the differences between the 2 groups.

The question:
What is it about the summative test that improved learning and retention?  Did it force students to spend more time in self-study preparing for the test or did it force them to concentrate better at the lectures, taking notes etc?

Reading this study made me think about how this might apply to even higher-stakes summative tests like the USMLE Step 1.  Scores from this one test can determine a student's career.

One student who had spent the first 2 years in small group PBL sessions, told me that as he was preparing for the USMLE Step 1 at the end of year 2, everything seemed to fall in place.  We discussed this further and it seemed that having to read all the different subjects in a concentrated time span helped him to develop mental connections between various pieces of information in his memory thus improving his understanding.

So what do you think?  What is the role of high stakes summative tests in medical education?  Are they good, bad, a waste of time, a distraction, or something that we should have more of?

More Thoughts about Conference Attendance

Last week while Lily was attending the AERA conference, I attended the Pediatric Academic Societies annual meeting. As a PhD medical educator, I was in the minority as most of the attendees were pediatricians in academic medicine. I must admit that I felt like a bit of a fish out of water when I first looked through the program guide as many of the sessions were scientific presentations with titles containing unfamiliar terms and acronyms. I was worried about whether I really belonged at this conference. Upon closer inspection, however, I identified lots of sessions related to medical education including topics such as reflective practice, developing measurement tools in education, and giving feedback. I decided to attend a mixture of sessions with most related to education along with a few scientific presentations and public health sessions in hopes of experiencing the “big picture” of pediatrics.

At the end of four days, I left the conference with a much greater appreciation for the educational issues that are of interest within pediatrics and the larger context in which they exist. A consistent theme was the need for the evaluation of educational interventions to build an evidence-base for educational activities that could be shared across programs.

I, like Lily, came away from the meeting more aware of the benefits of thinking about medical education through a lens other than the one in which I am trained and more convinced of the need for collaboration between those trained in medicine and in education.

We are wondering what your experiences are in attending conferences outside your area of academic expertise and what were the important takeaways from those meetings for you? Are there conferences that you would recommend to your colleagues in medical education?

Medical Education Conferences: Which meetings to attend and why?

This entry is a post from Lily .

Christine and I attended the AERA (American Educational Research Association) annual meeting in Denver the first week of May. For those of you unfamiliar with AERA, it boasts 25,000 members who are dedicated to educational scholarship and learning. The group is diverse, including educators, administrators, public employees, researchers, psychometricians, behavioral scientists and students. The AERA is committed to scholarly inquiry related to education and evaluation.
This was my first time attending AERA, and I was privileged to present a paper on our Cleveland Clinic REALL (Resident Educator and Lifelong-Learner) Program during a Division I (Education in the Professions) session entitled “Means and Effects of Scholarly Teaching.” Our project was was well-received and I got valuable feedback from the chair of the session, Casey B. White (University of Michigan Medical School), and Luann Wilkerson (UCLA). The feedback from these individuals was very specific and detailed. You could tell that they had read the paper thoughtfully and carefully. Drs. Wilkerson and White took their responsibilities to heart - they had much to offer in the areas of scholarship in medical education, using theoretical constructs and applying these theories to practice. They analyzed our REALL project within these frameworks.
I also chose to participate in Division I's pilot program, Peer Review and Feedback on Junior Scholar's Presentations, an interactive program where I obtained a critique of my presentation skills from Dr. Ann Frye (UT Galveston). Of course, I had the obligatory audio-visual problems from the get-go (there were 3 different feeds into the LCD projector, 2 Mac computers and my Windows netbook, which the AV assistant cursed because his fingers were bigger than the keyboard!) Dr. Frye offered me several useful tactics to improve my presentation style. As my background and education are based in clinical medicine, many of the topics and sessions at AERA were new to me. I found this meeting to be very different from the AAMC and CGEA; the vast majority of the topics fell outside the medical field – with Special Interest Groups such as Constructivist Theory, John Dewey Society, Motivation in Education, just to name a few. Like many doctors, I seem to attend the same meetings mechanically year after year, listening to the same people talk. Attending this one was an eye-opener, and a great opportunity for cross-fertilization of ideas. I encourage you to check it out at http://www.aera.net/ and consider attending the 2011 meeting in New Orleans, April 8 - April 12.Lily

What can Medical Educators do to Reduce Health Care Costs?

Molly Cooke in a recent article in the NEJM (Cooke, Molly Cost Consciousness in Patient Care -- What Is Medical Education's Responsibility? N Engl J Med 2010 0: NEJMp0911502) suggests that medical educators may have failed in their task of teaching medical students and house staff appropriate use of various diagnostic tests and therapeutic procedures.

While she acknowledges that there are various factors why physicians may order more tests or procedures, she states that medical educators need to do more to help change practice behavior of these future physicians.

Schools and residency programs already educate trainees on the concepts of Number Needed to Treat, Absolute and Relative Risk Reductions, Sensitivity, Specificity of Tests, Positive and Negative Predictive Values, Likelihood Ratios etc.

The New York Times in a recent article discusses whether the new health care reform act will lead to decrease in health care spending.

Do you think more training regarding cost conscious use of tests and procedures in medical school will lead to change in practice behavior of future physicians?  Will other drivers like financial incentives overcome any effect of training?  What do you think we can do as medical educators that will truly impact the cost of health care?

Neil Mehta