Friday, October 31, 2008

OCTOBER SUMMARY

Dear Teacher’s BLOG Readers:
The responses to the October’s Question, “GLOBAL RATING SCALES: USELESS AS A TOOL FOR IMPROVING PERFORMANCE” were insightful and honest as well as practical. If you have not already read them, I encourage you to go to the OCTOBER Archives and read each one. The following are just a few “pearls” from the list.

There was a consensus that “global ratings” alone do little to assist our learners in improving practice. Some quotes:
  • “The student requires a specific notion of what should be improved and therefore numeric value will be useless if there are no specific comments that will provide further feedback and insight”.
  • “I agree that a global rating scale number such as "7" alone provides little information to a student about performance. Descriptive feedback is definitely better at elucidating areas of strength and weakness. This type of feedback allows the student to internalize and re-assess their performance with opportunity to adapt to expectations and/or standards.”

Others responded to GRS reliability and the use of data to compare students or residents. Some notes and quotes:

  • “Just like with Likert scales, things are given a numerical assessment and each number is exactly the same distance from the other, yet what these numbers mean is not uniform in distance. For example, the distance between what someone would rate a 5 and an 8 on these scales is much smaller than the difference between 1 and 4. In fact I don't even know if 1 exists! This renders the scale even more arbitrary than what is just personal point of view of what an 8 means, etc. This has direct implications on using the numbers in any way - for example averaging and comparing.”
  • “…. I think that the information from these GRFs becomes more meaningful when there are a number of raters. In this case, if a student consistently is scroing much lower than his or her peers, (across several stations) then we can intervene with that student and provide targeted input, correction, remediation -- whatever you want to call it. Yet, the students say that they learn the most from the narrative comments that the SP's put on each section of the form, noting what the student did well and what could be improved. I don't think that it's really a question of GRFs vs. narative feedback; we need both as they meet different aims

There are good questions about the value and psychometric properties of Global Rating Scales. A nice short article that addresses the inherent problems with GRS is listed below.
Farrell, S E. (2005) Evaluation of student performance: clinical and professional performance. Academic Emergency Medicine. 12(4): 302e6.

Wednesday, October 15, 2008

GLOBAL RATING SCALES: USELESS AS A TOOL FOR IMPROVING PERFORMANCE

Probably, all of us can agree that the ACGME Six Core Competencies are important measures of a good physician. They have resonated with medical schools so well that a large number of schools have incorporated these competencies into their expectations for graduates. Who could argue with the concept that we want our graduates to be knowledgeable, clinically proficient, professional, good communicators, lifelong learners, and good stewards of the healthcare system in service to their patients.

The problem, as I see it, is not the competencies, but how we assess them. For the first five years after the introduction of the Competencies, new Global Rating Forms (GRFs) were introduced as the "answer" for assessing the competencies. In this approach, students or residents are assigned a number from a scale, for example, a "5" out of a possible "10" on one of the competencies. The faculty member has "done his/her duty", but how satisfying is the process for our teachers, and what in the world does that student or resident do with that "5".

Our medical school (CCLCM) is a competency-based school that approaches assessment systematically by emphasizing formative narrative feedback organized through a portfolio sytem. Students never are assigned a number, rather they receive narrative feedback about their "strengths" and "areas needing improvement". This seems to be working really well, although, there is a bit of a "learning curve".

So.. what do you think? Are there some strengths to GRFs that I am missing? Have you found ways to make those "numbers" tell a story that leads to improvement? Let us know what you think? Any residents or students reading this link? What do you think?