Who is rating the ratings?

Everything is being rated these days. But who is rating the ratings? As a public service, I have been blogging about the shortcomings of various rating systems since 2010. Two recent papers on this topic are worthy of review.

In a randomized controlled study, investigators from the University Hospital of Münster, Germany found that medical students who were provided cookies during academic course sessions rated the experience significantly higher than students who did not receive cookies.

The teacher and the course material were also rated significantly higher by the cookie eaters. On multiple regression analysis, the availability of cookies was the only significant factor. Characteristics such as the course instructor and student characteristics—sex, BMI, and age—did not significantly affect the variation in rating scores.

The presence of cookies explained 6.3% of the variation in summation scores for all questions, which is similar to the impact of other variables looked at in previous studies.

The authors cited previous research on student evaluations of teaching showing such factors as higher grades, small class sizes, classroom attributes, group harmony, and [my favorite] teacher sexiness were associated with improved evaluation scores.

Knowing all this, how can student ratings of teachers be taken seriously?

A research letter from the Cleveland Clinic published in JAMA Internal Medicine looked at over 8400 patient encounters for respiratory tract infections involving 85 telemedicine doctors and found 66% resulted in an antibiotic being prescribed. The estimated prevalence of bacterial acute respiratory tract infections in outpatients is low. A substantial number of the antibiotics prescribed by telemedicine physicians were probably unnecessary.

Physicians received 5-star ratings from 91% of patients who were prescribed antibiotics and 86% of those who received a non-antibiotic drug prescription. When no drugs were prescribed, 72% of patients gave 5-star ratings, a significant difference. Adjusted satisfaction ratings correlated with adjusted antibiotic prescribing rates with a p value of less than .001.

The study received wide media coverage. Lead author Dr. Kathryn A. Martinez felt the overprescribing of antibiotics was due to physician concern about satisfaction scores although that could not be proved by the study results.

There could be another reason. One article stated, “In an analysis of the same data set of telemedicine visits for respiratory infections, the researchers found it takes physicians about 20 fewer seconds per encounter to prescribe an antibiotic.” Martinez felt this may be influencing physicians to prescribe antibiotics in an effort to save time. An expert not involved with study said saving 20 seconds per encounter was not a good reason to give antibiotics.

Whether doctors want better patient satisfaction scores or to save time, they should not overprescribe antibiotics.

Since patients were almost as satisfied with a prescription for a non-antibiotic drug, it seems they just want some kind of prescription, no matter what it is.

The solution to achieving good ratings might be to create a decongestant that sounds like an antibiotic. How about “Mockacillin”?

Skeptical Scalpel is a retired surgeon and was a surgical department chairman and residency program director for many years. He is board-certified in general surgery and a surgical sub-specialty and has re-certified in both several times. For the last 8 years, he has been blogging at SkepticalScalpel.blogspot.com and tweeting as @SkepticScalpel. His blog has had more than 3,000,000 page views, and he has over 18,000 followers on Twitter.