Lessons from the Bell-Curve
Many of you may be in the process of revising for UCAT. A core element of doing this well is taking mock exams, each providing you with feedback through a score. The major difficulty I, and some of my friends, found in this process was knowing how to efficiently utilize this feedback. Using my own experience, as well as lessons from “The Bell-curve” ( a chapter in “Better”) I have a few tips on how to overcome this:
“Look at the deciles.” At the link here, you are able to look at the test statistics of the cut-offs for different total UCAT scores and SJT scores. This is useful as it gives you an approximate idea of where your scores sit. If you are using Medify, they automatically enable this comparison with other candidates who took the same UCAT exam as your mock. In terms of what decile to aim for, this will depend entirely on the university you are applying to, but for most Scottish Universities any score in the 8th decile or more is excellent.
Figure 1- Bell-Curve of UCAT Scores1
“Track your progress.” Although Medify can do this to some extent, having a separate excel or google sheets document where you record your different total scores and scores within each section overtime will be very useful. This will allow you to identify your weakest areas and help you identify when you are starting to get burnout (it is natural for scores to drop if you become fatigued).
To end this first section off, I would like to stress that UCAT is renowned for being difficult. So if you are struggling you are not alone.
Book of the week: Better: A Surgeon’s Notes on Performance by Atul Gawande
This week’s theme of ingenuity is the topic covered by Gawande in the third section of his book “Better.” The author draws upon 3 key stories, one of which will be covered this week.
“The Bell-Curve.” Gawande opens with a comment on how healthcare is to improve:
“Finding a meaningful way to measure performance is a form of ingenuity in itself, what you actually do with that measure involves another type of measure and improvement ultimately requires both kinds.”
The reader is transported into Cincinnati Children’s Hospital, where a young -newly diagnosed- Cystic Fibrosis (CF) patient is attending a consultation with their family. However, despite what the family initially thought, this was not one of the top centers for CF at the time (according to survival rates). This anecdote is used by the author to introduce the idea that despite the differences between different doctors in different hospitals being assumed to be minimal at the time, this is far from the case. If a graph was plotted showing all centers treating any disease people would expect the curve to look like a shark fin, with most places clustered around the very best outcomes. But the author claims that the evidence indicates otherwise, with the reality being a bell-curve found instead. Within this bell-curve, there is a handful of centers obtaining quite bad results, a handful obtaining very good results and a great undistinguished middle. Gawande provides a couple real life examples to support this claim (e.g the likelihood of pregnancy from a given attempt at in vitro fertilization).
On the other hand, some may argue against this line of reasoning, instead pointing to: differences in the ages of patients a centre sees, a centers willingness to accept high risk patients, and other factors accounting for this variability, rather than the performance of different doctors. Gawande argues against this, claiming for a given patient there are wide meaningful differences among centers and that a few are simply better than the rest. Gawande then tackles with the implications of this bell-curve:
“The bell-curve is a distressing phenomenon to accept for doctors, it belies the promise we make to patients- that they can count on the medical system to give them their very best chance. It also contradicts the belief that all of us have that we are doing our job as well as it can be done.”
He claims that although doctors are used to confronting failure (all doctors have deaths and complications), many are not used to comparing their records of success and failure to their peers.
This then leads the author to the question of how to measure performance in medicine? Gawande firstly looks to the federal government, who for 6 years released a report that came to be known as the “death list.” This list ranked all hospitals for their death rates in elderly and disabled patients. He argues that these rankings proved to be almost useless. The problem was that death among the elderly and disabled mostly had to do with how sick or old they are to begin with, and the statisticians could never figure out how to portion “blame” between nurture and nature. To make matters worse, hospital rankings seesawed dramatically from one year to the next, and it was ultimately unclear what kind of changes would improve their performance other than sending their sickest patients to other hospitals. Even with younger patients death rates are a poor metric for how doctors do, after all very few young patients die and when they do they already probably have a serious injury or condition. Gawande claims, thus, one really wants to know how doctors perform in typical circumstances. He suggests some sort of score for the immediate results, and perhaps also a measure of the processes involved (e.g how quickly patients with pneumonia are given the correct antibiotic, and on the whole how this patient does). The main concerns with this would be the difficulty, and expenses involved in this data gathering process, as well as how to stay within privacy regulations.
Gawande argues that there is “one small field in medicine that has been miles ahead in measuring the performance of their practitioners,” cystic fibrosis care. In 1964, a private organization was paid $10,000 to investigate the data of different CF centers in the USA. They found one centre to have a median age of death of 21 years their patients, 7 times the age of patients treated elsewhere- a positive deviant. The field tried to figure out why. In trying to do so, they discovered the center’s innovative perspective of CF as a cumulative disease, and their use of preventative treatments long before the patients became visibly sick. This treatment then became standard, contributing to the remarkable and massive transformation in survival rates in CF patients.
As the years went on, it was revealed that for every average increase in the bell-curve, some of the top centers managed to stay ahead of the curve. At these top centers the median age of survival of CF patients increased to 47 years and the average lung function of patients is indistinguishable from those who do not have CF. There are some skeptics, who claim these differences are explained simpily by the different genes that the patients inherit, or by the social class of the families that visit these top centers. Gawande argues against this, pointing to analysis that shows -at most- only ¼ of the variability can be attributed to these factors. The author then ponders over the implications of this:
“What makes the wide variability(in performance of different CF centers) puzzling is our treatment for CF care are so much more sophisticated than that for most diseases. CF care works the way we want all medicine to work, patients receive care in vigorous, specialized centers. All centres have vigorous authentication processes, and their doctors have a high volume of experience for caring for people with CF. The centers have detailed guidelines, and all participate in research trials to find new treatments. But the differences in their outcomes with patients are enormous.”
Gawande then addresses the elephant in the room, what happens when patients find this out? The author turns to Donald Berwick, the CEO of the Institute for Healthcare Improvement at the time, who created incentives for doctors to be open to patients about their performance. Gawande recalls one of Berwick’s speeches on the matter:
“ He(Berwick) opened with a story of a group firefighters caught in a wildfire… Panicking they ran, trying to make it to safety, but their commander saw that their plan wasn’t going to work. So he stopped, took out some matches and set the tall dry grass ahead of him on fire, the new blaze caught, and then he stepped into the burnt out area it left behind, laid down and called to his crew to join. In that moment he invented on the spot what came to be known as an “escape fire.” This later became a standard part of training. His men at the time, however, either thought he was crazy or didn’t hear his calls. Inside his escape fire he was virtually unharmed and survived, whereas most of the men died. Berwick explained that the firefighter’s organization had shown the men’s lack of ability to think coherently and to act together, to recognize that a life saving idea might be possible.”
It is described that Berwick draws parallels between the firefighter organization and medicinal organizations:
“This is what happens to all flawed organizations in a disaster, and he argued that this is what is happening in modern healthcare, as medicine is trying to cope with the increasing volume of information and technology available it is falling short in performing even the simplest of tasks.”
The question naturally posed is how can we then fix medicine? Berwick maintained that two things must be done:
“Measure ourselves and be more open about what we are doing.”
Ultimately, this analogy emphasizes the importance of routinely comparing performance between hospitals and doctors, and the need for hospitals to give patients total access to information; “ “no secrets” is the new motto of Berwick’s escape fire.” It is argued that this openness would drive improvement.
Adding to this line of reasoning, Gawande utilizes an anecdote from the Cincinnati Children’s Hospital. In this hospital, patients were invited to a meeting regarding their hospitals performance in CF care. They told patients how poorly they had performed and what they were going to do to improve, surprisingly not a single family left the program. A key part in this center’s improvement was to look at higher performing centers for CF treatment.
The author visited one such centre in Minneapolis. After speaking to one of the doctors in this CF center (Warren Warwick), he found that the secret behind their performance was simple: “do whatever you can to make your patient’s lungs as open as possible.” This approach is captured in one of Warwick’s consultations with a young CF patient who failed to undergo appropriate treatment at home:
“A person’s daily risk (who has CF) of getting lung disease who undergoes appropriate treatment that day is 0.5 %. The daily risk of a person (who has CF) of getting lung disease without treatment that day is 0.05%. So when you expereinment you are looking at a difference between a 99.5% chance of staying well and a 99.95% chance of staying well, seems hardly any difference right? On any given day you have a 100% chance of being well. But… it is in fact a big difference. Over a year it is a difference between a 83% chance of making it through the year without getting sick and only a 16% chance.”
Warwick believed that excellence came from seeing the difference between a 99.5% and 99.95% success rate. Many things humans do are like that, medicine’s distinction is that lives are lost on those slim margins. Additionally, Gawande highlights Warwick’s “focus, aggressiveness and inventiventiveness” and points to these traits as pillars of the center’s success. He notably invented “the vest” in CF treatments for example.
Gawande goes on to reflect of the implications of this:
“We usually think a doctors ability is based on science and skill, the lesson from Minneapolis’s is that these may be the easiest part of care. Even doctors with great knowledge and technical skill can have mediocre results, more nebulous factors like aggressiveness, ingenuity and diligence can matter enormously.”
Thinking to Berwick, Gawande claims that only if we know the results from all centers can we identify the positive deviants and learn from them. If we are genuinely curious about how the best acheive their results, we must be open. Berwick’s dream of an “escape fire of openness” finally came to fruition in 2006, with complete public access to performance of CF centers in the USA being available to everyone. As a result, all centers appeared to make significant progress.
However the centers in the top quartiles still improved the fastest, looking to why Gawande claims “What the best may have above all is the capacity to learn and change, and to do so faster than everyone else.” Gawande then ponders on the implications of the proof of this bell curve:
“The hardest question for anyone who takes responsibility for what they do is what if they turn out to be average? If they took all surgeons at my level of experience and discovered that I was one of the worst how could I justify putting patients under the knife if better doctors were out there? If the bell-curve is a fact, so is the reality that most doctors are going to be average, there is no shame in being one of them.”
As bleak a picture as this seems to present, Gawande ends on a positive and empowering perspective on the matter:
“What is troubling is not being average, but settling for being average. Everyone knows that averageness is our fate, but in your surgeon, your police department, your high school, when the stakes are your lives and the lives of your children, we want no one to settle for average.”
Thanks for reading,
The Vansoh is a reader-supported publication. If you appreciate our work, consider becoming a free or paid subscriber.
https://icanmed.com.au/ucat-guide/what-is-a-percentile/