Board Review Course Pass Rates Are Bogus. Here’s Why.

The great enemy of the truth is very often not the lie—deliberate, contrived, and dishonest—but the myth—persistent, persuasive, and unrealistic.

—JFK

I recently returned from a medical education conference and spoke to many residents and attendings who were preparing to take their certification or board exam. One question I was repeatedly asked was: “What is your pass rate?” This seems like a reasonable question, right?

---

But it drives me crazy.

Not because someone is curious about statistics, but because some companies actually publish “pass rates.” 

Companies that publish pass rates are misleading you. It’s that simple.

Let’s look at an example.

5,000 people use Rosh Review to prepare for their upcoming board exam. A pass rate of 98% means 4,900 of Rosh Review users passed their board exam and 100 people failed. Seems reasonable since the actual pass rate of most certifying exams is between 70% and 92% of all first-time test-takers.

A closer look reveals the catch: to promote or publish a 98% pass rate, Rosh Review has to track down and ask 5,000 users whether or not they passed their board exam. For those of you involved in research, you are familiar with response rates. A good response rate for a survey is around 20%. Assuming that holds true, out of 5,000 people, we could reasonably expect a response from 1,000 people. So, out of 1,000 respondents, 980 say they passed, which translates to a 98% pass rate.

This is pseudoscience. Plain and simple.

But there’s more…

I dug a little deeper into how some companies publish their pass rates and this is what I found.

Many companies, including Rosh Review, has a guarantee of passing. Meaning, if you use our service and do not pass your exam, we will refund you 100% of the cost (I’ve seen some companies who refund 150% of the cost…so you get paid if you fail!) What some companies do is take the number of people who asked for a refund and divide it by the total number of users of the product.

The big assumption is that all users who fail also ask for a refund, which I can tell from my experience with Rosh Review is nonsense (more on this later).

Here is the truth: the only entity who can publish an accurate pass rate is the certifying body itself.

Knowing this, Rosh Review takes a different approach.

First, we do not publish a pass rate. If we did, we’d be misleading you and be practicing pseudoscience. This is not part of our core values. 

Second, we do offer a “100% guarantee” to people who do not pass their board exam. These are our numbers (since 2012) [updated August 2021] for people who have acted upon our “100% guarantee” promise (based on approximately 225,000+ total customers):

• Emergency medicine certification exam: 156
• Family medicine certification exam: 9
• OB/GYN Qualifying exam: 7
• Internal Medicine certification exam: 3
• PANCE/PANRE: 317
• Pediatrics certification exam: 4
• Pediatric emergency medicine certification exam: 2
• Family nurse practitioner certification exam: 10
• Emergency nurse practitioner certification exam: 8
• Special purpose exam: 3

So…what is Rosh Review’s pass rate?

Well, if we calculated it using the pseudoscience method we’d say that 519 learners contacted us because they did not pass their exams. At 225,000 learners, this means that 224,481 must have passed. Which translates to a passing rate of 224,481 / 225,000 = 0.9976. A 99.8% passing rate!

I can easily state that Rosh Review has a 99.8% board exam pass rate. But the reality is…I don’t know what it is.

What I can tell you is that none of these customers who acted on our “100% guarantee” asked for their money back. Instead, they asked to continue to use the Rosh Review question bank as they prepared to retake their board exam. Our subscribers find it more beneficial to stay with Rosh Review than to ask for a refund.  

We understand how important it is to pass your high-stakes exam. Your career depends on it. This creates a new standard for us. Our “100% guarantee” is not simply to return a subscription fee.

Our “100% guarantee” means we promise we are going to stick by your side and do whatever it takes to help you reach your goals and achieve your dreams.

This reminds me of a blog post I once read by Seth Godin, who highlighted the difference between “dazzling people with hype” or “investing in delighting people.” The example he used to describe investing in delighting people is “the insurance company that points out 99% of its customers would recommend them—after filing a claim. Imagine that standard: dealing with the emotions and financial impact of an insurance claim, knowing that you need to maintain a 99% delight standard.”

So instead of promoting a “99% pass rate,” Rosh Review promotes a “100% delight standard.”

We aim to dazzle you by our commitment to your success; by always providing the highest-quality content, and to continually go one step further. We believe it is a privilege to work for you and will always stand by you.

Best Regards,
Adam Rosh, MD
Founder of Rosh Review