Board Review Course Pass Rates Are Bogus. Here is Why.

by Adam Rosh, MD

jfk3

I recently returned from a medical education conference and spoke to many residents and attendings who were preparing to take their certification or board exam. One question I was repeatedly asked was: “What is your pass rate?” This seems like a reasonable question, right?

But it drives me crazy.

Not because someone is curious about statistics, but because some companies actually publish “pass rates.” 

Companies that publish pass rates are misleading you. It is that simple. 

Let’s look at an example.

5,000 people use Rosh Review to prepare for their upcoming board exam. A pass rate of 98% means 4,900 of Rosh Review users passed their board exam and 100 people failed. Seems reasonable since the actual pass rate of most certifying exams is between 70% and 92% of all first time test-takers. 

 A closer look reveals the catch. In order to promote or publish a 98% pass rate, Rosh Review has to track down and ask 5,000 users whether or not they passed their board exam. For those of you involved in research, you are very familiar with response rates. A good response rate for a survey is around 20%. Assuming that holds true, out of 5,000 people, we could reasonably expect a response from 1,000 people. So, out of 1,000 respondents, 980 say they passed, which translates to a 98% pass rate.

This is pseudoscience. Plain and simple.

But there is more…

I dug a little deeper into how some companies publish their pass rates and this is what I found.

Many companies, including Rosh Review, has a guarantee of passing. Meaning, if you use our service and do not pass your exam, we will refund you 100% of the cost (I’ve seen some companies who refund 150% of the cost…so you get paid if you fail!). What some companies do is to take the number of people who asked for a refund and divide it by the total number of users of the product.

pseudoscience

The big assumption is that all users who fail, also ask for a refund. Which I can tell from my experience with Rosh Review is nonsense (more on this later). 

Here is the truth. The only entity who can publish an accurate pass rate is the certifying body itself. 

science

Knowing this, Rosh Review takes a different approach.

First, we do not publish a pass rate. If we did, we’d be misleading you and be practicing pseudoscience. This is not part of our core values. 

Second, we do offer a “100% guarantee” to people who do not pass their board exam. These are our numbers (since 2012) for people (based on approximately 100,000 total customers) who have acted upon our “100% guarantee” promise:

So…what is Rosh Review’s pass rate?

I do not know.

What I can tell you is that none of these customers who acted on our “100% guarantee” asked for their money back. Instead, they asked to continue to use the Rosh Review question bank as they prepared to retake their board exam. Our subscribers find it more beneficial to stay with Rosh Review than to ask for a refund.  

We understand how important it is to pass your high stakes exam. Your career depends on it. This creates a new standard for us. Our “100% guarantee” is not simply to return a subscription fee.

Our “100% guarantee” means we promise we are going to stick by your side and do whatever it takes to help you reach your goals and achieve your dreams.

This reminds me of a blog post I once read by Seth Godin, who highlighted the difference between “dazzling people with hype” or “investing in delighting people”. The example he used to describe investing in delighting people is the insurance company that points out 99% of its customers would recommend them–after filing a claim. Imagine that standard: dealing with the emotions and financial impact of an insurance claim, knowing that you need to maintain a 99% delight standard.

So instead of promoting a “98% pass rate,” Rosh Review promotes a “100% delight standard.”

We aim to dazzle you by our commitment to your success; by always providing the highest quality content, and to continually go one step further. We believe it is a privilege to work for you and will always stand by you.

Best Regards,

Adam Rosh, MD
Founder of Rosh Review


6 comments to Board Review Course Pass Rates Are Bogus. Here is Why.

  1. Bob Sobule
    October 18, 2016

    Great article! It was a pleasure talking with you at PAEA. As a PA student I love using RoshReview to help me study for my EOR exams. Looking forward to what your product will bring in the future!

    Bob Sobule PA-S2

    Reply
  2. Adam Rosh
    October 18, 2016

    Thanks for the feedback Bob. I met with our team and we worked out the new EOR matrix. It is being built as I write this.
    Best,
    Adam

    Reply
  3. Georges
    December 17, 2016

    This is a great article. I’m been using Rosh review and have recommended my classmates to use it as well. As I’m preparing for the PANCE, this is one of the best tool I’ve come across.
    I’m glad you are planning on EOR exams contents. This is what’s missing in the PA community.
    GP, PA-S3

    Reply
    • Adam Rosh
      December 18, 2016

      Thank you for your kind words. Yes, we are building an EOR-specific Qbank now and hope to launch it in the next couple of months.
      Best,
      Adam

      Reply
  4. Rangani
    January 2, 2017

    It is a scientific article.l am from sri lanka.i have been using this site only for one month.l am a registrar in emergency medicine.i found it very good.

    Reply

Leave a comment