Test-Taking Strategies, Overcoming Failure, and Building Confidence

“Why would I be wound up? I’m either ready, or I’m not. Worrying about it right now ain’t gonna change a damn thing.” 

– Floyd Mayweather Jr. just before a fight, as recalled by Paul Levesque

---

This is a really great episode for anyone who wants to learn more about studying for high-stakes exams and test taking in general. 

And instead of me doing the interviewing, I had the honor to be interviewed by Jessica Veale, a talented and motivated PA student at Duke University. 

Jessica is the host of the wonderful podcast “The PA Process”.

In this interview, Jessica and I explore test-taking strategies—but not your ordinary test-taking strategies.

We talk about some common learning theories such as retrieval practice, the testing effect, and elaboration, and uncommon ones such as the illusion of knowing.

But where things get really good is in our discussion about:

• How to digest large amounts of information, especially in the health professions.
• How studying failure can lead to greater success
• Engaging in self-talk to increase your test score
• Using the “I’m an average test taker” theory to answer the most difficult questions (one of my favorites)
• How to learn from the famous boxer Floyd Mayweather to feel confident on exam day
• And so much more

If you are a student of any kind or someone who has to take a high-stakes exam, I promise there is so much value for you in this episode. 

I encourage you to set some time aside, turn up the volume, and listen to this wonderful conversation with Duke University PA Student and host of The PA Process podcast, Jessica Veale.

AUDIO TRANSCRIPT

Jessica Veale:   Hi everyone. My name is Jessica Veale. I’m the creator of The PA Process. This podcast is dedicated to giving advice to pre-PA students, connecting with current PA students, and anything related to the PA profession as a whole. Thanks for tuning in today and let’s get started. Welcome everyone to season two episode two of The PA Process. On today’s episode, my special guest is Dr. Adam Rosh, an ER attending physician and the founder of the well-known question bank Rosh Review. He’s here today to share tips on test-taking preparation, strategies, and tactics. Welcome, Dr. Rosh.

Dr. Adam Rosh:   It’s great to be with you, Jessica. It’s really wonderful to be able to reach out and speak with all of the great PA students around the country.

Jessica:   Thank you. So Dr. Rosh, as a current PA student, I’m quite familiar with Rosh Review and all that it has to offer. But for my listeners who may be unfamiliar with it, can you explain it and what it is and what your motivation was for creating such a useful study tool?

Dr. Rosh:   Yeah, absolutely. I think most people who are in the health professions at some point in their career have taken a standardized test, certification exam at some point and have used a Qbank—a question bank. Rosh Review at its core, it’s a question bank. How that question bank is formulated and structured is very different, I’d say, than most types of question banks that people engage with. It’s not necessarily obvious on the surface. I think the attraction to question banks is based on a lot of good data that we have on the testing effect or retrieval practice. That’s another name for it, right. So when we want to actually test our knowledge because one thing we have to be very cautious of when we’re studying is to avoid something called the illusion of knowing. So just reading something in a textbook and saying to yourself, “I got this. I know it.” You’re probably lying to yourself in one way or another. 

The way to solidify your knowledge—one of the ways at least—is through the testing effect or using retrieval practice. Ultimately, multiple-choice exams are one of the best ways to do that. So we get a question in a question bank and we quiz ourselves. We have to then pull that information out of our brains. Every time we do that, your neurons are making stronger and stronger connections. Whether you know very well the answer or not, you have to think very hard about it. As itself, the retrieval practice is really critical. What we’ve done in Rosh Review is to take that and amplify it. So you have a question with answer choices, you use the testing effect, and then you have this explanation that comes after. This is what a lot of people give us feedback about of why this is such a valuable research for them through their training. So essentially, these comprehensive explanations are structured in a way that was built around how I studied. 

This kind of goes back to ultimately why this ended up being called Rosh Review. When I first started it, I named it something else. I named this product Next Step, and my wife laughed at me. She said it can’t be Next Step. You have to make it Rosh Review. I think I’m at least a modest person, although I’m sure I have tons of flaws as well. We went with Rosh Review because of the method of the content, of how the content is put together. So when you’re reading the content in Rosh Review you may notice that it’s very patternicity, believe it or not. What we include for learners is content that is organized in a way that is logical, it’s supportive. Each line kind of supports the other. It allows you to build connections as you’re reading it. So this amplifies the concept of the testing effect. It really provides a one-two punch for studying. So that’s a long-winded answer to that question. 

Jessica:   No, that’s perfect. It actually kind of falls into my next one. I suffered my first semester of PA school from this illusion of knowing that you are referring to. I battled with the need to make sure I reviewed every PowerPoint lecture, required reading, and then rarely found enough time to actually test myself on what I had studied. With your method of Rosh Review or this question bank, I would love to hear your thoughts on how you incorporate passive learning and making sure you have the content knowledge with active learning. 

Dr. Rosh:   Absolutely. This is something that, I would say, the majority of students deal with at some point in their career. In fact, I dealt with this probably up until my third year of undergrad. I didn’t know how to learn, right. So I set out on my own personal journey of learning how to learn. It was due to that journey that books and education and learning became so valuable to me. Going through a health profession school, the amount of information is enormous, right? We all know that. A lot of the traditional ways of engaging in that content has to do with highlighting and rereading and rewriting your notes, over and over again right. 

What’s really fascinating is as much as I am in people’s lives to help them pass exams and do well in their classes, I also spend a lot of time studying failure. I do that because I want to understand what are the behaviors that are leading to people not performing how I believe that they can perform. Because if you’ve made it this far, it’s unlikely to be a medical knowledge issue or a knowledge issue in general, right? It’s almost never the case. If you don’t pass your PANCE but you made it to PA school, 99% of the time it’s not a knowledge issue. It’s something that you’re doing in your studying. Oftentimes we hear this line. We hear, “They’re a great clinician. They’re great with patients, but they just can’t pass the exam.” Right? So that attracts me. I love to take a deeper dive into those situations. I often—I have many, many people that have reached out. I think this is where the greatest impact comes for me in my personal life in that because I struggled with similar things that I get to reach out and work with these people. So I look forward to that. 

So oftentimes if it’s not due to a distraction at home, it’s about how they’re studying. Almost always the person says to me, “Yeah, I rewrite the notes over and over again until I know everything.” Then I say to them well, you think you know everything. Because it’s in a format and a context that is very familiar to you. The minute you take it out of that context, you won’t be able to answer that question. That is exactly what an exam does. It takes the fact or the critical thinking out of the textbook page or the review book page and it moves it into a different context. So you have to be very mindful of that when you’re studying and preparing. I think if you have time to prepare—like you’re studying for a certification exam or you’re in a classroom, you’re studying for let’s say a rotation exam—it’s okay to do a first round of note taking, a first round of reading. That’s totally fine. You need to get familiar with it. The next step then is to employ educational techniques and theories that we know work. So some of the things that work for me, which I think works for other people as well, is while I’m reading, I will try to make connections in my brain. So not just digest the information that’s been given to me but start elaborating on that information. What that means is to take details and to talk through it. Start asking why. Why is this the way it is? How do these things relate to one another? And to talk my way through it.

In fact, I’m working on a test-taking course right now. I was just doing some experimenting with it. I opened up a review book, a very popular review book that PA students use, and I just read pretty much one or two lines. From that I was able to elaborate and create an understanding of topics that I never even thought about before. It was like osteomalacia and Rickets. After years and years of just kind of thinking like I knew it, I was able to build a really strong foundation around those topics. I won’t go into the details of that now, but it was really eye-opening. So things like elaboration, making connections will help get past the illusion of knowing. 

The testing effect though is the number one way to get past the illusion of knowing, right? That’s why people love Qbanks. It’s important, right. It’s an important thing. Now obviously I have a conflict of interest here. I have a product that we have Qbanks. So I would say this whether or not I was involved in Rosh Review, but it’s important that people know this. That that exists. I think if you look at the science that using the testing effect will help you get past the illusion of knowing. What’s even better is reading something, maybe taking notes, highlighting, then engaging in a Qbank or the testing effect, and then going back to those notes. You’re going to start understanding the material as opposed to just memorizing the material. 

One of the most important things that you could do in this process of study, of reading notes, highlighting, engaging in a Qbank is to write down or somehow make note for yourself of the areas and topics that you get wrong. Because what that is telling you is you don’t know what you don’t know. That’s a class statement. The more and more you can document what you’re getting wrong and then fix that, study it, learn it, eventually you’re going to end up knowing so much more. The amount of material that you don’t know you don’t know is going to be so much less. That is actually one of the tricks to performing well on any type of standardized or certification exam. The more time you have to identify your unknown unknowns, the higher likelihood you’re going to score high and excel. 

Jessica:   I can personally vouch for that. Like I said, first semester I was doing the just reading as much as possible, copying my notes. Then second semester I really started to incorporate the question banks, Rosh Review. I could see the level of recall improving. I could regurgitate some things, but when I would be asked a question it would be difficult to give an answer. That’s that illusion of knowing. You think you know something when you’re studying it, but when you’re asked a question not being able to retrieve what you’ve been studying for so long. So I can definitely vouch for that. When it comes to taking the test, what approach do you take when reading exam questions? For example, some people say you should read the answer choices first. Some say read the first and the last sentence of a question. What have you found to be one of the more effective strategies?

Dr. Rosh:  Yeah. So this is a very common question. I think it’s important and it’s a good question you’re asking because so many people ask it. I think though it matters less than we think. So approaching an exam, the number one thing that is most beneficial—believe it or not—is confidence. If you go into an exam saying, “I’m gonna fail. I don’t know the material. I’m not confident.” You’re going to lose points. Just the nature of your psychology. There’s a lot of studies that show that the simple act of writing down or verbally reciting positive thoughts before an exam leads to increased score. So right, that’s fascinating. You can apply that to lots of things in life, especially in sports and things like that. So okay. That’s the first thing in actually like exam day.

A lot of people—I’ll get to your direct question in a minute, but I think there’s a couple other really important points. A lot of people talk about studying the last minute. Do I study on exam day or the day before? How intense should I be studying as the exam is approaching? There’s a story about Floyd Mayweather. Some consider him one of the greatest boxers of all time. I read an interview of him. The interviewer was conducting the interview the day of a really important boxing match for Mayweather. The match was at like 7:00 p.m. The interview was at 11:00 a.m. The interviewer asked him, they’re like, “Aren’t you nervous? You’re just on the couch here talking to me. You’re walking around. You’re watching TV, you’re laughing. You’re playing ping pong.” Floyd Mayweather said—essentially and I’m paraphrasing here—is, “All my training is already done. How I perform today was decided weeks ago. I just need to show up.” That’s very powerful. What I take away from that is when you’re preparing for an exam, you’ve spent weeks to months preparing for it. Therefore don’t try and pack in last-minute studying the morning of the exam, the night before the exam. It’s much more important to be relaxed, to clear your mind, and to think positive thoughts when you’re entering into that exam. So that’s kind of everything before the exam.

As far as during the exam and how you analyze or read questions, I started off by saying I think it’s much less important. There’s no one right way. Everyone has the best way to do it. The best way needs to be your way. So I think intuitively for me if I have a very long question vignette, I may read the last sentence—what’s called typically the lead-in to the sentence—just so it gives me context. If a question has to do with the management of something and I start reading this long vignette and I realize right away that this patient has acute pancreatitis, then I’m going through in my mind as I’m reading what do I know about the management of acute pancreatitis? I’m looking for clues in the question. So that’s one maybe advantage of skipping to the question lead-in on a long vignette. I think that’s really the only time that we would have to do so because one-line questions is just read the one line. Two-line questions is just read the one line.

A lot people talk about maybe reading the answer choices first then. I don’t think it’s unreasonable to do that. Personally, I love guessing the answer choice before I actually look at the answer choices. So as you’re reading the vignette, you start thinking about what the theme is, what the topic is. Then when you get to the lead-in, you know exactly what the question is being asked. Then I say in my head, okay, what’s the answer? Then I look to see if the answer is there because if that happens, your gut is almost always right. That makes answering that question very easy, especially when let’s say I end up narrowing it down to two answer choices. I always go with what my gut tells me.  

Jessica:   That is such great advice. It brings up another question. Often I hear people say I can narrow down to two answer choices, but then I don’t know which one to pick. Do you have any advice for these individuals? 

Dr. Rosh:   So there’s a few different ways you could go about this, right. If you absolutely have no idea, like there’s no other hint what the answer is, there is non-knowledge tricks that you can do. For example, longer answer choices are sometimes correct over shorter answer choices. There’s things like that. I’ve written about this stuff in blogs. Like if I took a test on rocket science, launching a rocket to the moon, I could probably figure out some of the answers just simply by how the answer choices are written. People, we have to remember that these tests are written by humans. Humans make mistakes and humans have patterns. So you could take advantage of some of these things. The other way to think about this is—and this is something that I actually did an entire lecture on this for this course—it’s what I call…If a question seems really hard—We have to remember when an exam is created, it’s created by humans. Therefore there need to be different levels of difficulty for questions. That has to be the case, right, because that’s how you differentiate. So if you have a really difficult question, you could think about it this way. How would the average student answer that question and then choose the other one, okay? 

Jessica: Okay.  

Dr. Rosh:   Just think about that, right?

Jessica:   Okay.

Dr. Rosh:   If you are the average student and you have a really hard question, then it’s hard for a reason. You are likely to get it wrong. So you chose the one that you think is not the right answer if you don’t know. Like which one would you choose? Choose the one that you don’t think it is. So that’s one way to do it. That also works in the opposite, right? So if you have a question that seems like this is straightforward. This seems like a pretty easy question and the average person will get it right, but you’re actually a really good—your knowledge base is really strong, oftentimes those people outsmart themselves, right. They overthink questions. So you have to remember for an average question, that means almost everyone is going to get it right. Don’t outthink yourself. Go with what is the obvious answer. Don’t outthink yourself. So those are two kinds of strategies that you could employ on answering questions if you don’t know it right away. If you want to think about tactics like this. 

Jessica:   Okay, okay. Then kind of when you’ve gone through the exam and if time permits, do you feel it’s best for you to review all of your questions, only the flag questions, or only unanswered questions which you have to review? I guess in a follow-up, what are your thoughts about changing answers when you do review your exam?

Dr. Rosh:   Yeah, that’s a really great question. I believe you should use all of the time allotted to you. If necessary, reviewing questions is really important because you’re going to pick up on a mistake. So I would go through every single question if possible. Most of the time, 75% of the questions you probably know really well, and you don’t need to spend much time on it. As far as changing questions, that’s a whole other situation. You have to be super careful about that. Here’s kind of my take on this. It’s almost like test yourself before you do it. So some things that I did, and I’ll give an example of this in clinical practice. Whenever I would see a patient with abdominal pain and I suspected appendicitis, I would make a note. I’d say okay, my gut is telling me that this is appendicitis. What’s my likelihood? How sure am I? What’s my certainty that this patient has appendicitis? Am I 100% or am I 50%? I’d make a note. Then after the CAT scan I would refer back to my notes. I would do this to hone in kind of my gestalt. I would say if I’m feeling like the patient has appendicitis using these kinds of instinctual types of feelings and I’m consistently right, that kind of starts building my gestalt. It’s the same thing with changing items on a test.

So you look at on previous tests, like when I change things am I getting it right or am I getting it wrong? You could kind of think about that over time. Obviously if it’s your first exam that you’ve taken, you haven’t taken one in a while, then you’re not going to be able to do that. I always say go with your gut. You don’t want to overthink things. Usually your gut will be saying which one to go with. You have to listen to it. It’s telling you it for a reason. What’s interesting is it’s your subconscious speaking to you. That subconscious was formulated. Those thoughts were formulated while you were studying and doing things and engaging in things like the testing effect. Those were neurons that were built and solidified as you were studying. It’s trying to tell you what’s right. So I think when in doubt, go with your gut. That’s the best approach at least for me. 

Jessica:   Yeah. Yeah. I think I have, going through and taking tests, kind of have that same thing. Like I would go back through, review my answers. For a while changing an answer here or there worked well for me. Then it came to a point where changing answers stopped working well. So then I said well I’m going to stop changing answers unless I misread the question completely or left something out or selected the wrong answer. So I kind of get the understanding of if it’s working for you, continue to do it if you feel like you know. If it stops working and you’re getting more questions wrong then leave it. 

Dr. Rosh:   Yeah, I feel like that’s a great approach. 

Jessica:   So then we kind of talked a little bit about the certification exams and the PANCE, for those exams pretty much everything is fair game. Utilizing question banks is definitely very high yield. What have you found to be some best strategies for studying for like the certification exam or studying for a shelf exam or something like that when pretty much everything can be tested on?

Dr. Rosh:   You know, I think one important takeaway when it comes to education and exams is that you start studying for your exam the first day of school or residency or training, whatever it is that you’re doing. You start your first day. If you wait two years and just start studying for your certification exam at the end of your training program, you have a greater hill to climb. A greater battle in front of you. When I started residency training, one of the first things I did was find a graduating resident—a fourth-year resident at the time—and I asked that resident, “Looking back knowing what you know now, what would you tell yourself as an intern?” I listened very carefully. The number one thing universally across everyone I asked was read from day one. I never realized how challenging that was going to be to read from day one, but it kept kind of echoing in my mind.  

So it’s something that I am so grateful for that I was disciplined enough to do, reading from day one. It was hard, right? My first two years of residency training I was insecure. I came into an environment and a program where people were smarter, they had more experience. I felt very insecure. I was the one kind of reading every day. A lot of people may have not for whatever reason. It’s a challenging thing to do. Eventually my knowledge base built up. What I was seeking was to feel confident. So to feel equal with everyone else. That’s my own personal feeling. No one made me feel not feel equal as far as knowledge goes. So I think preparing for anything starts at day one.

When you have just a prodigious amount of information it means that trying to memorize everything is going to be challenging and not going to be effective. What becomes important is that you learn as much as you can and understand as much as you can because if you could understand the key principles and tenants of certain pathologies, you could extrapolate that information and that knowledge to be able to solve other questions based on that. So you could use your knowledge to eliminate answer choices because you understand what’s being asked and you could use knowledge to affirm answer choices because you understand it.

Jessica:   That makes a lot of sense. Then this question actually someone wanted me to ask regarding test taking. It was what advice you have for individuals that just suffer from test anxiety? 

Dr. Rosh:   Yeah. This is a great question. It’s a real thing that plagues a lot of people. There are a couple ideas that I would share. One is to put yourself as close as possible into the same situation before you take your exam. So as you’re studying and preparing for an exam, try and simulate the same environment. So maybe reserve a mock exam or some type of question set where you’re sitting in front of a computer or you have a sheet of paper in front of you—however, your exam’s going to be administered—isolate yourself, go into a quiet room. If you’re going to wear earplugs, put them in. Simulate the exact testing environment and practice that way. So in the weeks leading up to your exam, try and do that. That will help you be ready for your exam day. 

I think that another idea could be to do fear-setting exercises. This is something that I do not just for exams but in making decisions in my life and taking risks—or what I think are risks. Saying to myself what is the worst-case scenario in these situations? If I fail, what have I learned? What will be an advantage? What other opportunities could come from this? To do fear setting. You could just type into Google fear setting. The first five websites that will come up, I believe there’s some really good ones. It’s just an exercise that I like to do whenever I’m worried about making a decision. You could apply that for taking an exam.

Then I think the last idea is staying positive. Staying positive up until your exam and through your exam. I already mentioned this. I think it’s a very positive message. Even if you, for example, wrote down a paragraph about all the amazing things that you’ve done in your life or in the last two years let’s say. Write them down like you’re journaling. The act of doing that, the act of retrieving those experiences, those positive experiences, will build confidence in you. You could seamlessly transition from doing something like that right into your exam. Then there’s the standard kind of anti-anxiety behaviors of deep breathing, meditation. I use the Headspace app at home. It’s wonderful. I do it with my children. Before exams, I haven’t taken an exam in a long time. I mean three years or so, I haven’t taken a formal certification exam. My last one was in emergency medicine. I still got nervous. 

One of the things that I do is right before I start the exam I’ll take a minute, maybe two, and do some slow deep breathing. One, I want to clear out all of my carbon dioxide in my dead space in my lungs because I want my brain to have as much oxygen as possible. I know my brain is about to go into overdrive, so I want to supply it with fuel. So that’s one benefit of it. Also it’s a calming effect. You want to be calm, right? Calm is cool in a sense. In medicine sometimes when the stress level goes up, our response needs to go down to counterbalance these situations. That’s so true when there’s a trauma, when there’s a code. You see the chaos in the room. It’s the person who’s the calmest is the one who commands the situation. It’s no different in taking an exam. You want to go in—Like Floyd Mayweather, right? He went into his fights just “I’m prepared. I did everything, and now we’ll see the outcome.” It’s the same idea there with test anxiety.

Jessica:   Those are really great suggestions. Like I’m going to have to use some of those myself. So I just want to thank you again for being on the podcast. Before we end, I just wanted to ask. As someone like myself that—You told me that you experienced some rejection before being accepted into medical school, what advice would you give someone that’s listening that’s applying to PA school/med school and they’ve experienced rejections and they’re considering giving up? 

Dr. Rosh:   Well absolutely. I could already say that Duke never accepted me into any of their programs. So you’ve already gone much further than anything I’ve done. I took the MCAT maybe three times even, twice/three times. It was during a time where I wasn’t ready. I wasn’t ready emotionally. My maturity level wasn’t where it needed to be when I first took it and it showed. Eventually, I scored well enough that I got into medical school and I learned so much from that experience. I think one of the main takeaways is you may be qualified. You just may be one of many people who are qualified. It’s not because you’re not qualified that you didn’t get in, let’s say, to a program. 

I think there needs to be some pragmatism on everyone. You need to be pragmatic and ask yourself is this what I want or is this what I think I want? Because if it’s what I want then don’t hesitate a second. You need to just get up and get back moving into what you’re trying to achieve. I think a rejection also allows you to question your motives. It’s an opportunity to pivot to maybe something else that you think you may want to do instead. However, if you are the person—and I was in this boat for sure—where you don’t initially get what you’re looking for or what you’re aiming for. Let’s say it’s an acceptance into a program. In the timeframe after the letter telling you you don’t get in, you need to think about could you improve? Because you can’t do the same exact thing over and over again, right? I think it was Einstein that called that insanity. You need to use that as a moment of growth.  

Jessica:   Right.

Dr. Rosh:   That’s totally fine because you will become a better person with more skills, more network, greater networks, and some other opportunity will come up as well. So I think it’s very emotionally challenging to grapple with rejection no matter what it is. You go through anger, fear, guilt, sadness. All of those things will happen, and that’s a totally normal response. I think there is a time for mourning, right. You should mourn. Like I didn’t get it. But you pick yourself up and you ask yourself okay. What can I do now to add to who I am so that this school or program sees me in a little different way, that they seem that I am motivated to do this. Now, I was on the other end of this. Not only have I been a student, but I’ve been a program director. So I was assistant program director for three years and a program director of the emergency medicine residency. So every year—and even as chief resident at Bellevue at New York University—I was part of the residency selection committee there. You have to create a rank list. I have to interview hundreds of medical students every year. Any health profession school goes through the same thing. 

So there are some programs and people who are on selection committees who may only look at grades and may only look at a test score, and I think that’s a little short-sighted. I think many of the people who make the best clinicians are the ones who went through failure. Its how they dealt with that failure that is always so meaningful. From the program director standpoint, if you could identify those people who struggled for whatever it is and came through that adversity, those people will be always self-motivated. They will work extremely hard, and they will not give up. Life is full of setbacks. Life is full of roadblocks. That’s never ever going to change. What we have the power to do is how we react after a setback. We have a choice. There’s always a choice. If we choose to keep going forward and to find another way around that obstacle—Ryan Holiday wrote a really great book called The Obstacle Is the Way. How do you use the setback to your advantage? This needs to be for anyone who has or currently going through a setback like that. I encourage you to read this book The Obstacle Is the Way. I employ it in so many decisions in my life. It’s a really wonderful way to manage adversity. 

Jessica:   That was well said. I feel like it will really resonate with a lot of people who are experiencing setbacks. I get messages all the time of I don’t know if I can do this. Things are getting difficult. I think you made a lot of good points that a lot of individuals that can overcome adversities or setbacks. A lot of times those individuals, the perseverance really shows. They’re going to go after it. Your inspirational talk regarding this as well as all of the strategies for test taking I really appreciate and know my listeners will as well. 

Dr. Rosh:   I am now a listener as well.