EP192: Medical Image Perception: Insights with Claudia Mello-Thoms

Visual Expertise and Diagnostic Accuracy with Claudia Mello-Thoms, PhD

In this episode of The Girl Doc Survival Guide, Christine interviews Claudia Mello-Thoms, PhD, an Associate Professor of Radiology and Biomedical Engineering at the University of Iowa. Claudia discusses her research on visual search, medical image perception, and why errors occur in reading radiological images. Key topics include the use of eye tracking to detect unconscious viewing patterns, the different diagnostic approaches of experts and novices in pathology, and the concept of visual templates aiding rapid diagnosis. Claudia shares insights into how experts efficiently use low magnification to guide further detailed inspection, and the cognitive load differences between experienced and inexperienced diagnosticians.

00:00 Introduction to Claudia Mello-Thoms

00:47 Eye Tracking in Radiology

02:12 Pathologists and Microscopic Slides

06:56 Visual Search Strategies in Medical Imaging

09:42 Expert vs. Novice: Cognitive Processes

11:35 Templates and Rapid Recognition

20:48 Conclusion and Next Episode Teaser

Transcript

Christine Ko: [00:00:00] Welcome back to The Girl Doc Survival Guide. Today I am very happy to be with Claudia Mello-Thoms, PhD. Claudia Mello-Thoms is an Associate Professor of Radiology in Perceptual Research and Associate Professor Biomedical Engineering, Director of Perceptual Research, and Director of the Medical Image Perception Laboratory at the University of Iowa. She is originally from Brazil, but has spent time in the United States as well as Australia, and her research interests include visual search, medical image perception, and image understanding. She also looks at why errors occur in the reading of radiological or pathologic images. Welcome to Claudia.

Claudia Mello-Thoms: Thank you.

Christine Ko: Can you first share a personal anecdote? 

Claudia Mello-Thoms: Yes. So when I started my research while I was doing my PhD we used eye [00:01:00] tracking, which records what you were looking at in the image. And at that time we're looking at breast radiologists and radiology residents as they were reading mammograms. And so after they completed the case reading, we would show them how they had looked at all of the cases. It turned out that one of our breast radiologists consistently did not look at a certain area of the breast, and she didn't know that, so that was really valuable for her to learn. It was like an unconscious pattern that she repeated again and again. 

Christine Ko: Wow. And so you could find that out by eye tracking.[00:02:00] 

Claudia Mello-Thoms: Yes.

Christine Ko: She didn't look at this one area.

Claudia Mello-Thoms: Yes.

Christine Ko: Wow. And so then she could change. That's really good feedback. And then she could start looking at that area. 

Claudia Mello-Thoms: Yes, exactly. Exactly. 

Christine Ko: Have you looked at pathologists and microscopic slides? 

Claudia Mello-Thoms: Yes. I looked into digital microscopy with both dermatopathology slides and pathology residents, and with cytopathology slides and cytopathology techs and cytopathologists, and in both experiments what we found was that the lowest resolution image, the one that gives you an overall picture of what is in the slide is the major guidance for experts, [00:03:00] but it's not for people with a lower expertise. The expert will look at the lower resolution image, will select some areas to go in and zoom, medium and high magnification. It's a very economical search. They don't browse around looking at things at medium and high magnification. The cyto techs, their job is to browse around to find stuff to bring to the attention of the cytopathologist, but what we found was that they lost their sense of how much they covered in this slide, and so in low resolution, they were only covering 60% of the slide. And in medium [00:04:00] resolution, they were covering like around 40%. That could generate a lot of problems. For the resident, what we found was that for the ones that arrived at the correct diagnosis or differential diagnosis. They also employ this strategy in that they'll look at the low magnification, select specific areas, and just zoom on in them. Whereas the ones that didn't get the correct response, they would zoom around like they were looking for clues, and that is very cognitive demanding. We call it the fishing expedition strategy. They always got the decision wrong; probably because there was so much detail in the [00:05:00] image at medium and high resolution. It just overwhelmed their cognitive processes. We all have a working memory that can hold between three and seven items at the same time. The more complex the items are, the fewer it can hold. So if you are looking at medium and high resolution, you are talking about a lot of complex detail, so it immediately overwhelms your cognitive system. What we observed was that the more they didn't know where to go, the less likely was for them to arrive at a correct diagnosis. 

Christine Ko: Yeah. The novice who generally will not come up with the right diagnosis; they [00:06:00] are going at higher power too soon, in too many areas, and it's overwhelming, in a way. And so you lose the thread of what you really need to do, whereas someone with more experience or the expert does somewhat scan the whole thing and then knows exactly where to look and go to a higher magnification, and zooms in then, and gets to the right diagnosis.

Claudia Mello-Thoms: Yes. The expert uses the zoom very sparingly. When the pathologist and the pathology residents were correct, they would pick up some signal from the lowest resolution image, and they would then zoom in, down like in a column, browsing the image at medium or high resolution. 

Christine Ko: Okay. So is the recommendation then to look at the [00:07:00] lower magnification image more comprehensively first? 

Claudia Mello-Thoms: Yes. Yes, definitely. That was what was suggested by our results. 

Christine Ko: And you don't get overwhelmed in the same way because there's still a lot of information in the lowest magnification, right?

Claudia Mello-Thoms: But you get context in the lower, which you don't necessarily get as much as you zoom in inside the image. The deeper you go the less context you have. 

Christine Ko: Yes, absolutely. The recommendation then is scan the image or the whole slide image at the lower magnification view, and get a context, and then zoom in.

Claudia Mello-Thoms: Yes, definitely. During the low magnification examination of the image, you get the sense [00:08:00] of what's not exactly right. I can't claim that everything is going to work this way. But then from most cases. Then, at those locations, you can go and zoom in. But don't do that everywhere in the image because that is not going to be helpful. 

Christine Ko: Okay. And did you look at different levels of experience?

Claudia Mello-Thoms: Yes.

Christine Ko: As related to that? 

Claudia Mello-Thoms: Yes, we did. We looked at histopathologist, cytopathologist, and residents like first and second years versus third and fourth years. We observed that the first and second year residents tended to use the fishing expedition strategy that we talked about, whereas the third and 40 year residents were [00:09:00] more economical in their search strategy. They would follow more the strategy followed by the expert, which was to examine the low resolution image in more detail, and then zoom in at selected areas.

Christine Ko: It's interesting because it's still a combination of zooming in but that initial search of the entire image is apparently really important. And when you have less experience, you think you're doing it, but you just choose something and look at it at higher resolution too quickly.

Claudia Mello-Thoms: That's an excellent point because there are two components needed to acquire expertise. One is to develop efficient visual search [00:10:00] strategy. Initially learners look everywhere in the image because they don't really know what they're looking at. But as they progress through their residency, they start selectively understanding what they're looking at, so they don't look everywhere. Experts have 90% of the image in their minds already, even before they first look at it. So all that they do is to capture a difference signal between what they expected to see and what they actually see. That's why experts are so much faster and so much more precise than learners.

Christine Ko: Wow. Okay. What you just said is [00:11:00] fascinating to me. So you're saying that before I even look at something, if I'm an expert, I already am predicting what I'm gonna see, and I then, maybe without maybe realizing it or maybe not, comparing what I see immediately with what is stored in my memory.

Claudia Mello-Thoms: Yes. Yes. 

Christine Ko: Okay. 

Claudia Mello-Thoms: You have a template, and you contrast everything with that template.

Christine Ko: This goes along with what I've been reading in emotions research as well. Where they say that actually my body already knows. 

Claudia Mello-Thoms: I've only studied it in medical imaging, but I do know that in medical imaging, we've observed this behavior in [00:12:00] pathologists, in radiologists, in dermatologists. The way we analyze any image, right? We're all humans. We have the same visual system, so it works for physicians, non physicians, everybody. Our visual system only has one high focus, high resolution area, which sits at the center of our eyes, the fovea. So in order for you to look at a brand new image that you've never seen before, like a piece of art, abstract art. You have to direct your fovea all through the piece, so your brain can form an idea of what is there, right? Taking into account how long information takes to go from the eyes to [00:13:00] the visual cortex, from the working memory to long-term memory to short-term memory and whatever, the only thing that justified that speed was if they already had a template, and all they were doing was capturing a difference signal. So I'll give you an example. My mentors flashed for a quarter of a second a chest radiograph to a chest radiologist. There were like 20 chest radiographs in this experiment. They asked the radiologist what they had seen, and they laughed. And they're like, Oh, we didn't see anything! And they were like, No. Tell us what you saw. And they all detected the lung nodule that was present, describing, Oh, I saw a lung [00:14:00] nodule here, a broken rib there, and whatever. The most experienced radiologist in the room, who had been practicing for more than 30 years, he could describe every single abnormality on every single case, which would be impossible to do if he had to visually search the image. One more interesting fact was that one radiograph had a huge bullet in the shoulder, but nobody saw that because your brain is not expecting to see a bullet. So the only way you are going to detect a bullet is by visually searching that area. It doesn't match your template at the quarter of a second. [00:15:00] So that's where this idea of templates came from.

Christine Ko: But the experienced radiologist who had been practicing for 30 years did detect that bullet.

Claudia Mello-Thoms: No, he didn't.

Christine Ko: He also did not.

Claudia Mello-Thoms: His templates were formed around chest disease, like broken ribs, lung nodules, pneumonia, whatever. So none of his templates had bullets.

Christine Ko: So he detected all of the chest diagnoses, but not this bullet, which probably is much more prominent than some of these other findings. Is that true? 

Claudia Mello-Thoms: I think if they had done the experiment with novices, the novices would not have been able to see half of the abnormalities present because they just didn't have the templates. But in order to see the bullet, you would have to visually scan [00:16:00] the image to detect that finding. Yeah. 

Christine Ko: So if it wasn't a proscribed experiment where the image is just being flashed for a quarter of a second, the 30 year expert probably would've scanned it and probably would've zoomed into the bullet, and maybe if there's other chest pathology, but the one quarter of a second was too quick. You're saying the expert had to compare with the templates that they had in their mind, and none of the templates included a bullet in the shoulder area.

Claudia Mello-Thoms: Exactly. Exactly. 

Christine Ko: You said that was a quarter of a second, that experiment. Is there a typical time, average time, that it takes for an image to get to the fovea and be processed? 

Claudia Mello-Thoms: The second the image appears, already some part of it is in the fovea. One of our studies showed that in mammograms, when there is a breast cancer present, the [00:17:00] eyes of the radiologist, like they could have started anywhere in the mammogram. They jumped to the location of the breast cancer within one second of the image appearing on the display. And more than that, the longer they take to fixate the breast cancer for the first time, the less likely it is that they'll recognize it.

Christine Ko: You are saying that we look at an image, at least when you've looked at pathologists, radiologists, dermatologists, people who make diagnoses based on visual information, that we look at something and we're comparing with a template that we have in our mind because the speed at which we can recognize abnormalities is faster than the time it would take to truly [00:18:00] visually process what we're looking at.

Claudia Mello-Thoms: Yes. Yes. That's absolutely correct. And the way we interpret it is that as you acquire expertise, you refine this internal template about what the image should look like. And so when you look at the actual image in real life, all that you do is you collect the difference signal between what you expected to see and what is actually present, right? This is what experts do, and that's why experts are so fast at coming up to a conclusion about, what is in the case. Novices on the other hand, like trainees, residents, they don't have that template. So they have to [00:19:00] visually inspect the image to get an overall sense of what is being shown.

So not only that takes much longer, but that also makes them prone to more mistakes because it looking at an image like a chest radiograph takes, a big amount of cognitive resources. It takes a huge amount of cognitive resources, and they start dropping diagnostic hypothesis as they go along because they just cannot keep those many in their working memory. Whereas the expert, as he or she is looking through the stack, if you are talking about the chest CT, they're, Okay, I know what I'm gonna see [00:20:00] next, and next, and when that next doesn't match, I go to that location, I figure out what's wrong. My diagnosis is made, and I'm done.

Christine Ko: So that makes sense, what you just said, for a chest CT, which has a stack of images as you're saying, like multiple images that you're gonna have to look at, as if you see the tail of an animal before you see the rest of it, you can predict, Okay, if it's a certain type of tail, I'm gonna see a dog, the rest of the dog. But what about for a microscopic slide where it's just one slide? Or if you capture the microscopic slide and it's one image, virtually that you show. How would I have any idea what I'm about to see?

Thank you for listening to this point. Because this conversation has been so amazing to me, we are gonna take a break, and it will continue next [00:21:00] episode. Thank you.