Drug Safety Matters brings you the best stories from the world of pharmacovigilance. Through in-depth interviews with our guests, we cover new research and trends, and explore the most pressing issues in medicines safety today. Produced by Uppsala Monitoring Centre, the WHO Collaborating Centre for International Drug Monitoring.
The views and opinions expressed in the podcast are those of the hosts and guests respectively and, unless otherwise stated, do not represent the position of any institution to which they are affiliated.
Most pharmacovigilance professionals will have heard of masking – a statistical issue where reports for one drug hide signals for other drugs. But the problem gained fresh attention when record amounts of reports began piling up for the COVID-19 vaccines. How should we be unmasking data in the COVID-19 vaccine era?
This episode is part of the Uppsala Reports Long Reads series – the most topical stories from UMC’s pharmacovigilance magazine, brought to you in audio format. Find the original article here.
After the read, we speak to data scientist Sara Vidlin, who authored the article, to learn more about masking and how to deal with it.
Tune in to find out:
How masking evolves with the data
Which methods can be used to unmask data
What other pitfalls to watch out for when performing quantitative analyses
Got a story to share? We’re always looking for new topics and interesting voices. If you have an idea or any other feedback for the show, get in touch!
First came the COVID-19 pandemic. Then came the COVID-19 vaccines. And then came millions of reports of suspected side effects of the vaccines. What are the implications of that for data analysis? My name is Federica Santoro, and this is Drug Safety Matters, a podcast by Uppsala Monitoring Center, where we explore current issues in pharmacovigilance and patient safety. This episode is part of the Uppsala Reports Long Reads series, where we select the most topical stories from our magazine, Uppsala Reports, and bring them to you in audio format. Today's article is Unmasking Data in the COVID-19 vaccine era, written by UMC data scientist Sara Vidlin and published online in November 2022. After the read, we take a peek behind the scenes as I chat to Sarah about her research. So stay tuned until the end. But first, let's hear the article, read by Sarah herself.
Both regulators and the World Health Organization advocated for close monitoring and reporting of any potential adverse events, and this call was certainly adhered to. During 2021, 2.9 million reports flooded into VigiBase, the WHO global database of reported potential side effects of medicinal products. An additional 1.7 million reports were entered during the first 10 months of 2022. However, while these reports form a crucial asset in supporting the safe use of the vaccines, such a massive influx of reports does create an imbalance within Vigibase. COVID 19 vaccine is now by far the most commonly reported medicinal product in Vigibase, with 4.6 million reports. That is 13% of the entire database. In a modest second place comes the monoclonal antibody Adalimumab with 650,000 reports, followed by the recombinant protein Eternocept taking the third spot on the podium with 570,000 reports. And both Adalimab and Eternocept have been on the market for over 20 years. Overall, the response to the call for reporting adverse events following COVID-19 vaccination must be considered a success. Nevertheless, when a single medicinal product makes up such a large proportion of the entire database, it will affect statistical signal detection, even when performed for other drugs. Let's take Bell's pose, a type of facial paralysis as an example. 87% of the 13,000 entries in Vidge base comes from COVID-19 vaccine reports. It doesn't matter which measure of disproportionality you use, the expected accounts for Bell's posy will be inflated massively by the COVID-19 vaccine reports, which means that it will not appear as disproportionate for other drugs as long as the COVID-19 vaccine reports are in the way and hide the statistical signal. This is an effect called masking, also known as cloaking or competition bias. Masking is not a new problem in pharmacovigilance. In 2013, UMC developed a method to uncover masking by identifying influential outliers. Last year, 2022, we applied this method to all adverse drug reactions that have been reported for COVID-19 vaccine reports. What it basically does is compare the frequencies of a reaction with and without COVID-19 vaccine reports. If we see an increase by at least 50%, we consider the reaction masked. The method also applies statistical shrinkage to the calculation to get a more robust measure, limit the effects of random variability, and shift focus away from reactions that are overall very rarely reported in the database. Using this threshold, we identified 132 adverse drug reactions as masked by COVID-19 vaccine reports. Of these, 21 are on the European Medicines Agency's list of important medical events. Examples of reactions we see masked are sweating fever, bals palsy, vaccine breakthrough infection, hemoparesthesia, myocarditis and pericarditis. And notably, for some of these reactions, a majority of the reports in BGBase now relate to COVID-19 vaccine reports, as in the case with our example, Balse palsy. In a very interesting study made in VARS, the US FDA vaccine adverse event reporting system published in july 2022 in drug safety by Harpas and colleagues, we can see the opposite problem. How statistical detection of early COVID-19 vaccine signals in the very beginning of the vaccine rollout were delayed because of masking due to other drugs. For example, that study found that the statistical signals of myoditis with Pfizer and the Moderna vaccines were masked by a smallpox vaccine. As more data accumulated, the Pfizer and Moderna vaccines then started to mask the signal from each other. This is also a very interesting piece of the puzzle, understanding that masking is not a static phenomenon, but something that evolves with the data. This study also teaches us that it might be especially important to consider masking when a new drug is introduced, or as long as there are few reports on that drug. Now the CISO have flipped for the COVID-19 vaccines. Signals are no longer masked from them, but due to them. So where do we stand now? Knowing that 13% of VGBase is made up of COVID-19 vaccine reports. The large proportion of COVID-19 vaccine reports may hide or delay the detection of statistical signals of many adverse drug reactions when calculating disproportionality for other drugs. This is a problem that must be addressed. One way is to simply remove all COVID-19 vaccine reports when doing statistical signal detection for other drugs. Another way is to perform a method of unmasking, like the simple unmasking suggested by Yulin and colleagues in 2013, which identifies any outliers and carries out a parallel analysis after they have been omitted. Regression analysis has also been proposed as a method to deal with this issue. In the end though, whichever method we choose is not that important. What is important is that we stay aware of the problem of masking due to COVID-19 vaccines as well as due to other drugs and do something.
So, first off, I have a question about the causes of masking. You write that masking is not a new problem in pharmacovigilance, but I guess the community had not experienced such an avalanche of data before. Or are there other factors apart from the volume of reports that can cause masking?
SPEAKER_01
Well, it it is all about counts in some sense, but it doesn't have to be extreme counts, as in the COVID-19 vaccine case. If we're talking about a rare reaction or a rare drug, the counts can be relatively high despite being in absolute numbers, just hundreds or thousands. But the phenomenon masking is all about counts. It is observed counts versus expected counts. And if the expected counts get inflated due to other reactions or drugs, we see masking.
Now, as you say in the article, masking poses two types of problems. The first one is that those millions of reports for COVID-19 vaccines can end up masking signals for other medical products. Tell me more about this.
SPEAKER_01
Yes. When we do this proportionality analysis, it is based on observed over expected calculation. We have an observed number of reports for a drug event combination, and we want to know is this higher or lower than the expected. And the expected value that is just an estimation that we make. It is an estimation of how many reports we expect to see if there is no association between the drug and the event. And this expected estimation is calculated from the number of reports for the drug, the number of reports for the reaction, and the total number of reports in the database. And those three are the elements that we use to estimate what is expected for this combination. And it is not an exact truth. It is a quite crude estimation that works overall quite well, most of the time, but not always. So when estimating this expected value, we assume that different drugs and different reactions are reported in kind of a similar fashion. For example, we know that we have huge under-reporting when it comes to spontaneous data like in InvigiBase. And when doing disproportionality analysis, which do come down to some kind of observed over-expected calculation, we assume that the under-reporting for the reaction is somewhat similar across all drugs. And we assume that the under-reporting for the drug is somewhat similar across the reactions. And if it is enough similar, it will cancel out. And the same goes for other biases in the data. If it is enough similar across the reactions and drugs, the biases will cancel out. Thing is, for the COVID-19 vaccine reports and other drugs with intensive monitoring, there is less underreporting. So the under-reporting for adverse events from COVID-19 vaccines and from, say, prednisolone that has been on the market for a very long time, it is not even close to similar. So this basic assumption for disproportionality analysis, it does not hold in this case. So if we take the example with the reaction Bell's palsy again, we have 13,000 reports in media base. If we would remove COVID-19 vaccine reports, this number would be eight times less. So divided by eight. For the drug prednisillon with Bell's palsy, the expected count is 27. But if we remove COVID-19 vaccine reports from the calculation, the expected counts would be divided by eight. So the expected counts for Bell's palsy with pregnicillon would be around three instead of 27. And that is quite the difference. This is masking. The statistical signal for Bell's palsy with pregnicillon is masked by the COVID-19 vaccine reports because the COVID-19 vaccine reports inflate the expected.
And that's already a big problem per se. And then there's a second issue that you bring up in the article and that was highlighted by the US FDA. And that's that COVID-19 vaccine signals themselves can be masked by other reports in the database. So let's talk a little bit about that now.
SPEAKER_01
Yeah. We can take the reaction myocarditis as a good example. So in the beginning of the pandemic, when there were still very few COVID-19 vaccine reports, myocarditis was mainly reported for smallpox vaccines in VARES. Smallpox reports masked the initial statistical signal of myocarditis with Pfizer and Moderna vaccines. So what that means is that it required more reports for the statistical signal to show, which means a delay, of course. And if it weren't for the smallpox myocarditis reports, the expected value for both myocarditis with the Pfizer vaccine and myocarditis with the Moderna vaccine would be lower. So the expected counts would be lower, and the statistical signals would have been seen earlier. Of course, now the numbers have flipped dramatically. Now it is the Moderna and Pfizer vaccines that drives up the expected counts for myocarditis, which might now mask it for other vaccines in Bears.
Federica Santoro
So a constantly evolving issue, and we'll go back to that and how the problem is influenced by the nature of the data.
We've talked about the problems now. Let's move on to the solutions. So is the solution as simple as encouraging pharmacovigilance professionals who are currently using, so to say, standard disproportionality methods to adopt more advanced methods that can deal with masking.
SPEAKER_01
Well, the simplest way to address masking by COVID-19 vaccines is to simply remove vaccines from the background when doing statistical signal detection for drugs. This can now be done in Vigilis, for example, by restricting the therapeutic scope to just non-vaccines. However, this doesn't address masking that is not caused by COVID-19 vaccines. So to address masking overall, you would need to either identify influential outlier combinations, so the extremes in the database and remove them before performing dysproportionality analysis. Or you would need to use a more advanced method such as regression. Regression can be designed to produce dysproportionality measures with kind of adjusted background rates that can control for
But of course, before people can act on an issue, they need to know about it. And your article and this podcast, obviously, are a first step towards raising awareness of masking. How else are you planning to bring attention to the problem?
SPEAKER_01
I think it is quite known issue since long. It is nothing new to PB, but of course, the COVID-19 vaccine reporting has been putting the spotlight on the problem recently. So, apart from this article in Uppsala reports, we also presented this at ICPE last year, so the International Conference on Pharmacoepidemiology. And the issue was also highlighted at a workshop led by WHO and UMC about a year ago. And how to deal with masking, it is actively being discussed right now. For example, at the EMA, the European Medicines Agency, and at several European national centres, some of which have been undertaking and some of which are about to undertake similar studies as we did, but in their respective datasets. So I think that awareness is quite high, and I hope that more insights will come. But it is, of course, important to keep this discussion alive.
And aside from masking, what are some of the other issues pharmacovigilance professionals should keep in mind when performing quantitative analyses?
SPEAKER_01
Well, one fundamental thing to always keep in mind is confounding. Confounding due to the underlying disease, confounding due to co-medications, confounding due to demographics. For example, just because a reaction is often reported with the drug, it doesn't necessarily mean that there is a causal relationship, of course. The reaction might be a symptom of the underlying disease that the drug is used to treat, or it might be an effect of another drug that is often co-prescribed with the drug of interest. It might also be confounded by demographics, for example, age. I mean if if a drug is mainly given to elderly, coincidental events that is completely unrelated to the drug will be higher than for a younger population. So that is one thing confounding. But we must also stay aware of the under-reporting that we have been talking about, that we know is a fact. And also keep in mind that the under-reporting might change with drugs, it might change with reactions, and it might change with time. If a problem gets media attention, for example, the under-reporting will be less, at least for a while. Drugs on special monitoring and new drugs also have less underreporting than old drugs. And it also differs depending on if it is an over-the-counter drug or a prescription drug. And when it comes to reactions, some reactions are more underreported than others due to severeness or seriousness, for example. Another thing yet to keep in mind are some more complex reactions or syndromes with fussy symptoms or several symptoms. They might be reported and coded with different terms. So the reaction is kind of diluted and spread out across several reported terms. So each term individually might not stand out as a statistical signal, but if we would combine them, they might. This of course complicates the picture. A reaction can also be captured or coded differently depending on the reporter of the report. If it is a healthcare professional or a consumer, for example, and we also see different coding choices in different countries, possibly even different regions within a larger country. Different countries also have different approaches of what they share in PidgeBase. Some share everything, some only share, for example, robots that are serious or with serious reactions. And this we must also keep in mind when looking at a combination and trying to understand it. And there are so many factors influencing the reporting and the counts we get in the end. And this, of course, leads to a very important conclusion, and that is that statistical signals should never, no matter how advanced the method is to calculate them, they should never be the sole evidence for a signal. Statistical signals should always be analyzed in in conjunction with a comprehensive qualitative assessment, including like the clinical review of the cases and other available sources of evidence.
And yet, despite all these caveats you described, quantitative tools remain important in pharmacovigilance. Why is that?
SPEAKER_01
Well, if we look in Vegibase, it has over 30 million individual reports and almost 4 million unique drug event combinations. So 4 million different combinations. Quantitative analysis, it gives us a hint where to start looking, like where to focus our attention. It does not serve us the overall truth. It doesn't give us the complete picture, but it sort of points us in the right direction.
Federica Santoro
Thank you for the interesting chat. That's all I had for you today. Thanks again for taking the time to talk to me. Thanks for having me.
Send us comments or suggestions for the show, or send in questions for our guests next time we open up for that. For Drug Safety Matters, I'm Federica Santoro. I'd like to thank Sara Vidlin for her time, Matthew Barwick for production support, and of course you for tuning in. Till next time!
.jpg)
