Federica Santoro 0:15

Welcome to Drug Safety Matters, a podcast by Uppsala Monitoring Center where we explore current issues in pharmacovigilance and patient safety. How do you find a needle in a haystack? Modern pharmacovigilance databases contain tons of information. And that is especially true of Vigibase, WHO's global database of reported potential side effects of medicines and vaccines, which we maintain here at UMC. By 2022, Vigibase is expected to surpass 30 million reports. How do we see through such massive amounts of data? And more importantly, how do we cut through the noise and make sure we're picking up the true warning signs of a safety issue, or true signals, as they're known in the field? I'm your host, Federica Santoro, and I recently invited two of my colleagues to the studio: data scientist Jim Barrett and pharmacovigilance scientist Joe Mitchell. We talked about which statistical methods have been traditionally used to identify signals, UMC's new clustering algorithm VG Group, and how it can improve signal detection, and finally how that method has helped us monitor the safety of COVID-19 vaccines so far. I hope you enjoy our conversation. Well, welcome to Drug Safety Matters, Jim and Jo. It's an honor to have you both on the show today. Thanks for joining. Now, our loyal listeners will know that I like to start with the basics. So, Jim, statistical methods for signal detection have been around for a while. Can you walk us through the most common approaches to look at big collections of pharmacovigilance data and especially what their limitations are?

SPEAKER_00 2:22

Yeah, absolutely. So one of the key problems we're trying to solve when employing statistical methods for signal detection is to basically filter through the huge amounts of data that we get. For example, VigiBase now has something like 29 million reports in it, which is far too many for humans to go through the cases one by one to look for problems that are happening. So we employ statistical methods to kind of try and find the needles in the haystack, as it were. So the uh most commonly employed method in large databases is uh called disproportionality analysis. And how this works is when we receive adverse event reports, the things like the adverse events or diagnoses are coded to a standard dictionary of terms. This is called the Medra dictionary. And we basically look for terms in this dictionary, uh, reactions and symptoms that people are experiencing more commonly than we would expect in the background when associated with a particular drug. And basically what we do is uh look for those reactions which are occurring more often than we expect them to. And we filter on those to pass those over to clinical experts to then assess in more detail as to whether they're uh a real relationship that we need to worry about. So you asked about the limitations when it comes to disproportionality analysis, and the key limitation is that disproportionality analysis works looking at individual symptoms or diagnoses as they relate to drugs. And uh individual symptoms are not necessarily sufficient to characterize a particular clinical condition that people are experiencing. So I'm thinking particularly in terms of things like syndromes, which are kind of constellations of symptoms that people are experiencing that together constitute a clinical condition. Moreover, and another drawback that can occur is some terms are kind of vague in the sense that they can apply to multiple clinical conditions. For example, if we were to consider one term is uh nausea. So nausea could be associated with other terms like migraine and aura and stuff like that for a patient's experiencing migraines, or it could be associated with terms like uh abdominal pain, diarrhea, and constitute gastrointestinal issues. So the same terms can be used in multiple contexts, so that can kind of muddy the water in terms of looking at individual drug event combinations.

Federica Santoro 4:38

Right. And so, because of these limitations you described, you and your colleagues here at UMC recently developed a new method called Vigigroup. So, first of all, why that name? And secondly, how does the method work?

SPEAKER_00 4:53

Yes, indeed. So Vigigroup is a method that we've developed as part of a suite of methods that we work on here at the Uppsala Monitoring Center that help facilitate uh statistical signal analysis. So that's where the VIGI part of the name comes from. Uh the group part is because the goal of the method is to group together adverse event reports which uh relate to the same clinical picture. So as I mentioned before with disproportionality analysis, which looks at the level of drugs to events, like single terms to drugs. Vigigroup instead looks at the level of reports and tries to find groups of adverse event reports with all of the reported symptoms that are related to one another. I won't go into too much detail into how it works, but we use a technique called a probabilistic model, which is a type of mathematical model which allows us to make deductions about our data. So the key deductions that we try to make when looking at adverse event reports are how likely is it that certain terms are reported uh with one another, like the same report having two different terms, how how likely are they to be reported to one another? And also we like to deduce, given two reports and the set of terms that they have on them, how likely are they to be part of the same clinical picture? So, yes, so we use this probabilistic model as some mathematical machinery to try and deduce these things about our data set.

Federica Santoro 6:12

Thanks for sparing us the gory mathematical details. Uh, but how does the algorithm know which terms are clinically similar? And so which reports should be grouped together? Is there a risk that you'll end up grouping reports that don't really have anything to do with each other?

SPEAKER_00 6:31

So um one of the interesting things about the method actually, and an important thing to note, is that it doesn't know a priori which terms are clinically similar to one another. It simply looks at which terms are reported in conjunction with one another and what kind of patterns in terms we're seeing reported more commonly or less commonly in our data set. So through looking at terms which are reported like in similar situations to one another, then we can determine terms which are clinically similar to one another, although that's more of a byproduct of the method than anything else. So the method is more interested in looking at which reports are clinically similar, where they're reporting the same patterns. Now you ask the is there some risk of grouping reports together which are not relating to the same thing? And of course, I mean the method is not perfect, no method is perfect, which is exactly why we simply use it as a kind of statistical method to screen our data and find possible associations, possible links, um, and then pass those on to the clinical experts, the pharmacovigilance scientists who um can look at the associations in greater depth or use them as inspiration to try and figure out what the safety profile of a drug might be?

Federica Santoro 7:46

Can Vidigroup be applied to all reports in Vigibase? Because obviously you're talking about looking at multiple adverse event terms. How many reports in VigiBase contain multiple terms as opposed to single terms that could be looked at with disproportionality alone?

SPEAKER_00 8:07

Yeah, so you're absolutely right that you can't make a pattern of one term on a report. So uh, as of this morning when I checked, there was 29 million reports in Vigibase, of which a little over 13 million consisted of only a single reported term. So that means that we can apply Vigigroup to a little over half of the current reports that we have in Vigibase.

Federica Santoro 8:30

You mentioned Medra earlier, the medical dictionary for regulatory activities. And I assume many of our listeners will be familiar with that and maybe wondering how Vigigroup compares to standardized Medra queries or SMQs. Because they were also created with the aim to aid signal detection and group reports that contain terms that describe the same clinical condition. So how would you say Vidigroup differs?

SPEAKER_00 9:02

So the key difference between what Vigigroup does and what the standardized MEDRA queries do is that the standardized MEDRA queries have to be defined upfront by somebody with clinical expertise, like relating to something that we expect to find, or a kind of expected grouping of terms. Whereas Vigigroup is totally agnostic to what groups of terms or what kind of patterns we expect to find in our data set. So an excellent example of this is actually, as we're all aware, there was this uh signal of thrombosis and thrombocytopenia syndrome that occurred with the COVID-19 vaccines. And there was no standardized MEDRA query for thrombosis, thrombocytopenia syndrome, because it's not something that we were expecting. It's not a constellation of symptoms that we necessarily expected. Whereas Vigigroup would be able to pick up unexpected patterns like this because it's agnostic, it just looks for what's in the data. There's no pre-knowledge going in there.

Federica Santoro 9:54

Thank you, Jim. And now over to you, Joe. So once Vigigroup had been tested on pre-existing Vigi-based data, you guys decided to put it to real use, so to say, and try to identify new safety concerns as they emerged. And of course, with COVID-19 vaccinations in full swing, it's no surprise that you decided to turn your eyes to those products. Joe, you came in with your expertise in medicine and the pharmacovigilance. Why did you think Vidigroup would be particularly useful for COVID-19 vaccine safety monitoring and what was the process like?

SPEAKER_01 10:32

Yeah, so as Jim already alluded to, with the COVID-19 vaccines, a lot of what we were going into was the unknown. We didn't there was use of new technologies within the vaccines. And as part of that, we wanted to sort of supplement the traditional disproportionality statistical screening of VigiBase by by sort of seeing what is being reported together consistently throughout these new COVID-19 vaccine reports, with the eye of looking for something either like a a known clinical syndrome or a new possible syndrome or just what constellation of symptoms are being reported together as an extra way of detecting those things to sort of try address some of the inherent weaknesses of screening just by the individual term level. We were able to try to create a more fuller picture of what's actually being reported together.

Federica Santoro 11:30

Capture the unknown and the complexity, perhaps.

SPEAKER_01 11:33

Yep.

Federica Santoro 11:34

You were tasked with reviewing the clusters of reports that Jim and his colleagues were pulling out of VigiBase with Vigi Group. What were you looking for? I'm thinking what would make you sort of stop in your tracks and think, okay, I really need to look into this.

SPEAKER_01 11:50

Yeah, so as we were sort of going through the clusters, what my sort of thought process was to try and look at the individual clusters and then sort of see if there was an obvious diagnosis that was attributable to these clusters, and then compare that to the listed adverse effects that are known to the COVID-19 vaccines. And then on top of that, we'd sort of supplement that with the information that could be automatically derived from the data that Jim and colleagues were sort of able to provide whilst we were looking at these clusters. So for example, we were able to look at measurements of how many of these reports were deemed serious by the reporter, how many of them were reported after which doses, and the time to onset of the reporting as well.

Federica Santoro 12:41

Is there a difference in the way you would assess a case series highlighted with a clustering algorithm like Vigigroup compared to one you would pull out with disproportionality?

SPEAKER_01 12:54

So there isn't a difference in how you would assess the case series to start with. So when once we identified a cluster that we deemed was of interest, either because it was something that was unexpected, something that was not listed within the adverse events related to those vaccines, or because it had a particularly interesting rate of seriousness or a different time to onset that we were expecting, then we would start as we would do normally looking through a case series and go through those cases one by one and uh try piece the information together that way. But what we did find was that we could then use the information that we'd get from the clusters. And for example, if there was a diagnosis that we could attribute to that cluster, we could then search within Vigibase for that diagnosis or other synonyms that are related to that diagnosis, and then use that to check the more traditional features such as disproportionality measurements, and then also to use that to collect more cases for the case series as well, and then vice versa. If something was found of interest within the more traditional disproportionality methods, then they could then look at the clusters and then use that to identify new cases and use that as a support to the traditional methods.

Federica Santoro 14:16

Let's talk results. Could Vidigroup identify any of the known side effects of COVID-19 vaccines?

SPEAKER_01 14:24

Yeah, so as Jim said, no technique is perfect, but uh we were able to identify both the known and expected side effects of the vaccines. And then on top of that, we also noticed that as the reporting developed, we were seeing clusters that related to the adverse events that sort of became more apparent after the initial marketing of the vaccines. So, for a good example of that, we saw clusters related to localized delayed reactions around the vaccination site or myocarditis and inflammation of the heart muscles, which have been sort of discussed at length by other stakeholders and now are recognized as side effects for some of those vaccines.

Federica Santoro 15:11

What about new signals? Did you find any?

SPEAKER_01 15:14

Yeah, so um using this method, we we have identified signals which were new. I very briefly touched upon the delayed local skin reaction, which was uh primarily seen after the Moderna vaccine. And then on top of that, the clustering method at Vigigroup has been useful in identifying cases and identifying the diagnosis for other signals such as a short-term hearing loss and associated tinnitus and uh appendicitis. But just because these signals were identified using Vigigroup doesn't necessarily mean that we think that there is a that the vaccines are causing these reactions to happen. It's more that we are consider them to be areas of interest that require further research and further investigation.

Federica Santoro 16:04

Absolutely. Should we talk a little bit about the future of the method? How do you guys see disproportionality analysis and clustering algorithms like Vigigroup used in the future? So can you imagine situations in which one or the other method will prove most valuable? Who wants to take this one?

SPEAKER_00 16:26

So, um, in terms of how I see that being used in the future, I think that both methods are actually excellent complements to one another. And we're hoping that Vigigroup will become a standard tool in the tool belt of our uh pharmacovigilance experts here in performing their kind of signal detection duties in the future. Now, in terms of whether there are any scenarios in which one method is is better than another, potentially, so disproportionality analysis has proved itself to be very effective, especially when you have a very specific term or diagnosis that you're searching for. Whereas Vigigroup tends to come into its own when the analysis is more exploratory or or looking for unexpected signals or previously unknown things. But to reiterate, um, I think the strength of Vigigroup is that it kind of plugs the gaps that disproportionality analysis has in it. So it both methods can be used in tandem with one another to kind of uh lift both up.

Federica Santoro 17:22

And complementing each other. We got a couple of questions from one of our listeners. Pharmacist Josue in Mexico wonders first of all, if Fiji Group can be implemented in small databases. And secondly, can it be useful in determining causality?

SPEAKER_00 17:42

So to answer the question of whether it can be employed for small databases, it depends a little bit on how small we're talking. From our experience in employing the method with the COVID vaccines and other medications, we started seeing uh sensible results out of the method after just a few thousand reports.

Federica Santoro 17:59

How about uh whether it can be used to determine causality? Joe, you want to take that one?

SPEAKER_01 18:05

Um so Fiji group and other clustering methods can't or shouldn't be used as a replacement to the manual assessment of case series. It's more of a way of identifying interesting case series and interesting groups of terms that are being reported together that then still require that manual assessment to sort of to corroborate that information and draw it all together.

Federica Santoro 18:29

Yeah, and that's an important thing to stress, right? The statistical methods, I guess, will never replace manual assessment. So they're only a way, as you say, to highlight, to bring about hypotheses that then need to be further investigated and looked at. I'm glad actually Hosue asked about using Vigigroup in small databases or in any other pharmacovigilance database, really, because a few of our listeners might be wondering if they can use Vigigroup at all. What would you say to that?

SPEAKER_00 19:00

So um Vigigroup is a very new method. We literally only just published it a few weeks ago, and um it's a method that we're still actively developing and and uh understanding internally exactly what its strengths and weaknesses are. It's a very promising method, but we're still ironing some kinks out of it. So the implementation that we have is currently only accessible to people within the Uppsala Monitoring Center. That being said, um we very much hope to make it more widely available in the future.

Federica Santoro 19:29

Okay, I'm letting you off the hook soon, but before you go, I was curious what did you learn from this collaboration? Because Jim, you're a data scientist, a former astrophysicist, you could say, and uh Joe, you're a clinician and a pharmacovigilance scientist. So how did this collaboration work out? And what advice would you give to other people trying to set up similar interdisciplinary projects?

SPEAKER_01 19:57

So I think just to embrace each other's skills and use their strengths to their benefit, and just because if an area is a particular weakness of yours, then you can plug that gap almost by having a teammate who is better in that area. So, for example, my poor data science knowledge and Jim's excellent data science knowledge.

SPEAKER_00 20:19

I think one of the great strengths of how we used this method in tandem for the COVID-19 vaccines is that we worked very closely with one another. We would have like uh weekly meetings or bi-weekly meetings. Uh, at the beginning, uh, when we first started employing this method on the COVID-19 vaccines, Vigroup and the kind of data presentation tools we had surrounding it were very rudimentary, very basic. But they formed what we would call in software and data science is a minimum viable product, which we were then able to iterate upon by getting regular feedback from the pharmacovigilance experts. So, for example, um, one week they might say, Oh, it would be really, really useful if we um had time to onset information for all of these clusters that you're showing us. So then uh me and my data science colleagues would go off that week, implement this uh that viewing the time to onset for a certain cluster. And then the next week we have the next meeting with the pharmacovigilance experts where they're like, that's fantastic. But now actually we'd like to see which uh dose these uh clusters. And iterating like this and working very closely together and implementing feedback quickly in a way that we can get immediate feedback was very valuable actually for making this project a success.

Federica Santoro 21:31

That was lovely. Thank you both for taking the time to talk to me. It was a real pleasure to have you on the show.

SPEAKER_00 21:36

Thanks for having us. Thank you.

Federica Santoro 21:43

That's all for now, but we'll be back soon with more conversations on medicine safety. If you'd like to know more about Vigi Group or UMC's work on COVID-19 vaccine safety monitoring, check out the episode show notes for useful links. And don't forget to browse through our archive for more episodes on vaccines and more interviews with UMC staff. If you like our podcast, subscribe to it in your favorite player so you won't miss an episode. And spread the word on social media so other listeners can find us. Apart from these in-depth conversations with experts, we also host a series called Uppsala Reports Long Reads, a selection of audio stories from UMC's pharmacovigilance magazine. So do check those out too. Uppsala Monitoring Center is on Facebook, LinkedIn, and Twitter, and we'd love to hear from you. Send us comments or suggestions for the show, or send in questions for our guests next time we open up for that. For Drug Safety Matters, I'm Federica Santoro. I'd like to thank Jim Barrett and Joe Mitchell for their time, Matthew Barwick for production support, listener Josué for contributing questions, and of course you for tuning in. Till next time.