Federica Santoro 0:01

Welcome to Drug Safety Matters, a podcast by Uppsala Monitoring Center, where we explore current issues in pharmacovigilance and patient safety. This episode is part of the Uppsala Reports Long Reads series, where we select the most topical stories from our magazine, Uppsala Reports, and bring them to you in audio format. Our pick for today is the article Found in Space. New method reveals related drug and reaction terms, which appeared in issue 82 of Uppsala Reports in April 2020. When reporting adverse reactions to drugs, people can choose from a plethora of different terms to describe what they experienced. So, how do you tell whether two individual case safety reports are similar when they don't use the same words? Lucie Gatpaille is data science team manager at Uppsala Monitoring Center. In her article for Uppsala Reports, she explains how natural language processing can help and how a journey through space could lead to a new dimension in case report analysis. After the read, you'll get your peek behind the scenes as Lucie chats about her research with Uppsala Reports Editor Jared Ross. So make sure you stay tuned till the end. But first, let's dive into the article. Here's Found in Space, written and read by Lucie Gatpaille.

SPEAKER_02 1:32

How do you tell whether two individual case safety reports are similar? This is the question I began asking myself early in my employment at Uppsala Monitoring Center. At the time, I had just come out of a two-month project for improving our duplicate detection algorithm, and some aspects of it had left me a bit unsatisfied. The gist of how our algorithm works is that it looks at different important fields of two reports, for example, age, gender, country of origin, reported reactions, reported drugs, and it computes a score that rewards matching information and penalizes mismatches, weighing in how frequent each piece of information is in the first place. When it came to the reported reactions, this approach just felt too rough. Consider how large Medra is. If two independent persons review a given case, it is not at all certain that they would choose to describe the observed reactions using the exact same set of Medra terms. If I choose blood pressure increased and you choose hypertension, are we really describing different things? To circumvent this, we could of course soften the strict match no match criterion by going up the medra hierarchy and allow for some reward when the two terms share the same high level term instead of penalizing the mismatch. But this would not solve our problem above since the term I chose blood pressure increased and the one you chose hypertension are from different system organ classes, which is a classic problem when dealing with lab results and their associated diagnosis. And conversely, there are terms in MEDRA that share a high level term, but that we would not want to match. For example, we would not want to match a case of foot amputation with a case of hand amputation when trying to identify duplicates. We needed something else. We needed a way to take relatedness of terms into account in a much more nuanced way than MEDRA allows. Obviously, even the whole staff of UMC would never be enough for manually scoring all pairs of terms for relatedness. So whatever solution we could come up with, it would have to be data driven. Thankfully, UMC has over the years gathered a team of smart data scientists and all it took was to draw a parallel between our current problem and a very similar one in a quite different domain, namely natural language processing, or NLP for short. In NLP, people have been struggling for decades with the problem of representing words in a clever way, a way that would somehow model the meaning of words. How do you tell whether two sentences are similar? Well, they're similar if the words they are made of carry similar meanings. To a linguist, the words Monday and Tuesday, however different, are similar in meaning. From an NLP perspective, they are in most cases entirely interchangeable. What NLP researchers then tried to achieve was to develop methods of producing numeric representations of words so that similar words get similar representations. With these methods, algorithms for NLP could be built upon these representations to become meaning aware, distributional semantic methods arose out of these efforts. Distributional semantic methods are best summarized by this quote by John Rupert Firth, a British linguist. You shall know a word by the company it keeps. The central idea is that if you see two words surrounded by similar contexts, that is, with a high degree of interchangeability, then these words are likely to carry a similar meaning. The approach gained great popularity in the early 2010s when a distributional semantic method called Word2Vec was published, leading to a great boost in performance of algorithms in many kinds of classical NLP tasks, such as part of speech tagging, machine translation, question answering, and document classification. In essence, what Word2Vec does is to model every word as a vector. Think simply of a list of numeric values, and learn each vector component with machine learning by trying to predict center words in context windows using massive amounts of text. Details of the method are probably quite uninteresting for the non-technical crowd, but the essential concept to understand is that the method creates a space. Imagine a 3D space, for example, where similar words have similar coordinates. In a region somewhere in the space, you might find Monday, Tuesday, and all the other days of the week, while in another region you might find cat, dog, and other kinds of pets. From here, addressing our problem of automatically finding similarities across case reports becomes quite straightforward. After all, just like a sentence provides a context for words, an individual case safety report provides a context for both reported drugs and reported preferred terms. So we applied word to VEC to more than sixteen million VG based reports, creating a novel space of meaning for both drugs and events. Preliminary quantitative and qualitative analysis showed that the resulting vectors were indeed meaningful. If you want to see for yourself, check out this article online at upsalareports.org, where you can find example tables that show lists of nearest neighbors in the space for different concepts of interest. Now, if I choose blood pressure increased and you choose hypertension, the new representations show that we are not so far from each other. All these new vector representations create a space of meaning built in an entirely data driven way based on co-reporting patterns present in Vigibase. In theory, only a human with the proper expertise can make sense of each reported term and each reported drug on a given individual case safety report. In other words, we can't expect a machine to make any sense of them because this is not what Vigibase has been designed for. Nevertheless, in the way people report drugs and reactions together in Vigibase, there is meaning in these concepts, and it is truly mind blowing. But as much as being data driven is a blessing, it is also in some respects its own curse. At the core, it is difficult to understand why the concepts get the coordinates they do, why x is the nearest neighbor of y instead of z. Evaluating the quality and soundness of the space is also a challenge because there is no clear truth to compare it to. In my attempt to validate the vector representations, I use the medra and ATC hierarchies based on the assumption that two preferred terms or two drugs that share an ancestor in the hierarchy should, in general, have more similar coordinates, and that similarity should decrease as the ancestor gets less specific. For example, we would expect two preferred terms sharing a high level term to be closer in the space than two sharing only one system organ class or none at all. And I did see the expected pattern. Overall, similarity between drugs and preferred terms decreased as we relaxed the constraints on their shared ancestries in the hierarchies. However, we see that there is still a large overlap between distributions, indicating that the relationships retained in the space are more complex and certainly go beyond the MEDRA and ATC hierarchies. We also have to keep in mind that for the algorithm to be able to build good vector representations, it requires data. So representations of rare preferred terms and drugs are likely to be noisy and less reliable. But we believe that by using these vector representations in all kinds of pharmacovigilance tasks, we will be able to prove their potential and their usefulness. Using vector representations, we now have a way, albeit not perfect, to automatically quantify relatedness between important concepts such as drugs and preferred terms. This means that we are now in a much better position to compare reports with one another, as was my original intention. So duplicate detection, report clustering, case series expansion, these are tasks that could already now benefit from this new way of representing drugs and preferred terms. But that's not all. For example, we could also improve disproportionality analysis by harnessing the power of entire neighborhoods of terms. Suggestions for coding alternatives during data entry could be integrated into Vigiflow, and since the new representations of drugs and preferred terms are numeric, they can be the basis of insightful visualizations, like adverse reaction profiles for drugs, which could be integrated and Vigilize in the future. By computing distances to labeled adverse drug reactions, we could speed up signal detection, identifying drug reaction combinations that are captured in basic form in the label information. And thanks to their numeric form and semantic content, the representations are likely to be much more powerful inputs for machine learning algorithms and statistical methods. As shown with all these examples, I truly believe that the paradigm shift in representing drugs and preferred terms introduced by this new space has the potential to reshape many areas of pharmacovigilance science and spur novel advancements in the field. We have now at our disposal an entire space that begs to be explored. So let's board our spaceships and boldly go where no one has gone before.

SPEAKER_00 13:07

Hi Lucy, thanks for coming in.

SPEAKER_02 13:09

Hi Jared.

SPEAKER_00 13:10

First of all, congratulations on your research, but also on what I thought was a really engaging article. I just wanted to ask you, in your research, you set your tools loose on a huge body of individual case safety reports. From a data scientist's perspective, can you tell us what challenges you face with the information in those reports?

SPEAKER_02 13:31

In the ICSR database, VG base, we have a lot of variety in the kind of information we receive. So, of course, uh quite a bit about the drugs and the reactions, but sometimes we get some stuff more about patient medical, um, medical history and stuff like this. But there is a lot of missing information. There is also not only structured data, but data that comes in in free form. And from a data science perspective, it's actually very, very interesting because we can we can try to apply data mining techniques to extract as much information as we can also from this unstructured kind of information. But for this particular problem that I was trying to solve, it was really a challenge to deal with terminologies because when people want to analyze this data, they have to decide on some terms to describe what happened. But the the current terminologies do not allow us to actually compare in an easy way uh the different ways of describing things. So that was the challenge I was trying to address.

SPEAKER_00 14:45

So in your article, you say you had more than 16 million records to work with. Is there a minimum threshold of data required to get meaningful results from these types of vector representations?

SPEAKER_02 14:57

Yes, there definitely is. Although I actually do not know what that number is. To be honest, I was myself very surprised that 16 million reports were enough to get something uh meaningful. So that was a very happy surprise. And now, since VigiBase is growing uh every every day, uh it's only gonna get better from here. So yeah, more data will will help us build even better representations uh over time. But yeah, there is a certain number necessary to make it meaningful, but it's apparently below 16 million.

SPEAKER_00 15:39

Will you be continuing to fade new reports into it incrementally?

SPEAKER_02 15:44

Absolutely. So we will need to probably think of uh a good way to apply this algorithm at at regular intervals, maybe try to do some linking towards previous representations. As of today, there is a bit of randomness associated to it. So from one set of report to the next set of report, it it might not be so easy to link the representations between the two sets. But um there are ways, there are ways, and I have ideas. I think uh it should be possible. And it will be interesting to see actually that that time component, how representations are gonna be changing over time in relation to others. And I'm wondering that that's a very wild research question, whether that variability over time could also reveal patterns of safety concerns.

SPEAKER_00 16:43

Languages tend to be quite messy. And from your article, I can picture how this technique can reveal synonyms, different words with similar meanings. But how does it deal with things like homonyms, words that look the same but have multiple meanings?

SPEAKER_02 16:59

Aaron Powell Yeah, so that is definitely a huge challenge with this method. What you're feeding in in the method is is um is just words. And so if words have different meanings but the same spelling, the algorithm will believe those are actually the same and try to build representations based on on actually two kind of contexts that it will appear in. So the result will be that the picture will be blurred. You would need to first apply what we call disambiguation techniques to realize that um there are two meanings or more for a given word and somehow find a way to distinguish between them. That can be just a little trick, for example, adding end letters to the word uh depending on the meaning. But but the disambiguation technique itself is actually it it's a difficult problem to solve. Now for this particular problem, so madra is definitely uh not ambiguous, so every single medra term is unique, so you won't find this problem. When it comes to products, you can find product names that actually mean uh different products with even different substances in it. But we were we were not using product names, we were actually using the um the active ingredients. And actually, down so under the hood, this this entire thing was done actually on uh on integers, integers that are actually linking a certain concept to to a single integer. So there was no chance of of uh having these kind of problems. Aaron Powell Okay.

SPEAKER_00 18:52

So in in other types of fields that might be a much bigger problem, but in this one, because there is a greater structure underlying the information, it's it's less of an issue, is that?

SPEAKER_02 19:01

Aaron Powell Yes. Yes, in regular text, it's definitely an issue. Uh for for VGBase it is not.

SPEAKER_00 19:07

Aaron Ross Powell Since you first presented your research, what have been some of the most interesting or potentially useful applications that have uh that have cropped up?

SPEAKER_02 19:17

Yeah, so there's gonna be some quite exciting work in the coming months, I believe. We are gonna use these representations to actually expand the scope of of some reactions. So if you're interested in in what particular potential safety concern, you can look at at the terms that are around that reaction in that vector space and start to look at the case reports that also mention these neighboring neighboring terms. And so it can allow you to expand your case series and have hopefully a more uh comprehensive uh set of reports to look at when you're looking at your hypothesis of a safety concern.

SPEAKER_00 20:04

Aaron Powell And have you had much reaction from the wider community uh or has most of the development talk been coming from within UMC?

SPEAKER_02 20:11

Aaron Powell The Pharmacovigilance community has been uh very positive to it. I went to a meeting uh organized by the International Society of Pharmacovigilance in in Boston. They have uh an annual meeting that is um very much focused on what AI technologies can do for pharmacovigilance. This new research was really well received with with a lot of interest, people wanting to collaborate. So it's it's very it's very positive.

SPEAKER_00 20:39

Let's finish off with the big picture. How important is data science, things like artificial intelligence and machine learning and other techniques, to the future of pharmacovigilance?

SPEAKER_02 20:49

I think it it really has a big role to play. Just for the simple fact that VigiBase is not the only thing there can be in in PV. There is a large amount of data out there from all kinds of data sources, observational databases, uh, social media. The data can can come in in a lot of different forms, free form, structure form. And I think linking all this data sources in in one single like knowledge base, so so from data to actually knowledge about the drugs, data science will be will be key to to help doing this.

SPEAKER_00 21:32

Okay, uh that's great, Lucy. Thanks for coming in again. And uh also once again congratulations on your work.

SPEAKER_02 21:38

Thank you very much, Jared.

SPEAKER_00 21:40

Okay, bye-bye.

SPEAKER_02 21:41

Bye.

Federica Santoro 21:46

That's all for now, but we'll be back soon with more long reads as well as our usual conversations with medicine safety experts. In the meantime, check out the episode show notes for more information. And visit uppsalareports.org for more pharmacovigilance stories. If you like Drug Safety Matters, make sure to subscribe to it via your favorite podcast directory. And do send us comments and suggestions on social media. We love to hear from our listeners. You'll find Uppsala Monitoring Center on Facebook, LinkedIn, and Twitter. And you can join the conversation there with the hashtag Drug SafetyMatters. Thanks again for listening, and till next time.