Intro

Federica Santoro 0:15

Where do pharmacovigilance experts look for signals? The obvious answer is spontaneous reports. That is, suspicions of adverse drug reactions submitted by patients or healthcare professionals. But what about other types of data, like clinical trials or observational studies? How does the community use that evidence? And how do they decide what is a safety? My name is Federica Santoro, and this is Drug Safety Matters, a podcast by Uppsala Monitoring Center, where we explore current issues in pharmacovigilance and patient safety. Joining me today is my colleague Daniele Sartori, pharmacovigilance scientist at UMC and PhD candidate in evidence-based medicine at the University of Oxford in the UK. As part of his doctoral research, Daniele sifted through more than 2,000 papers to find out how pharmacovigilant signals are detected. A few months ago, he published those results in a scoping review, and so I invited him to the studio to learn all about it. So buckle up as we go on a deep dive in the world of signal detection.

Welcome, Daniele!

Federica Santoro 1:43

Hi Daniele and welcome to Drug Safety Matters. Thank you. Are you ready for this? Yeah, I am. Let's dive straight in then. So today we're talking about spontaneous reporting and other sources of evidence for signal detection. And this is clearly a topic that resonates very much with our audience because we received a bunch of questions from our listeners when we announced the interview on social media. So why don't we start with one of those questions?

What is a signal?

Federica Santoro 2:13

I think we should start with Sudarshan from India, and I hope I pronounced his name correctly. He has an excellent opening question for you. When does an adverse event become a signal?

SPEAKER_01 2:27

So before I answer this question, I think it is important to preface that there are many definitions of signal. And the one that I will be using today is the Symes definition of signal, the harmonized definition of signal, SIMES 8. I will consider signals of disproportionate reportings in the word signal, even though it's not strictly speaking correct. But to be brief, I will encompass signals of disproportionality in the word signal. So I think this is a a bit of a the million dollar question in pharmacovigilance, when an adverse event becomes a signal. The way that I interpret the when is how much of a type of evidence, how much evidence do we need before we can say that there is a signal, or when, sort of in times of temporal matters. I think Mayboom put it best in 2002 when he said that you can describe a signal as a hypothesis. This is not strictly speaking a definition of a signal, but you can describe it as a hypothesis supported by both data and evidence in favor or against a possible causal relationship between a drug and an adverse event. So a catalogue of the evidence as presented in the scoping review could give some insights into the level of evidence necessary to say that there is a signal, or in terms of, say, the calculation of the time to communication in terms of when an adverse event becomes a signal?

Sources of evidence

Federica Santoro 4:27

Right, so let's dive into your publication then. We tend to think of spontaneous reports as the cornerstone of evidence in pharmacovigilance. And while they certainly are the most common source of evidence, they're far from the only one. Where else then can pharmacovigilance professionals look to in the hunt for signals?

SPEAKER_01 4:51

As you said, we do tend to think of spontaneous report as the primary source of evidence for signals. And indeed, most signals tended to be based on spontaneous reports throughout the scoping review. I think most often spontaneous reports tend to be the only evidence available at a time of detection of a signal. But other sources to detect signals might well be the raw data from clinical trials or the results of a unobservational study, or they can be pools of clinical trials, meta-analyses of clinical trials. In any case, any sorts of data that we consult to detect signals should account for the kind of adverse event that we're studying, the rarity of the adverse event, uh, as well as the population of interest or other factors that might be very much specific to the uh drug and adverse event that we are considering. So in some cases, some research questions can't always be answered by spontaneous reports alone, but we need more than one type of evidence.

Why did you research this?

Federica Santoro 6:14

So many different types of evidence. And what you set out to do in your recent publication was to understand how they have been used over the years to identify signals. Why did you choose this particular research question?

SPEAKER_01 6:28

Yes. As we've touched a bit upon earlier, we have typically thought of spontaneous report as the main source of evidence in pharmacovigilance. But pharmacovigilance has evolved over time. We are now able of analyzing large databases, whereas 60 years ago we couldn't. And most importantly, I think the definition of a signal has changed. I think in the earliest instances, uh in the late 60s, early 70s, a signal was defined differently than it is defined today. So I wanted to learn whether and how this change in definitions and the how the evolution of pharmacovigilance has affected the types of evidence that we have used to detect and appraise signals. Importantly, there had been no prior systematic research on the topic, and I thought a scoping review design could be helpful in better understanding this body of evidence.

Scoping review

Federica Santoro 7:35

And in a minute we'll look at what you found. But first, how did you conduct your study? What is a scoping review?

SPEAKER_01 7:43

Yes. So a scoping review is a type of systematic research. It's a type of research that allows you to map a body of literature. It allows you to better understand whether there are any gaps in that body of literature. It opens up questions for future research, it opens up opportunities for systematic reviews, and it also allows you to understand what are the limitations of a body of literature. And a scoping review uh relies on the use of systematic queries in databases like PubMed or Ibisco. These queries must be reported as transparently as possible. And in fact, indeed, transparency is something that I strove for, in the sense that whoever wants to reproduce the results of the search queries can, as they are publicly available. I have followed the Prisma reporting guidelines to ensure transparency in my work in such a way that people could critically interpret the results of the scoping review and better understand the limitation of my work. And I've also published the research protocol so that if people wish to reproduce the findings, they can. So in this scoping review, I have retrieved studies in which the findings were described as signal. So to put it simply and colloquially, when somebody said, I believe that there is a signal, I have considered these studies as eligible. Of course, these studies had to concern human participants and they had to concern medicinal products, so they had to concern drugs, colloquially speaking. And they had to present signals that had not been previously documented, in the sense that whichever adverse event was being considered in a study, it did not have to be already available in summaries of products' characteristics or similar documents like drug labels in the US. And where that information was unavailable, I checked whether that adverse event had been previously studied in other observational studies or randomized clinical trials. So I've made sure to the best of my abilities to include signals that at the time of communication were not known. Finally, I've collected all of the studies that I found eligible and I've analyzed them descriptively and over time.

Federica Santoro 10:28

Sounds like a whole lot of reading.

Signals based on case reports

Federica Santoro 10:31

But on to the results now. So you found that most signals are indeed supported by case reports, which was expected, I guess, based on what we discussed earlier. But you also found that signals based on case reports alone, so with no other evidence to support them, have become more and more common. Is that worrying?

SPEAKER_01 10:54

I wouldn't describe that as necessarily worrying. As I've said, case reports may just well be what is available at the time of communication of a signal, or at the time of signal detection even. Part of what makes a signal a signal, in my view, is that a signal can be early and can be tentative. And there is a risk that signals, the signals that I have included in the review, may have been later falsified down the line. Some signals may not have necessarily been followed by a regulatory action, in the sense of, say, an amendment to the summaries of product characteristics or a communication to health carers.

Federica Santoro 11:41

And by falsified you mean disproven.

SPEAKER_01 11:45

Technically, it would be a refuted signal. So the possible causal relationship aspect of the signal was shown to no longer be the case after further evidence was analyzed. And I think it is entirely fine to have refuted signal or to have these early communications, provided that they're grounded on a thoughtful assessment of the evidence and provided that the data that underpins the signals is genuine and good. Also, it is vitally important that we communicate these signals clearly, that we communicate our findings clearly. So long as the communications of our findings and signals is clear and how we have detected our signal is made transparently, that's fine even if a signal is based on case reports. I would say that the overall the message is, insofar as we can critically interpret a signal that is based on case reports, that's not worrisome.

Features of causality

Federica Santoro 12:53

You also looked at what features of case reports helped authors assess causality. Why don't we expect people to follow established guidelines for causality assessment?

SPEAKER_01 13:08

There are different methods to assess causal relationships between drugs and adverse events. You've got your single-case causality, and we all know the various algorithms, the Naranko algorithm is one of them. So there are these checklists that allow you to assign scores to certain properties that you have identified, and based on the final score, you can give a qualitative judgment, a probability that there is a causal relationship. It can be unlikely, probable, possible, certain, and so on. You have also case series tools, and the most famous one is possibly the Bradford Hill viewpoints. I wouldn't claim that these are formal guidelines. These are tools that help us clinically assess a single case or a case series. But when it comes to following guidelines on how to report these considerations, to the best of my knowledge, I couldn't find one. So it is, to the best again, of my knowledge, I don't know of journals mandating the use of a checklist to report these findings. So to pick out these features was a good exercise in critically reviewing the documents that I retrieved and to verify whether these considerations had been reported explicitly. And I have expanded a bit in the manuscript in the proportion of these findings.

Experimental evidence

Federica Santoro 14:44

And on that note, here's another question from our listeners. This one's from our very own colleague, Magnus. He asks, could you elaborate on the use of experimental evidence in your publication? And are there any subtypes within it?

SPEAKER_01 15:00

Yes, I I think so. So looking at the evidence from experiment in the original Brad of the Hill manuscript, uh it is contemplated in the in the viewpoints in the original paper. You could consider it as experimental evidence different types of studies. You could consider it, for example, as experimental evidence, evidence from a clinical trial to some extent. But when it comes down to case reports, I found positive dechallenge and rechallenge to be the most applicable, but positive dechallenge and rechallenge, they require whoever makes the discontinuation of a drug to readminister the drug at the same dose. So to identify a subtype of that, you could manipulate the dose at which the drug is administered. And if you see that administering the medicine at a lower dose produces, say, an adverse event that is no longer maybe serious or at a lower degree of severity, you could suggest that by manipulating the dose you would have a sort of experiment. At that point, though, you would be spilling over into the dose response relationship, this biological gradient in the Bradford Hill viewpoint. There is more to say about the Bradford Hill viewpoints in the sense that these were developed by an epidemiologist for epidemiology, and we have only later on adapted them to pharmacovigilance. So we're a bit parasitic on these methods, and we have tailored them to the best of our abilities to case reports. There are different ways that we can map the various viewpoints to our scenario in pharmacovigilance. In fact, some have interpreted experimental evidence as evidence coming from preclinical studies. So to answer the question, yes, you can. I think there is a bit of wiggle room, but I found the challenge and very challenge to be, strictly speaking, the most applicable interpretation of evidence from experiment.

Reporting clinical assessments

Federica Santoro 17:22

And something I personally found surprising as I was reading your review is actually how few clinical assessments explained why the authors drew certain conclusions on causality. And you have numbers in there, so you reviewed more than 1,600 assessments and you found that only 225 did that. That's only 13%. How is that possible? Um, shouldn't journals be requesting that kind of information from authors?

SPEAKER_01 17:53

Yeah, so as we touched upon, we don't really have guidelines on how to report these clinical assessments of case reports. I find it humanly challenging to not have any sorts of conclusions or any sorts of judgments on case reports when we go through them. I find it more likely that these judgments have simply gone unreported. I do, however, think that this sort of reasoning should be and could be made more explicit. So when I was going through the case reports, any such judgment that the author has made on the case series had to be explicit. So simply reporting that the author has recorded a certain number of positive D challenges or a certain number of reports in which there was a dose response relationship, it wasn't strictly speaking explicit. In my view, the author had to justify how that finding was relevant or how that finding was supportive of a causal relationship. So, well for journals to request this sort of information, we would first need guidelines for reporting signals, otherwise we would all be doing things a bit differently from one another. And it is on the roadmap of my PhD down the line to develop a checklist that would allow authors to better report signals or signals coming from clinical reviews of case reports. So stay tuned for anyone interested.

Federica Santoro 19:34

We will.

Case-by-case assessment

Federica Santoro 19:36

Magnus had another question on the value of case-by-case assessments. How does that support signal evidence?

SPEAKER_01 19:46

If done properly, case-by-case assessments, they can provide uh useful insights in decision making. But as we can imagine, there are thousands of signals out there, and there are very many case reports. And in some cases, case-by-case assessment is not always feasible. As we have also seen with the pandemic, there have been many case reports coming in, many databases across the world. So assessing a large amount of data, even for a single medicinal product and one adverse event, it becomes challenging. So I think if case-by-case assessments done thoughtfully can provide good evidence for decision making, we need good prioritization systems to ensure that the most important adverse events are tackled first. So at UMC we have tried to develop methods to best focus our efforts where it is more likely to find signals. Fairly pragmatic methods where we try to investigate those drug adverse event combinations where we have the most complete data, so where we can make the best use of our resources, as case-by-case assessment is a time-consuming endeavor. So Vigirank is a good example that can allow us to best focus our energies where we have sufficient data. So if we had virtually unlimited resources, we could do every assessment case by case, but this is not always feasible. Again, for this, we would need appropriate systems of prioritization, either at the time of signal detection or after signal detection.

Federica Santoro 21:50

That makes total sense.

Unexpected findings

Federica Santoro 21:52

I mentioned earlier what I found surprising in your study, but were there other unexpected findings according to the. To you?

SPEAKER_01 22:01

To me, it was a bit unexpected to find that some types of evidence, particularly evidence of higher quality, that is, an Oxford Center for Evidence-Based Medicine level, say between one and three, that these types of evidence were sort of increasing over time. Before drawing any strong conclusions, I would say this would be best investigated in follow-up analyses. To me, and I did not really expand much on this in the paper, but I plan to expand on this in my thesis. But to me, it was fairly surprising to see across how many platforms signals were scattered. There were, I don't remember how many tens of thousands of records retrieved during the scoping review. And there were equally well, there were very many web pages that I had to go through to find uh various signals. So to me it was a bit surprising that there was really uh no centralized repository of of what we could describe as a signal. And in my small attempt, I have tried to put together one that is uh available for anyone who's willing to uh use the data set that is uh in the supplementary materials of the paper. But I think the concept of signal as something that has floated around pharmacovigilance for now almost 60 years. I think it was fairly unusual, at least in the published domain, to see some findings described to be signals. So it shows that signal is almost a concept that is insular to pharmacovigilance and it's not permeated into other disciplines. And it was it's almost a widely used and a fairly accepted term in our realm, but it has yet to spill over in other disciplines despite its age. So to me that was quite unusual.

Value of other types of evidence

Federica Santoro 24:08

So you said you found that other types of evidence have been used increasingly in addition to case reports. What value do they add, and should we be using them even more?

SPEAKER_01 24:22

When we assess signals, and signal assessment in this case is used in the sense of the good vigilance practices, uh, module nine published by the EMA, when we assess signals, we are supposed to use all available evidence. So in the whole signal management process, which is a fairly long, lengthy process and well detailed in the guidelines, uh, this comes at the later stages of signal management. So we should be using uh evidence ranging from preclinical mechanistic evidence to published and unpublished data from interventional or non-interventional studies. At UMC, we have tried to look into whether routinely collected data could be helpful in signal validation and signal prioritization. And these routinely collected data came from the Eden Consortium. So these data may help us better understand if there are any possible competing explanations. When we appraise case reports, we might come across uh case series that share a possible alternative explanation for a causal relationship. So these data from large databases of, say, claims data or electronic health records, they could be helpful in better understanding these alternative explanations. Uh we're talking about confounding by indications, and these can be better understood by looking at the rates of occurrence of an adverse event prior to the administration of a medicinal product, or they can give us some contextual information by looking at the demographics of a certain population taking a medicine. And of course, these are limited to the databases that you're analyzing. So we have tried to explore the usefulness of these uh sources of evidence. There is added value in these sources of evidence as to whether we should use them. That is still, I find, an open question.

Pharmacological reasoning

Federica Santoro 26:38

Back to our listeners, both Magnus and Sudarshan were wondering about class effect and pharmacological reasoning. To what degree, they ask, have they been used as evidence for signals? So, in other words, when performing signal detection, do people also look at drugs with, say, a similar chemical structure or a similar mechanism of action?

SPEAKER_01 27:03

So I think chemical structure and mechanism of action are two separate uh entities that I should tackle each one individually. I think overall we are in the realm of what Bretford Hill described as analogy in the original paper. And the assumption, say, is if you have a given medicinal product that belongs to a drug class, is it uh sensible to suspect that another medicinal product falling under the same drug class produces the same adverse effects? This is an entirely sensible question to answer, but there is a lot to say here, I think. If we consider it, for example, NSAIDs, some NSAIDs are known to cause taste disturbances. Nabumatone can cause taste disturbances, and you can measure these taste disturbances by administering patients with substances that taste bitter or that taste sweet or that are salty, and you can measure how much this sense changes after the administration of nebumaton, but other NSAIDs they don't necessarily produce taste disturbances, nor they produce the same taste disturbances as nebumaton. So this is an example of two medicinal products in the same um drug class, which is non-steroidal anti-inflammatory drugs, that can have different effects. You could have another example of proteinase inhibitors such as atazanavir and retonavir. One is known to induce gallstones, but the other one isn't. Or you have to sufloxacin and moxifloxacin, one can give you corneal deposits and the other one can't. And these couple of other examples are both falling under the same class of medicinal products and they have roughly a similar chemical structure. Now I have used the word similar chemical structure, but there are, say, scores that would allow you to say that two chemical structures are similar. One score is the Tanemoto coefficient, and this is a numerical indicator that says these two drugs are similar if they have this score, if they exceed this threshold. So it is important on how we define the class of a medicinal product, it is important how we define how two chemical substances are similar. I think my colleagues Lucie Kirant and Evalisa Meldau, they have worked on structural similarity to support timely signal detection. But when it comes to the review I did not consider mechanistic evidence, and even here there is more to say in the sense that you could deconstruct mechanistic evidence in evidence of a mechanism, so evidence that there exists a mechanism of action or a mechanism that an adverse event is produced in some way, or that a drug can mechanistically produce an adverse event, and evidence from a mechanism. And I would say that the latter kind is what you might expect in pharmacovigilance. Overall, it is best to use well-established mechanisms to support a causal relationship between a drug and an adverse event. So even though the Oxford Centre for Evidence-Based Medicine levels of evidence, they do contemplate evidence from a mechanism to be supporting evidence, some have argued that this type of evidence is well, it is a category error to include this in the classification done by the Oxford Center. So for these reasons we have opted not to include this type of evidence in the review, and so I did not consider mechanistic evidence throughout the review as being supportive of signals. If anyone is interested in expanding more on the division between evidence of a mechanism and evidence from a mechanism, there is a great chapter in uncertainty in pharmacology. It's the first chapter of the book, and it explains both the possible category error and these definitions more in detail. But in my reviewing the manuscripts and the various published studies, I did see that some authors used mechanistic evidence to support signals. I simply did not collect that data.

Federica Santoro 32:13

Interesting. Sounds like there's a lot more to discuss there. Maybe we should devote a whole other episode to this. But we will definitely include the reference to that book in the show notes. Moving

Time to communication

Federica Santoro 32:25

on to something else, you also looked at time to communication in your review. And that's the time it takes for a signal to be, well, communicated once it's been detected. And uh one of our listeners, Heba from Egypt, has a few questions on that actually. First, she asks, how can we shorten that time and improve the way we communicate signals to our target audience?

SPEAKER_01 32:54

So the time to communication is a measure that I introduced in the manuscript. This is based a bit on what my supervisor, Iko Onakapoya, did in his uh systematic review of drug withdrawals, where he measured the time that it takes from the very first report to the withdrawal of a medicinal product from the market. And I thought of uh doing something similar using the reports in VigiBase. I interpret the time to communication as the time that it takes us to have sufficient evidence to suggest that there is a signal. It doesn't have to start necessarily from the detection of a signal. In fact, the time to communication starts from when we received the very first report in Vigibase, or when the first report was sent to the National Center sending data to Vigibase. It's a bit of a nuance, but I I think it's worth uh expanding on this. My original understanding of this metric, which was I think in median nine years, was that perhaps a way to shorten it, if desirable, would be to ensure the completeness of the case reports that we submit as reporters. Because if we read the time to communication as the time that it takes us to have sufficient evidence, and we know that the strength of evidence in case reports is generally guided by the quality of the case reports or the quality of a case series. The more complete it is, the easier it can be to assess a case series clinically, because you you just have more information and you can better rule out alternative explanations and you have a better clinical picture of a single case or a case series. So one way would be to ensure the completeness of the reports. But I do have a follow-up study and we are writing up the manuscript, so I hope to have uh drafted the manuscript ready rather soon, and I hope to publish it equally rather soon. So again, everybody stay tuned if you're interested. As to how we can ensure that the communication of a signal is done properly, or is done is of good quality, there are guidelines that the target, whoever communicates to health carers, and these guidelines are available, say, on the EMA website, the guidelines for direct healthcare professional communications, or the FDA has the drug safety communications. And these guidelines uh specify which elements should be part of a communication directed to healthcarers. So I think these can be an excellent starting point to ensure the quality of communication.

Using signal information

Federica Santoro 36:02

Right. But Hebai is also wondering how the information is used, because even when signals reach our intended audience, say healthcare professionals, it's unclear, she says, what they will do with that information. So how can we fill that gap?

SPEAKER_01 36:21

This is a fairly complex matter. We cannot strictly predict how healthcarers will use the information in the European Union for direct healthcare professional communications or in the US for drug safety communications. What I've seen so far is that these communications, uh, when they are, for example, they concern ongoing investigations. If the information therein is described as very early and tentative, what a regulator would usually do is say to the healthcarer, don't change your practice, and they encourage the patient to not stop their treatment based on this very early information. Now, whatever we do, I think it is important that the recommendations that we provide in these communications is grounded in the evidence, and it is commensurate to the evidence that we have available at a time of communication. And there are guidelines that based on the strength of evidence allow us to produce recommendations. Uh, one example is the grade guidelines, they allow you to say how strong the evidence is and to produce recommendations based on the strength of evidence. And Murad and colleagues on the EBM BMJ, they published in 2018 a method to evaluate the goodness of the evidence of a series of case reports. And this is envisioned to facilitate this interpretation of the evidence into the grade scales, again to produce sensible recommendations commensurate to the evidence. So one of the advantages of the grade scale is that even though you might have a low strength of evidence in principle, and we're talking about hierarchies of evidence, you can produce recommendations nonetheless. And a good example of, again quoting the original paper, is that in the case of aspirin and race syndrome in pediatric patients, the regulators could say, do not administer aspirin to children because the adverse event in itself was very serious, and we also had an alternative, which was paracetamol. So if we use validated methods to appraise the evidence and to produce recommendations, if you will, the role of the pharmacovisions professional is to present a summary of the evidence to the busy clinician, to present the best available evidence to the clinician in such a way that they can make the most up-to-date information commensurably to the evidence. Now, whether a drug safety communication or a communication to health care produces the effect that we want is an entirely different matter. There are unintended consequences of communications to healthcarers, but that would open up an entire podcast probably on this topic.

Federica Santoro 39:26

You're feeding me lots of new ideas. We will certainly not run out of inspiration for future episodes.

Is artificial intelligence a threat?

Federica Santoro 39:33

One final question from our listeners. Tanji in the UK asks with the adoption of automated systems and artificial intelligence, both to retrieve data but also to identify signals, are pharmacovigilance professionals becoming endangered, he says. So are we all going to be replaced by robots?

SPEAKER_01 39:55

Um So this is not, strictly speaking, my area of expertise, and I can only express my opinion on the matter. I barely have an understanding of machine learning, uh, and I barely scraped the literature on these respects. My personal opinion is that I don't think we are endangered as a category of professionals. My understanding of, say, machine learning models is that humans we are still necessary to interpret the outputs of a machine learning algorithm or the output of an AI. And even then the goodness of an AI is as good as the data on which it's trained gets. It's a bit of a you know, garbage in, garbage out kind of matter. So for extremely complex tasks, such as taking into account multiple elements in a case series, and considering multiple alternative explanations within a massive causal web, which is your average clinical review of case reports, I don't find that clinical professionals or pharmacovigilance professionals can easily be replaced by AIs, nor I find that this can happen anytime soon.

Impact of the study

Federica Santoro 41:26

Thank you. And we've now dissected your study in all sorts of ways, but to wrap up, what impact do you hope it will have on the pharmacovigilance community?

SPEAKER_01 41:38

Yeah, uh I think at least a couple, maybe three, come to mind. The first is that we may want to become a bit better at presenting our reasoning when we write down clinical reviews of case reports. This is important to enable others to interpret critically why we suspect that there is a causal relationship. The second finding I think is I would hope that we better reflect on the bearing that signals of disproportionate reporting have on regulation. I could not find um studies that measured the impact that these studies have on decision making. So it may be helpful to better understand this, especially since we found that they are growing in in numbers. And indirectly, I think the scoping review makes available a fairly large data set of signals that whoever is interested can use for future studies.

Future research

Federica Santoro 42:43

And finally, what's next for you? What other burning questions are you hoping to tackle soon?

SPEAKER_01 42:49

Yeah, so the these burning questions will be part of my PhD thesis. And I've said that I've I'm writing up a manuscript that will better look into the time to communication, but I do want to also look into the follow-up of signals. So what kinds of regulatory actions follow both the signals that I have already retrieved and signals that I will retrieve by updating the searches in the scoping review. And uh finally, I am hoping to uh write down a checklist on how authors of clinical reviews of case reports can better report signals where causality assessment plays a central role, at least.

Federica Santoro 43:32

Well, best of luck with all that, and we look forward to reading more of your research. I will end by thanking you on behalf of all our listeners. Thanks for being here and for taking the time to answer so many of the questions we received from our audience.

SPEAKER_01 43:49

Thank you, audience, for all of your very insightful questions, and thank you for having me.

Outro

Federica Santoro 44:01

That's all for now. But we'll be back soon with more conversations on medicine safety. If you'd like to know more about Daniele's research and sources of evidence informal covigilance, check out the episode show notes for useful links. We only touched upon real-world data and the Eden project in this interview. But if you'd like to know more about that, then don't miss the next episode with Patrick Bryan from Janssen on observational studies and their use in public vigilance. If you like our podcast, subscribe to it in your favorite player so you won't miss an episode. And spread the word on social media so other listeners can find us. Apart from these in-depth conversations with experts, we host a series called Uppsala Reports Long Reads. A selection of audio stories from UMCs from CoVigilance magazine. So do check that out too. Upsala Monitoring Center is on Facebook, LinkedIn, and Twitter. And we'd love to hear from you. Send us comments or suggestions for the show, or send in questions for our guests next time we open up for that. For Drug Safety Matters, I'm Federica Santoro. I'd like to thank Danile Sartori for his time, Matthew Barwick for production support, and of course you for tuning in. Till next time.