Intro

Federica Santoro 0:15

Where do pharmacovigilance experts look for signals? Mainly in spontaneous reports, as we discussed in the previous episode. But another important source of evidence is real-world data, the data we collect as patients move through routine clinical care. This information is vast and complex, but extremely rich and valuable. So how do we unlock its full potential? My name is Federica Santoro, and this is Drug Safety Matters, a podcast by Uppsala Monitoring Center, where we explore current issues in pharmacovigilance and patient safety. Joining me today is Patrick Ryan, Vice President of Observational Health Data Analytics at Johnson Johnson, an adjunct assistant professor in the Department of Biomedical Informatics at Columbia University. For years, Patrick has been developing and applying analytical methods to bring out the value of real-world data, especially for drug safety. So I called him up in the US where he's based, and we chatted about data access, data quality, study a thones, and much more. I hope you enjoy our conversation.

Welcome, Patrick!

Federica Santoro 1:38

Hi Patrick and welcome to Drug Safety Matters. I'm delighted to have you on the show. How are you?

SPEAKER_00 1:44

I'm doing great. Thank you for having

What is real-world data?

SPEAKER_00 1:46

me.

Federica Santoro 1:46

So we are talking about real-world data in pharmacovigilance. And first, I'd like to get the terminology straight because I came across a few different terms as I was reading up on the topic. And I must confess, I'm a little confused. So let's start with the basics. What is real-world data?

SPEAKER_00 2:07

Absolutely. And there are a lot of confusing terms that get mixed around. When we think about the use of data for pharmacovigilance, we can oftentimes think about the use of spontaneous adverse event reports that are submitted by patients or providers when they've experienced some sort of an event. And that's a large part of our safety infrastructure for post-approval. Prior to approval, our primary data source is randomized clinical trials, where we're conducting interventional experiments and prospectively collecting data about a set of individuals that are exposed and followed on. Real-world data is the current buzzword that's used for something different and complementary to randomized clinical trials on one extreme and spontaneous data on the others. Here we're talking about data captured during the routine course of clinical care, whether that be insurance claims records that are captured for financial reimbursement purposes, or whether we're talking about electronic health record data, it's the data that's captured when a patient is engaging with a clinician as part of their health care experience. And these data can be brought together, de-identified, and then analyzed to try to identify patterns about what has happened across a collection of patients. So when I use the word rear-world data, I'm typically referring to data that has already been collected through routine course of care that we can then think about conducting secondary analyses on this. An important distinction between real world data versus randomized clinical trials is there is no explicit intervention. So in a randomized trial, one is actually determined what exposure they're going to have. Whereas in observational data, we may follow a patient longitudinally. However, we don't decide what treatments they get. The patient decides for themselves, and we're trying to learn from the decisions that they've made and the consequences of those decisions.

Federica Santoro 4:06

Thanks for clarifying that a bit for me and hopefully for some of our listeners who might have been confused as I was by the terminology.

Sources of real-world data

Federica Santoro 4:15

So, as usual, we told our social media followers about this interview, and we invited them to send a few questions for you. We'll start with Sudarshan from India. So he's wondering where we can get real-world data from? And you mentioned a few of the sources. Can we go a little deeper into that?

SPEAKER_00 4:37

Absolutely. So there are a wide array of different types of real-world data. In terms of the general groupings that we see around the world, there are many private insurers or government entities that provide healthcare benefits for populations, and they're capturing data for administrative purposes, so insurance claims records that may be captured, and there's a wide array of different insurance claims data sets that are used for real-world data analysis. There are electronic health record data systems that capture data either from general practitioners in the network of clinicians that are part of that, or through integrated health delivery systems, including both outpatient and inpatient care. There are also real-world data sets that are principally based within hospital systems, so capturing all of the interactions that happen from the time that you are admitted to the time that you're discharged. And then we see in various countries where there is more coordinated and centralized care, that there are national registers which capture information about individuals throughout various parts of their disease state. And so all of these can kind of broadly fall under the umbrella of real-world data, particularly as we think about secondary analysis of these data. There are also research efforts underway to create disease-specific or treatment-specific clinical registries. And most often those actually involve primary data collection. So trying to identify a set of individuals and then capture information about those individuals. All of these can be broadly in the same umbrella. For the purposes of what we'll talk about today with pharmacovigilance, I'll largely focus on the secondary use of existing data that has already been captured during the course of routine clinical care.

Federica Santoro 6:30

So there's a bunch of different sources, and I guess one of the challenges will be how to bring them together and harmonize them. But we'll get to those challenges a bit later. Let's start with the opportunities.

Value for pharmacovigilance

Federica Santoro 6:43

In the previous episode, I spoke to my colleague, pharmacovigilance scientist Daniele Sartori, about the different types of evidence that pharmacovigilance experts can use to find signals. And the upshot of that conversation was that spontaneous reports do remain the cornerstone of signal detection, but other types of evidence can add value. What kind of value do they add?

SPEAKER_00 7:12

So within pharmacovigilance, our goal is to try to identify and evaluate potential safety concerns of medical product exposure. And spontaneous adverse events reporting provides a foundational element because we do observe cases where a patient or a provider suspects that following exposure some adverse event has occurred. One of the deficiencies that that offers, however, is that we don't actually have some notion of a denominator of how many people were exposed. We can't actually estimate an incidence rate to know how often that adverse event happens, nor can we actually do causal attribution to understand what is the magnitude of the strength of the association between exposure and outcome. And real-world data actually provides an opportunity to fill all of those gaps. When we think about real-world evidence, we can classify the different types of evidence into three primary categories. There's characterization. We can think about that as studying disease natural history, doing treatment utilization, or understanding the incidence of an event. And so this is descriptively describing what has happened to a population. The second category is population level effect estimation. So this is doing safety surveillance or comparative effectiveness, where the goal is to do causal inference to try to estimate the relative risk of an exposure on an outcome. Then the third category is patient-level prediction, the idea that you could apply machine learning algorithms to develop predictions about which patient is at high risk of experiencing some sort of event. And the exciting part about how I see real-world data complementing what we learn from randomized clinical trials during development onto spontaneous data is that we can have access to very large populations and we can characterize their experience, estimate their relative risks, and ultimately get to the point where we can provide precision care once we've identified a risk and understood its magnitude to try to uh mitigate that risk within the population of those benefiting from the medication.

Signal detection or assessment?

Federica Santoro 9:19

And to continue along those lines, here's a question from uh one of my colleagues, epidemiologist Judith, who asks: where in the pharmacovigilance cycle do you see real-world data having the greatest impact? Is it in signal detection, signal assessment, or perhaps both?

SPEAKER_00 9:39

It's a great question. When I think about the opportunities for real-world evidence, I'm really excited to see how we can bring this evidence to bear across the entire pharmacovigilance lifecycle. I'd really like to see a situation where we were conducting analyses to identify safety signals, complementing our ability to detect uh issues with spontaneous data. But I'd also like to see real-world evidence being used to confirm or refute hypotheses that are arisen from spontaneous data. If we look at the current state today, real-world evidence is not fully exploited for pharmacovigilance purposes. Most organizations that are doing signal detection activities are primarily relying on spontaneous data, and few are aggressively using real-world evidence. And when it comes to signal evaluation, a real challenge that we see is that very often we need to generate evidence efficiently in order to make decisions to satisfy public health requirements. And so real-world evidence has a history of requiring epidemiologic studies that may take many months or years to conduct, and that doesn't really necessarily fit into the typical regulatory timelines that we think about when trying to identify and mitigating harms to patients. However, with a lot of technologic advances, we now see that it is possible to generate reliable real-world evidence very efficiently, and that now opens the door for us to bring that evidence to bear throughout the pharmacovigilance lifecycle. So I'm really eager to see real-world evidence that's generated by individual organizations or entire communities become another founding pillar of pharmacovigilance alongside spontaneous adverse event reports.

Federica Santoro 11:27

Exciting to see how the field will evolve.

Post-authorisation safety studies

Federica Santoro 11:31

Our loyal listener, Josue from Mexico, has a question on post-authorization safety studies instead. And these are another way to gather information on a medicine safety after it's been approved. So could real-world data replace the need for such studies?

SPEAKER_00 11:49

It's a great question. When we have requirements for conducting post-authorization safety studies, one of the first questions one has to ask is what kind of study are we going to conduct? And very oftentimes the interest in doing that requires conducting prospective data collection. That can actually take many years, can be extremely expensive, and ultimately results in evidence not being available for quite some time. The opportunity for doing secondary analysis of existing real-world data is that we may be able to generate analyses that provide reliable evidence in a fraction of the time and a fraction of the cost. And indeed, we are seeing opportunities where the need to do post-authorization studies is still very real, but that real-world data could be the source of that information. So that's already taking place. And I think as we establish a more credible approach to generating evidence from real-world data, we'll see this data be used even more aggressively for honoring regulatory commitments for all regulated medical products.

OHDSI and EHDEN

Federica Santoro 12:55

I'd like to talk a bit about the projects you're directly involved in now. You've been collaborating with UMC for years to explore what real-world data can do for pharmacovigilance. And you've done this first within the Odyssey Consortium and now lately with the Eden project. What are they other than clever acronyms, I have to say?

SPEAKER_00 13:18

Thank you. Yes. The Odyssey is the Observational Health Data Sciences and Informatics Collaborative. It's an open science community where we're bringing together researchers from around the world to try to make use of real-world data to generate reliable evidence that can improve health decisions around the world. Odyssey now has thousands of researchers from industry, academia, and government covering over 80 countries. And we've also established the world's largest real world data network where we can conduct secure distributed analyses across data sets to generate evidence, whether it be about questions of medication safety or comparative effectiveness or just learning about disease natural history. And one of my personal interests is thinking about how real-world data can support pharmacovigilance. So I've really enjoyed my collaborations with the team at UMC in thinking about how we can develop novel algorithms for analyzing real-world data and thinking about how real-world evidence that is generated can complement what we have learned from spontaneous adverse event reports. In Odyssey, one of our efforts that is underway is to try to create a truly international network across North America, Europe, Middle East, Asia Pacific regions. But one of the large projects within our community is Eden, which is the European Health Data and Evidence Network. And Eden is an IMI-funded project that has established a data network across the European Union that's quite impressive in terms of its scale, of working across academic institutions and industry partners to standardize a very large collection of real-world data sets to the OMOP Common Data Model, which is an open community data standard that Odyssey maintains, and working with those data partners to be able to establish a European-wide network to conduct analyses. And one of the founding work packages of the Eden project is thinking about its application for pharmacovigilance. And so we're excited to have UMC as a partner in Eden to think about how we advance the science of real-world evidence to support pharmacovigilance.

Study-a-thon

Federica Santoro 15:31

And actually, the last time you were here in Uppsala was September 2022 when UMC and Eden scientists got together for a week-long study-a-thon. So, first of all, what is a study-a-thon and what came out of it?

SPEAKER_00 15:48

Within the Odyssey community, we think a lot about how to establish collaborations across our researchers. And we try to innovate not just on the science of conducting analyses, but also on the social sciences of how to collaborate most effectively. And one of the ideas that we developed was that there's a need to have focused attention amongst collaborators to work together hard on very difficult problems by bringing them together and focusing on a hard problem as a community. And so the idea of a study-a-thon was inspired by the idea that instead of projects being something that are done meeting by meeting, week by week, dragging on across the world, what if we could actually have an ambitious goal to deliver a high-quality product within a short burst of energy where all collaborators were brought together? And so we've conducted numerous study a thons thematically focused on specific clinical areas, specific analytic domains, and we've focused on multiple days or a week where we just focus really hard on delivering evidence. In Uppsala, we held a study-a-thon specifically focused on pharmacovigilance. And the goal of that study-a-thon was to bring together pharmacovigilance practitioners at UMC who are using their spontaneous adverse event data to try to identify new safety signals and bring together a collaboration across the Eden community of data partners who had access to real-world data to try to determine how could that data support the UMC pharmacovigilance efforts when new safety issues arose. And during the week we had together, the UMC team identified novel safety issues. And each time one of those was proposed, we brought it to this real-world data community and said, what kind of analyses can we conduct to provide evidence that can support the thinking through that safety issue? And it was a really gratifying week because we were able to demonstrate the feasibility of conducting real-world evidence analyses very efficiently. Whereas earlier I mentioned that studies could typically take months or years, we were now conducting analysis in a matter of hours. And we were also able to demonstrate that that evidence was meaningfully impactful to support and inform the pharmacovigilance review process to determine whether or not a particular drug outcome pair warranted moving forward as a declared signal, or whether or not there was some other explanation that the real world data could offer that could help mitigate the potential concern. And so it was a really fun week to learn about where the observational data could support pharmacovigilance activities and also a great chance to collaborate to really put real-world evidence in practice into pharmacovigilance.

Federica Santoro 18:25

And I was there only briefly, but it was so inspiring to see people from so many different backgrounds and disciplines come together for a set amount of time and work on a specific problem. When's the next one?

SPEAKER_00 18:39

We're hoping to have yet another study-a-thon on this problem. I think what we learned was that we made good progress, but where certainly a lot more collaboration can be done. And we also learned that there's a lot more opportunities to bring even more data to bear. It was exciting because we had representatives from Croatia and Spain and the UK all coming together to conduct these analyses. But the Eden network now comprises over 25 countries and has over 100 data partners. And so we really are just scratching the surface of what's possible. And so I look forward to the next pharmacovigilance study a thon where we can even bring more real-world evidence to bear on the safety questions that the Uppsala team is identifying.

Federica Santoro 19:19

Wonderful. And we look forward to hearing more from that.

Challenges

Federica Santoro 19:23

Here are a couple more questions from my colleague Judith. We've discussed the opportunities and it's clear that there are plenty, but what are the challenges of using real-world data in pharmacovigilance and how can we overcome them, perhaps?

SPEAKER_00 19:39

One of the challenges that I see is that many organizations who are working in the drug safety space are intrinsically siloed, whether it be industry or government or academic institutions. We need to see more collaboration across the space. This particularly becomes apparent as it relates to real-world data because data access is a particular challenge. These data sources are extraordinarily large. There's concerns about patient privacy, and so we need to develop both technical and socio-political approaches that makes it acceptable to learn from the data while not jeopardizing or risking patients' privacy. Within Odyssey, we've established this distributed data network and we've demonstrated that we can conduct these types of efficient analyses, but largely it requires the will of organizations to want to collaborate together on a shared problem. And I think that the opportunity that exists there is very real, but the challenge is how do we actually get the appropriate support and willingness to collaborate across industry, government, academia on this shared problem. From a technical perspective, one of the challenges that I see is that spontaneous adverse event reports has tremendous value in providing a narrative, a description of what is suspected to be a cause of an adverse event associated with a medication. The language that is spoken in spontaneous reports is quite different than the language that is spoken in real-world data, even to the tactical level of adverse events commonly coded in Medra terms in spontaneous adverse event reports, whereas languages that are used in most insurance claims or electronic health record data are different types of vocabularies, such as ICD or SNOMED CT. And so just like the challenges of working in a foreign land and trying to figure out how to work through translating different languages, we have the same problem that our pharmacovigilance scientists and our epidemiologists working with real world data need to come up with an established common language so that we can speak about medication exposure and adverse events in a common tongue, so that when we learn from real world data and we learn from spontaneous data, that we can successfully integrate that information together.

Federica Santoro 21:57

Definitely.

Data privacy

Federica Santoro 21:58

And on the topic of challenges. Judith is also wondering about data privacy and how data privacy laws will impact the use of real-world data. So basically, how do we make the data open and accessible without infringing patient privacy?

SPEAKER_00 22:15

This has been a large focus of what we have worked on in the Odyssey community. And a large part of the solution is this idea of conducting a distributed data analysis. When one says they want the data to be open, we need to draw the distinction of what do we mean by data? If we think about the patient-level data that we have, it may be reasonable that we don't want to actually share that because that may jeopardize concerns of patient privacy. Within the Odyssey community, we run distributed data analysis. What that means is that the patient-level data is maintained securely behind the firewall of the institution that has access to that data. And instead of sharing the data, what we do is we share the analyses. That is, we write standardized software programs that are able to be distributed. Those software packages execute statistical analyses that translate the patient-level data into aggregate summary statistics. So think of things like rather than having your individual medical chart, you just learn that there's a certain number of people who have been exposed, or you estimate the relative risk of an exposure. And it's that answer to the question that is actually shared. In this particular way, what we're able to do is we're able to develop these standardized analytic packages, have a network of sites execute the packages and share results, and then we can learn from the patterns that are observed across that network without ever exchanging patient-level data. And so this is demonstrated to be a pretty effective approach that mitigates risks of patient privacy while still allowing us to learn quite a lot from the data without ever having to share it explicitly.

Data quality

Federica Santoro 23:55

Another topic that seems to be on our listeners' minds is data quality. And here's Sudarshan from India again. He asks, how do we know that the data we're looking at is true? And how do we avoid duplication?

SPEAKER_00 24:12

Data quality is a particular challenge for both spontaneous adverse event reporting and for real-world data alike. The particular challenge we face in real-world data is that we are trying to represent the longitudinal experience of a patient in all of their encounters with the healthcare system. However, we know that most organizations who have access to data have an incomplete picture of a patient's life. If you think about yourself, how often do you have some sort of personal ailment, but you didn't necessarily go to the doctor to have it recorded? Or you went to different doctors who might not necessarily talk to each other. This represents a particular challenge. A cornerstone of conducting real-world evidence analyses is the idea that one has to first construct phenotypes, which is basically the idea that you're creating a cohort of people who are believed to belong to some health state, whether it be exposure to a medication or which patients have experienced an adverse event. And it can be quite complex to create these algorithms to identify the people who are believed to have exhibit this health state that you belong to. And those are subject to measurement error. That is, sometimes you develop an algorithm, but you haven't identified all the right patients. Sometimes you find patients that are not the correct patients. A lot of our statistical analyses that we develop are trying to account for that measurement error in our analysis so that if we know that we have data that is not perfect, that we can account for the uncertainty that comes with the data quality problems. Within the Odyssey community, we have a lot of work that's focused on developing tools and strategies to identify data quality issues and try to remediate them so that we can improve the reliability of the evidence that we generate.

Quantity vs quality

Federica Santoro 26:04

In the age of big data, it's tempting to think that quantity is all that matters, but we both know that poor quality data doesn't get us very far. So, how should we balance quantity with quality in pharmacovigilance? He asks.

SPEAKER_00 26:21

Yeah, that's a great question. Um I'll answer it from a statistical perspective, and I'll I'll actually start from the vantage point of randomized clinical trials. In a randomized clinical trial, we oftentimes think that that may be the best experimental design we can to generate unbiased estimates of effects, but we often recognize that randomized trials are prohibitively expensive, and so you have to limit the sample size that you collect, which means that you get very accurate but not necessarily precise estimates of effects. The other extreme is to consider having data that is of poor quality, but a large quantity of it, so that you could end up with something that potentially is very precise but not accurate. The balance that we have to strike when we're doing our real-world evidence analyses is that random variation, which is basically how much data do you have, as data increases, that random variation goes down. However, the concerns we have with the analysis of real-world data is systematic error or bias. And bias can come from factors such as confounding or selection bias or measurement error. And bias doesn't go away no matter how much data you have. And that one of the key elements in conducting real-world evidence analyses is to try to generate evidence that truly reflects the uncertainty that comes from both the random error, which is sample size, and the systematic error, which is due to these quality issues. And as a field, we've been making good progress on trying to prove that evidence can be reliable from real-world data sources. But what we also have done is advanced the statistical analyses to actually explicitly quantify error and incorporate it into our analyses. And so right now at this stage, particularly with the analyses we do in the Odyssey community, we feel quite confident that we want to use as much data as possible because we've developed strategies to quantify that error and incorporate it into our analysis so that we can, if we don't have the highest quality data, we at least can understand what uncertainty that brings into our analysis.

Federica Santoro 28:35

So we ended the listener segment of the podcast. Thank you for taking the time to answer all of their queries. But if you would allow me a couple more minutes, I'd like to end with one final question, and that one's from

Using real-world data more

Federica Santoro 28:51

me. When I asked Daniele about real world data in the previous episode, he said there is added value in these sources of evidence. As to whether we should use them, that is still, I find, an open question. So considering everything that we've discussed, where do you stand on that? Should pharmacovigilance professionals be using real world data more? And if so, how?

SPEAKER_00 29:18

Yeah, I'm glad to end on this question because I'll give a very emphatic yes. I think when I think about the public health responsibility we all take in the drug safety space, it's our real responsibility to try to do everything we can to identify the potential harms of medications, to evaluate the magnitude of those risks, and to communicate that to patients so that we can improve the lives of everyone around the world. And so, from my perspective, we should all be doing everything that's possible to do that job as efficiently and as reliably as possible. And while there is still some science to be worked out about how we use re-world data in a pharmacovigilance process, the counterfactual is to not do it at all. And so one would have to believe the only way we shouldn't do it is if we actually believe that there's more harm that comes from using extra information and decision making than the good that could come from using extra information. And across all the areas that I've been in, I've never seen a situation where using more data is a bad thing. I'm also really excited about the possibilities that while spontaneous adverse events has a strong history in pharmacovigilance and continues to deliver high value, we have to recognize that that also has substantial limitations, and that doesn't prevent us from using them for drug safety purposes. And randomized clinical trials are tremendous ways for us to generate evidence, but it has tremendous limitations, and that doesn't prevent us from using it. And so I think we need to take the same open-minded approach to the use of real-world data and consider what we can learn from this information and how can we integrate it with what we're already learning. And I think we've seen enough successful demonstrations that real world data can help us characterize the exposed population, understand the incidence of adverse events, understand which patients are experiencing which types of adverse events, quantifying the magnitude of the effect, understanding the time to event distribution, understanding whether real-world patients experience de-challenge successes or rechallenge failures, and ultimately predicting which patients are potentially at risk of having adverse events. And I think given that we've demonstrated that this can happen, I think it's incumbent on all of us to figure out how we integrate real-world evidence into our own processes.

Federica Santoro 31:33

And I think that's a great place to end. I look forward to seeing the developments in the field, and I have great confidence that we will be able to solve a number of challenges. Thank you so much, Patrick, for being with me today and answering all these questions from our audience. I'm sure they'll find it valuable. Thank you for having me.

Outro

Federica Santoro 32:09

If you'd like to know more about real-world data and its use in pharmacovigilance, check out the episode show notes for useful links. The Eden Project has its own podcast, The Voice of Eden, where you can hear directly from the network's partners about the challenges and opportunities of working with real-world data. And spread the word on social media so other listeners can find us. Apart from these in-depth conversations with experts, we host a series called Uppsala Reports Long Reads, a selection of audio stories from UMC's Pharmacovigilance magazine. So do check that out too. Uppsala Monitoring Center is on Facebook, LinkedIn, and Twitter, and we'd love to hear from you. Send us comments or suggestions for the show, or send in questions for our guests next time we open up for that. For Drug Safety Matters, I'm Federica Santoro. I'd like to thank Patrick Ryan for his time, Matthew Barwick for post-production support, our listeners Jim, Josué, Judith, and Sudarshan for contributing questions, and of course, all of you for tuning in. Till next time.