SPEAKER_00 0:16

We're trying to move from a world in which it was all based on publications and the authority of the people writing those publications to one in which we can say all that is fine and well, but we also want to give every reader access to the data as well.

Federica Santoro 0:31

Welcome to Drug Safety Matters, a podcast by Uppsala Monitoring Center, where we explore current issues in pharmacovigilance and patient safety. In this episode, we place clinical trials under the spotlight. Whenever you take a medicine or are administered a vaccine, you assume that it's been rigorously tested, that there's a solid system in place to make sure you'll get the maximum benefit with the minimum harm. But what if I told you that sometimes that system doesn't function as well as it should? That sometimes those same trials that are supposed to shed light on the safety and efficacy of medical interventions are either poorly designed or their results not reported properly. My name is Federica Santoro, and my guest today is Peter Dosci. Peter works as Associate Professor of Pharmaceutical Health Services Research at the University of Maryland School of Pharmacy in the US. He is also an associate editor at the BMJ, the British Medical Journal. But above all, he's a passionate advocate for greater transparency in clinical trial data and has been widely recognized for his efforts in that arena. I sat down with him to chat about access to data, trust in medicines, and icebergs. So let's dive right in. You think there's a real problem with clinical trials nowadays. What is it?

SPEAKER_00 2:01

There are many problems. Let's start off with what's good, though, about trials and in particular randomized trials. What's good is a couple things. Methodologically, they're very powerful tools to discern, especially subtle effects. So drugs can have subtle effects that are beneficial. They can have subtle effects that are harmful. If the effect is subtle, it's very hard to know what that effect is without some kind of test where you can really discern there's a real effect here. Trials are a powerful tool for that. In addition, trials are powerful, I think, because they are a methodology that the world has adopted and there's wide consensus that we can trust the evidence we get from trials. And that trust is also important, right, for consensus building and that there's a lack of constant concern over do we really know, do we not know? A trial often is able to settle a debate as the definitive truth. Whether or not that's true, separate question, right? But it has that power in our society. And so I think those two aspects are very important to understand. The problems, though, are that trials are not necessarily designed well, and the design can be undermined by things like studying the wrong endpoint, looking at an endpoint that really doesn't matter to a patient. Maybe it matters to a clinician, but does it matter to the patient? So you can do a trial, and the trial can be immaculate, fantastic, well performed, great follow-up. Everything looks good in terms of its internal validity, but it's answered a question that perhaps one could argue wasn't a question that needed answering. So that's one of the big problems is from the start, a trial might have design problems that lead towards results that don't have importance. They can also be, of course, undermined by problems in the design or conduct of the study while it's ongoing. We always have known about that. Even when trials are answering good questions, though, another problem can occur, which is we may not know what exactly happened in the trial. The trials may not be reported well. And that can happen for unintentional reasons, it can happen for intentional reasons, right? There's a number of reasons why what we see in the literature may not fully, adequately and completely reflect what actually went on in that trial, both in its design, its conduct, and the analysis. That poses a huge problem for evidence-informed decision-making, because if we don't have an accurate picture of what happened in the trial, we won't really be able to make our decisions on sound evidence. And if you look at this broader than a single trial and you look at now a host of trials, if we don't see all the trials, what people call publication bias, where not all trials are out in the literature, we have an additional problem in that maybe we're seeing trials that are published properly, but they're only half of the story. So we have a sort of a host of problems from the start and their design all the way through the reporting that can undermine the entire process, which they were designed to answer. Basic questions about how does a drug work?

SPEAKER_01 5:11

How do we get to this? How do we get to this situation where trials are poorly designed and results are not reported as they should?

SPEAKER_00 5:19

Oh boy, what a what a tough question. How did we get here? We did the history of um trials. Well, trials sort of ascended into prominence over the last half century, in particular after thalidomide and the regulations changed regarding the types of evidence that were necessary to put a drug on the market to demonstrate the safety and effectiveness standards that currently exist. So trials uh gained prominence for that reason. Where things perhaps went astray is probably hard to pinpoint. But certainly, my sense is that as the RD pipeline became one in which drugs were not as remarkable in terms of their effects as they were in a previous day, there became more and more of an incentive to try and demonstrate that your drug is better than something else on the market in ways that the evidence itself couldn't tell. If the effects of a drug are very impressive, you don't need large trials. You can do trials on small numbers of people because the effects are so large, you will see statistical significance with those small numbers. As your effects become much more smaller and subtle, that's when the trials need to get bigger and bigger in order to show statistical significance, right? Some basic statistical principles. But as that effect gets smaller, unfortunately, I don't think the data sells itself now anymore. And I think Donald Light has a concept called the inverse benefit law. Essentially, his point is that as the benefit decreases, the need for marketing of the drug increases. And I think that's part of what's going on here when we talk about the problems with trials. There's many components, though, and that's only one of them. Another component, of course, is what are the regulatory standards for approval. These are issues related to basic definitions of what do regulators require for a drug to get on the market. And if the hurdle, if you will, of what it is required to get on the market is set at a certain position, then many sponsors will understandably set that as the hurdle that they need to clear. And so if the hurdle is not set properly and is set on, say, a surrogate outcome that we don't know with certainty predicts clinical benefit, then we run the risk of performing a trial or allowing somebody to perform a trial that's answering a question that we still have uncertainty about whether or not that is going to result in a true clinical benefit. So that's another part of this is what are the expectations of trialists as based on the standards set by regulatory agencies?

SPEAKER_01 8:01

So when we talk about reporting bias or publication bias, just to be clear, where is the problem?

Federica Santoro 8:08

Where does the under-reporting or the non-publication happen? So is it the pharma company conducting the clinical trial that is keeping data from the regulatory agency? Is it stuff that is not published out into the public literature? Where is the roadblock?

SPEAKER_00 8:25

So publication bias refers generally to the lack of publication in the literature. Journals that are indexed by PubMed. That's where most people think about publication bias. So it's it's not the case of companies not submitting their data to the regulator. You know, one can always point to exceptions, but I think it's a fairly reasonable assumption that the regulator sees and has the power to see most of everything they want to see.

SPEAKER_01 8:53

Aaron Powell So then where's the problem?

Federica Santoro 8:54

Because if assuming the trial has been conducted correctly, right, if the regulatory agency has all the necessary information and we assume that the regulatory agency can take an informed decision based on this data submitted by the pharma company, then where's the problem?

SPEAKER_00 9:08

Aaron Powell The problem is in the assumption that because a regulator has seen it, because the regulator has evaluated, and because the regulator controls what goes on the labeling, that that also limits and controls appropriately what we see in the literature. The literature is not regulated. There is no requirement, of course, for a journal editor to send an article to a regulator and say, Do you agree with this? Are we approved for publication, right? That doesn't occur. So 10 studies can be submitted to the FDA. And of those 10 studies, only seven may be published. So if you only look at the published literature, and who's only looking at the published literature? People who do systematic reviews generally look at the published literature. A systematic reviewer, and this is a good thing, wants to conduct an independent assessment. Because if the systematic reviewer said, well, all I need to know is what FDA said, then what's the value added, right? There's no independent assessment. So if you want to conduct an independent assessment, the idea has been, well, how do we do this in a systematic fashion that leads towards the least biased estimate of the drug's effects? And the concept has been, well, you search the literature far and wide to make sure you find everything that possibly is known by this. But that's undermined if you happen to also know that the literature doesn't reflect everything that's been done. So that's the big publication bias problem. The reporting bias issue is far more, I think, complex in that there's many things going on here from outcomes that are selectively reported, where a trial might have looked at five different outcomes, calling one its primary outcome, four secondary outcomes. And in the published manuscript, we see three outcomes reported. And if you had access to the earlier documents, you'd realize the three outcomes were actually all secondary outcomes, and we don't see anything on the primary outcomes. So what happened? But you wouldn't know this looking at the journal article itself. So that's selective outcome reporting type of reporting bias. It could be the case that the five outcomes were studied in the trial, and all were reported in the paper, but it's the way in which the discussion is written that leads towards emphasizing one now more and sort of a spin on the results that you might say, well, that's I'm not sure that's really supported by the data. They're saying now that the secondary outcome, and they're not really emphasizing that this was a secondary outcome or exploratory outcome when they did the trial, they're not talking about this as really the key take-home result from the trial. They're saying this is really the most clinically relevant. And one could ask, well, if that were the case, then why wasn't that your primary outcome? Why didn't you power the trial with a hypothesis to actually test that outcome? Form a hypothesis and test your hypothesis about that outcome if you think it's the most important. Why only after the trial's finished do we now hear that that's actually the most important outcome? Right? That you actually hadn't designed the trial to study in terms of hypothesis testing basis. You were simply studying this as an exploratory endpoint.

Federica Santoro 12:14

Perhaps no other case exemplifies the problem better than the Tommy Flu saga. In 2009, Peter and his colleagues set out to answer a simple question about the anti-influenza drug osultamivir. Does it work? That question had important consequences both for health and the economy. The H1N1 swine flu pandemic was spreading worldwide, and governments around the globe had been spending big chunks of their healthcare budget to stockpile osultamivir, sold by Swiss company Roche under the brand name Tamiflu. An earlier review had concluded that the drug reduced the flu's severity and prevented more serious complications. But as Peter and his team were getting ready to update that review, someone pointed out to them that even though Tommyflu had been tested in more than a dozen clinical trials, not all results had been published in the literature. And so the battle began. After nearly four years of pressuring Roche into releasing the underlying data, the reviewers finally got hold of full clinical study reports. Hundreds of thousands of pages. With that new evidence, the group concluded that Tommy flu didn't make much of a difference in preventing influenza complications or infections. And they also raised new questions about the drug safety. But the greatest legacy of the investigation is that it helped galvanize a general movement towards greater transparency of clinical trials.

SPEAKER_01 13:50

So what do we do about this?

SPEAKER_00 13:52

Well, I can tell you what I'm doing about it. I think there's a lot of things that need to be done at various levels. What I'm doing is running a support center. We call it the Riot Support Center, Restoring Invisible and Abandoned Trials. And what we do is we attempt to counter both of these problems of trials that are invisible, wholly invisible, because they've never been published. And so although they occurred and tons of documentation was collected and patients were recruited, and maybe even a report was internally written and submitted to regulators, we don't see anything in the published literature. We want to get those trials published. And we also want to correct the misreported trials, right? Trials which are somehow resistant to the correction of error, even after people have pointed out that there's errors that need correcting. That also, of course, is an important problem that we want to address. And we think that the key to both of those problems is access to data. And people who are willing take the baton and say, look, if the original investigators or sponsors are not willing, not interested, don't have the resources or whatever it is, they're not going to at the end of the day publish the unpublished trial or correct the misreported trial, then somebody else can. And how can they? Because they now have access to the data. So even without the permission of the original investigators or sponsors, someone else can go ahead and do this. And so access to data is enabling this because it's one thing to say, well, I've heard and I have a sense that there's a problem in the way this trial was published. Or I know that you did this trial because I saw this in a trial registry, but I know you haven't published it. So please go ahead and publish it, would you please? And they say, Well, yeah, we're going to get to it, and then one year passes, two years passes, and still hasn't happened. So what can you do? If you don't have the data, you really can't do anything except keep telling them, well, Ian, why aren't you publishing? But if you can get access to the data either directly from the investigators or indirectly by getting them from a regulator, say, through a Freedom of Information request, you're in a position where you can do your own analysis of the data and you can publish that in a peer-reviewed journal. Go through the process of getting that published, have it refereed like every other paper. But we advocate, in addition to just having the journal article, that you also have the data available. We're trying to move from a world in which it was all based on publications and the authority of the people writing those publications, which I think describes well our contemporary world, the world we've been living in, to one in which we can say, all that is fine and well, but we also want to give every reader access to the data as well. So they can do what they want with it, they can spot check it, they can look at it for their own analysis of it, a disagreement. And I think that's what science is. To have an open exchange of ideas with the access to data there for everybody.

SPEAKER_01 16:44

What made you decide then to get involved with Riot and the Support Center?

SPEAKER_00 16:48

So the idea just came to me uh one day when I was listening to a seminar by one of the Riot Declaration co-authors, Swarup Vejula. And he was talking about the case of Gabapentin, which he had done his PhD research on. And he was talking about the unpublished trials. And he knew all about these trials in great detail because he had access to the internal clinical study reports, hundreds of pages on these trials, details and details and details. In some cases, he had the electronic data sets. I mean, he just had enormous amounts of detail, yet he was referring to them as unpublished trials. And it just it sort of hit me. It's like, why are we continuing to talk about these as unpublished trials, refer to unpublished as a problematic category that the trials should be published, when it seems we have all the data we need to actually publish these trials. So why don't we just do it? And so that's where that was the germination of the idea. Maybe you don't actually need the original investigators or sponsors to do the work. If they're not planning to do it, not interested, then somebody else can when you have access to the data. So that was the beginning. And we published the idea in BMJ, and we opened the Riot Support Center to try and sort of jumpstart the movement. You know, it's very tough because this is not quick work. You can't analyze a trial in a week. It's slow work, it has to be done carefully, right? It requires concentration, it requires a team. And those things don't come cheap and everybody's busy. Right. And so we wanted to try and figure out well, how can we help groups that want to do this, that do have the interest, but are having trouble finding the time? Or the problem might be that they simply don't know exactly how to get their hands on the data, would love to get their hands on it, but don't know how. And as a result of not knowing how, don't have the time to really figure out how, because who knows how long that will take to figure out. And so in the end, don't do that. Even though it might be something that if you said, Hey, I've got everything you need to get going, would you like to do this? They say, Oh, of course. This is important work, it needs to get done. So we're fortunate enough to find funding to buy out some of our time to provide that support free of charge to people, to say, need help finding data, we're here to help. Need help understanding the landscape of data. Because many academic investigators who don't have experience in industry or the regulatory environment are not aware of what types of documents and data are collected in the course of a clinical trial.

Federica Santoro 19:28

I can imagine it's easy to get lost among all those documents or just not even look nowhere to look for them.

SPEAKER_00 19:34

There are masses and masses of these documents. If you're a regulator, you know how to navigate these. You understand how they're structured and ordered. If you're not, it takes time to learn that. And so that's the kind of knowledge that we've built up over the last decade now, in particular working off our experience with Tammy Flu. That we've, you know, we've started to understand this landscape in a way that we can now assist others and help them on their journey to understand this. Because I think really diving into the detail is absolutely key to understanding the effects of interventions. Regulators don't allow a company to send them journal articles and say, okay, based on our reading of the journal article, we're deciding to approve or not approve your product. Regulators demand detail. They require detail in order to determine the benefits, the risks, and make a judgment as to whether benefits outweigh risks. That's what they require. And a systematic reviewer or a clinical guideline writer, anybody working in in decision making, trying to understand the effects of interventions, I think deserves access to the same information.

Federica Santoro 20:38

The problem I see though is that you still have to get your paper published in a journal, right? So that means you have to get past journal editors. And it's a general problem in biomedical science that journals don't like to publish negative results.

SPEAKER_01 20:52

So how do we go about that?

SPEAKER_00 20:54

I think that's more of a historical problem than it is a contemporary problem. There are journals that are, you know, now openly advocating for people to send their negative results to them. So I think in times past, that was a concern that journal editors, their views, their you know, interest in more higher impact, exciting results shaped the ability of investigators to have their studies published. I don't think that's the case anymore in the sense that there are always places one can publish their research today. Does it mean that you can get any paper published in a high impact journal? No. But that shouldn't necessarily be the goal of every research project. So I do think that we've at least passed that hurdle of journal editors being a barrier to people getting results published. Sometimes a negative trial is is a negative trial because the harms have outweighed the benefits. And that became very apparent to the people in the trial. If that kind of trial is not published and there is a risk for somebody else to Repeat that experiment. And now we're talking about an ethical, right? A fundamental ethical principle that the trial was there to create knowledge. If that knowledge is not disseminated and people are put at harm as a result of that lack of knowledge separation, then we have a serious ethical question about that second trial, whether that should have ever gone forward in the first place.

Federica Santoro 22:21

That's a really interesting point you bring up, because it's not only about not disclosing information about the efficacy of the product, but also about the harm it's it causes. And so bringing it back to pharmacovigilance, then what do you think the impact of all this situation around clinical trials is for drug safety?

SPEAKER_00 22:39

Aaron Powell The research on research, the research that has looked at other papers, in particular research that has compared the same trial published in a journal versus that trial's report in a more internal, longer, unabridged format, like a clinical study report. So same trial looking at it published in two different places, if you will. One of which is a journal short, one of which is long and detailed granular. They found pretty consistently that the number of adverse events and types of adverse events are underreported in journal articles. So I'm hoping that with access to the data, we can do far more justice to a full and complete reporting in journal publications of the harms profile of drugs, or if you will, the adverse event profile of drugs without any suggestion of causality. Just the information that was collected about adverse events in trials. That's currently not completely reported. It's almost, you know, a general practice because that wasn't the primary efficacy endpoint. And so not much attention is given to that issue. So with access to the data, not only can we publish more complete reports, but we can also have the full data set available for people as a supplemental appendix in an archive somewhere, a repository, so that it's much more easily accessible. I think there's a lot to be learned on the harm side from trials that we haven't tapped into because it's just been inaccessible to people. And as we move into a world of greater access to clinical trial data, of course, I mean the detailed granular clinical trial data. I think there's a lot of new methodology that can be developed around the study of harms here.

SPEAKER_01 24:19

Aaron Powell And I noticed the riot symbol is an iceberg, is that correct?

Federica Santoro 24:23

Yes. I guess that's an apt image for the amount of information that is lying there and needs to be brought up to the light.

SPEAKER_00 24:30

Aaron Powell That's right. It's um a visual representation that what one sees is just a fraction of what actually exists for every clinical trial. And so what we're used to thinking as the clinical trial is a journal article or a conference presentation, maybe it's a poster, an abstract. These are the things that we're used to seeing as academics. That's just a small, small fraction, right, of all the documents, data that were collected in the course of running that clinical trial. And so FDA and industry, they've been understanding the trial by what's underneath the waterline. They're not relying on the tip of the iceberg. And I think that more and more people need to become divers to get underneath the water and start really evaluating things based on the totality of the evidence and not just the selected representation that we find in the literature.

Federica Santoro 25:21

Unfortunately, the debate over clinical trial transparency is not behind us yet. In a commentary published in the European Journal of Internal Medicine last year, Hans Gerg Eichler and Guido Razi from the European Medicines Agency point out that clinical trial publications remain far from ideal. For starters, trials undergo design changes more often than we'd like, and those changes are rarely reflected in the final journal publications. That's why, the authors write, relying solely on the publications of clinical trials and scientific journals as the basis of healthcare decisions is not a good idea. Drug regulators have been aware of this limitation for a long time and routinely obtain and assess the full documentation. And so, the work of the Riot Support Center is as relevant as ever. Since its establishment, the center has helped restore four trials and has proposed about a dozen more for restoration. They're also now compiling links to various study documents for COVID-19 vaccine trials. And Peter has been quite vocal about the need for transparency even in the face of a pandemic. In a recent article for the BMJ, he encourages doctors and professional societies to publicly state that, unless the underlying data are also publicly available, they will refuse to endorse COVID-19 products as being based on science. For what do we gain from withholding data? he asks. We risk overestimating a product's benefit, underestimating its side effects, and perhaps more importantly, eroding the public's trust.

SPEAKER_01 27:10

What does this all mean for the public's confidence in medicine? Because you said that clinical trials are the golden standard and they're the cornerstone of the general public's trust in medicine. So if they're not conducted properly, what does that say about the drugs we have on the market and about their safety profile?

SPEAKER_00 27:31

You know, it's hard to quantify the degree to which things aren't published properly, the degree to which there's error in the literature. It's very hard to quantify this because our evidence around this problem is really driven by a lot of case studies, gabapentin or osultamavir or tamiflu. These case studies have widened our understanding of the problem, but it's still hard to really put a sort of point estimate of the prevalence of this problem. So it's it's not something that I can say, you know, is it widespread? It seems to be far more widespread than people might have expected 10, 15 years ago for sure. How widespread is it? It's still hard to say. In terms of the trust, I honestly don't know exactly what the elements are that lead towards public trust. Is it an understanding that drugs are approved based on rigorous phase three randomized trials? I'm sure that's the case for some percentage of the population. Not everybody, though. Not everybody is going to have that level of detailed knowledge about the process. I think for many people it's going to be their trust is based on an expectation that the system only allows drugs on the market that are safe and effective, for that's the function of government is to assure that level of quality. And so it's it's more an expectation that one can trust the system at large to be producing things that are safe for consumption. We have that expectation with food, we have that expectation with household products, right? With our cars. We don't know all the details of how it works, but we know that there's supposed to be experts and public officials whose job it is to make sure that all this is working. And the problem I suppose in medicine and the evidence underlying medicines is that the people who we trust to you know make sure things are working as they should be are often people who don't have intimate knowledge of the problems with evidence-based medicine. So, for example, the average doctor is not rigorously doing their own evaluation of all the drugs they're prescribing. The doctor, him or herself, is relying on the system to ensure that the only drugs that they could dispense are ones which there's rigorous evaluation and assurance of substantial benefit. And so there's a need for, I think, a greater training regarding what are the sources of evidence and how reliable are they. You know, evidence-based medicine is a very powerful tool in that it um has allowed for a generation of people who are trained in this area to think far more critically about evidence rather than relying on shortcuts such as is it published in a well-known journal? Was it peer-reviewed? And if those boxes are checked, then we know it's right. Or one step further, publish in a peer-reviewed journal that I know, and there's a p-value that's less than 0.05. Therefore, it's trustworthy. Evidence-based medicine is saying, hold on a sec, we need to go a lot slower and really look into the details here. And that's where I think training at the level of medical pharmacy education can help a great deal to say, okay, you know how it's supposed to work. Now let's look a little bit closer at how the evidence is actually generated and become a bit more self-critical about what the things that we thought we could trust and how trustworthy are they. And that's where this engagement in the process, you know, real critical thinking does not happen quickly. One can be skeptical very quickly, but to get real into the real details, it takes time. And um I think there aren't many rewards for taking that time today. It's often impressive to be the person on rounds in the hospital that can tell you about the latest study and all the details, and it looks very impressive. But a real engagement with literature doesn't happen that way, right? It's a slower, more deliberative process. And that's what I work on as an educator to really try and cultivate amongst the students a passion for going deeper, for asking for more detail. And over the past uh decade, I'd say, in particular the last five, six years, we have made strides in getting a lot more people aware that there's whole degrees of granularity, far deeper than we ever thought, than the journal article.

SPEAKER_01 32:05

One final question. What would you say has been your greatest accomplishment so far?

SPEAKER_00 32:09

I think I'm I'm very happy that the work that we've done related to broadening access to clinical trial data has, I think, and you can see this with the Tammy Flew case, which you know follows me wherever I go, that people know this story. I'm very happy that this has helped highlight the enormity of the iceberg. Awareness that there's an enormous amount of detail for clinical trials that actually exists yet most people don't have access to. And I think that realization has spurred many people to say, hmm, that doesn't seem right. We believe in transparency, so why don't we have all the data? I don't understand why independent investigators don't have the data. And that broad sense across the community is what's helping push changes. And I think that I am happy that our story of Hammy Flu, our going through years and years of trying to get data, really helped highlight what a huge problem this is and how important it is to solve it. And many people know this now, that this is an important problem that needs solving. It hasn't been solved yet, but we have made progress, and that is an important accomplishment to recognize.

Federica Santoro 33:20

Great. Thank you very much for joining the UMC podcast. You're welcome. That's all for now, but we'll be back soon with more conversations on medicine safety. If you'd like to know more about clinical trial transparency and the Riot Support Center, check out the episode show notes. If you'd like Drug Safety Matters, subscribe to it via your favorite podcast player. And don't forget to check out the Uppsala Reports Long Reads, the best stories from our pharmacovigilance magazine brought to you in audio format. If you have any suggestions for the show, reach out to us on social media. You'll find Uppsala Monitoring Center on Facebook, LinkedIn, and Twitter. And you can join the conversation there with the hashtag drug safety matters. Thanks to Matthew Barwick for postproduction support and to you for listening. Till next time.