Scientists are building giant ‘evidence banks’ to create policies that actually work
Funders launch projects with US$70-million to develop tools that make rapid syntheses of the world’s science
Education is one sector that is creating a database that brings together research on how to boost learning.Credit: Martin Bureau/AFP via Getty
Funders are injecting tens of millions of dollars into an ambitious plan to solve the biggest problem in science advice: supplying evidence to governments. Their goal is to build a system that allows policymakers worldwide to generate rapid syntheses of science that help them to make evidence-based policies aimed at solving critical issues such as climate change.
“We ultimately could benefit immensely from a world in which there is comprehensive evidence synthesis on every major social problem, and that is continuously updated and available to everyone,” says Will Moy, who leads the Campbell Collaboration, an international non-profit organization that supports social-science reviews.
Although researchers generate vast numbers of studies in areas relevant to policy, syntheses that show the weight of evidence on a topic are rare in many fields, and are not routinely used to guide policymaking. “There is huge demand” from policymakers for such syntheses, says Jen Gold, director of research at the Economic and Social Research Council (ESRC), a UK funding agency. “But supply doesn’t match it.”
Evidence syntheses are “everything the world knows about how to solve an important problem in one place”, says Moy. In medicine, doctors routinely use thousands of systematic reviews — meticulous syntheses of studies such as randomized drug trials — that show whether a treatment helps or harms. But most other fields lack such an extensive foundation (see ‘Shortfall of syntheses’). It can take months or years to extract meaning from a massive body of research — and funders have historically spent peanuts on synthesizing knowledge compared with the billions they spend on new research.
Source: Campbell Collaboration
To start correcting that, the ESRC and Wellcome, the biomedical-research funder in London, announced on 21 September that they are investing £9.2 million (US$12.2 million) and around £45 million, respectively, over five years in databases and tools that can help to synthesize research. UK science minister Patrick Vallance and Wellcome head John-Arne Røttingen announced the funding at a New York event linked to the United Nations’ Summit of the Future, a meeting to try to shape a better world, including through science.
Researchers have welcomed the news — thought to be one of the largest one-shot investments in evidence synthesis — and say that it’s timely because artificial intelligence (AI) advances are accelerating the process of finding and combining studies. “It’s so exciting,” says Isabelle Mercier, a researcher at the United Nations Development Programme who has been championing evidence syntheses at the UN. “Four years ago, this was too big to think about, but now we’re starting to see we can actually do it,” she says.
But AI is also making the mission harder, because AI chatbots such as ChatGPT can generate credible-sounding but potentially misleading summaries of research. “And the challenge is, how do you make what is really reliable stand out from what is not?” says Moy.
Slow and difficult
Producing syntheses is usually slow, difficult and expensive. Researchers undertaking a systematic review must scour databases worldwide of published and unpublished work for potentially relevant studies. Then they whittle a longlist of thousands of studies down to the most relevant few, rate their reliability, extract the data and combine the results, sometimes using a statistical method called a meta-analysis. Even once complete, evidence syntheses often don’t reach policymakers and quickly become out of date as fresh research pours out.“If a policymaker comes with a question, it shouldn’t take three months to find the research,” says James Thomas, a research- synthesis specialist at University College London. “It’s ridiculous.”
The problem became acute during the COVID-19 pandemic, when authorities everywhere wanted quick syntheses to inform decisions on drugs, masks and lockdowns. At first, scientists couldn’t provide them fast enough — but then they produced too many duplicate syntheses and shoddy reviews.
Scientists’ dream is that anyone, anywhere, could assemble a synthesis tailored to their question and part of the world, virtually at the push of a button. To do this, researchers would like to have 'evidence banks': shared databases of pre-selected studies tagged with information, such as method and location, and with data in a common format so it can be combined. Trained AI tools would do most of the drudgery of sorting studies and synthesizing data, with humans checking for quality — for instance, by assessing any bias in the underlying studies.
Some databases are partway there. The Education Endowment Foundation (EEF), a charity in London, has a database of more than 3,500 education studies. Using this, the organization has built an array of systematic reviews that reveal the impact on learning of tutoring, homework and class size. It shares the database and reviews with several countries to avoid others repeating the work. Ideally, “rather than doing six separate systematic reviews, you do one brilliant review that we share the capacity for”, says Jonathan Kay, who leads work on summarizing evidence at the EEF.
Living syntheses
The latest investments might eventually lead to a series of databases similar to the EEF’s, containing ready-to-synthesize studies for important policy areas such as environmental protection. From these, advocates would like to build banks of ‘living’ — or constantly updated — evidence syntheses that would reveal, for instance, what works to reduce climate change, improve mental health and boost youth employment.
During the COVID-19 pandemic, policymakers yearned for rapid evidence syntheses to answer questions such as whether drugs effectively treated the virus.Credit: George Frey/AFP via Getty
Wellcome intends to fund consortia that develop data platforms and tools that help to reach that point. This is “an unusual play from Wellcome”, says Tariq Khokhar, head of data for science and health at the charity. The organization is known for funding health research, but this money could help to make sense of evidence across any discipline. “This is really a foundation for anyone to build on,” he says.
The ESRC plans to fund one consortium to accelerate evidence synthesis and to develop trial versions of living syntheses in areas such as healthy ageing. The two efforts would be different but might overlap, says Khokhar.
The ESRC also wants its consortium to develop ways to make it easier for policymakers to use evidence syntheses. For instance, some UK civil servants are starting to use an AI tool called Redbox Copilot to analyse and summarize government papers and speeches. Researchers might build tools that funnel “systematic review evidence into that process as well”, Gold says.
The £55 million won’t be enough to achieve the type of seamless evidence synthesis that advocates dream of. But the funders hope to add more to the pot — and that this initial investment will encourage other funders to pitch in. “The idea is that any investments can build on top of a bunch of work that's already been done,” Khokar says.
doi: https://doi.org/10.1038/d41586-024-03100-2
This story originally appeared on: Nature - Author:Helen Pearson
