At the OAD we are in the process of compiling a library of evidence relevant to astro4dev projects. Until this is ready, here is a collection of other resources for obtaining evidence on interventions.
The Campbell Collaboration “promotes positive social and economic change through the production and use of systematic reviews and other evidence synthesis for evidence-based policy and practice.
The Campbell Library is a library of systematic reviews. The reviews focus on social welfare topics. Reviews are conducted following Campbell Collaboration protocols and guidelines, which require registration of a review title and protocol prior to the completion of the review. Campbell Reviews are thus higher quality reviews than many systematic reviews published in journals. However, most Campbell Reviews include non-experimental as well as experimental evaluations. It is important to therefore pay attention to the quality of the studies found in any given review and weight its conclusions accordingly.
Topics: You can search the Campbell Library by keyword and by categories. The category topics are: Crime & Justice, Education, International Development, Methods (i.e. evaluation and research synthesis methods), Nutrition and Social Welfare. When searching, it is useful to restrict your search to “Review” (i.e. do not tick “Title” or “Protocol”); otherwise, your results will include the review title registrations and/or protocols for reviews that have not actually been completed.
Cochrane Database of Systematic Reviews (CDSR)
The CDSR is the original and most comprehensive systematic review library in the world. It focuses on reviews in the health care. You can search the database using keywords or through topics. Cochrane Reviews set the ‘gold standard’ for systematic reviews, applying rigorous inclusion criteria and conforming to strict procedural and reporting guidelines. In most cases, only randomised controlled trials (i.e. experimental impact evaluations) are eligible for inclusion in a review.
Topics: Anything related to public health; some reviews are also topics related to science communication (in the context of communicating with the public about medical issues) and education (in the context of medical education). For example, topic categories include consumer & communication and promoting acquisition of skills and competencies.
What Works Clearinghouse (WWC) – Practice Guides in Education
The WWC aims to offer reviews of “existing research on different programs, products, practices, and policies in education.” Also contains a library of over 10,000 primary studies, focused on U.S. American evidence. Provides easy to use summaries of evidence, based on their systematic reviews, for specific topics. Click on a topic and a list of interventions/activities pertaining to that topic will appear.
Topics: Education, including pre-school, primary and secondary school as well as science and mathematics specifically. For example, clicking on “Science” on 13 December 2016 produced a list of 7 results with titles that indicate the intervention (e.g. “Astronomy Resources for Intercurricular Elementary Science (ARIES): Exploring Motion and Forces”) and an adjacent column showing the age/grade level of the students involved (in this example, Grade 6). Next to each intervention, an icon appears. If the icon is a coloured,, there is evidence that the intervention had positive effects. If the icon is a grey, the evidence has been found to have no or negative effects (as in the example Astronomy intervention here).
If you click on a specific intervention or click to compare interventions, you can get more detail on the intervention itself; the studies conducted to evaluate it; and the WWC summary of the evidence about it. This includes an Effectiveness Rating: A summary of the effectiveness of an intervention in an outcome domain, rated on a scale of 5 from evidence of negative effect (indicated with –) to no effect (0) to positive effect (++) based on “the quality of research, the statistical significance of findings, the magnitude of findings, and the consistency of findings across studies”.
Note that the WCC summaries of evidence are based on systematic reviews but the systematic reviews include non-experimental evaluation studies. Even though the WWC gives you a rating of the strength of the evidence, we strongly recommend that you look at the individual studies listed for a particular intervention to check whether at least one or two experimental (i.e. randomised control trial) impact evaluations have been conducted.
3ie Systematic Reviews
Topics: This database covers systematic reviews of the effectiveness of social and economic interventions in low- and middle- income countries. The reviews are drawn from many sources, including from the Campbell and Cochrane Libraries. The database contains summaries of the reviews that include findings, methodologies and quality appraisal of existing reviews and protocols of ongoing studies. The OAD database of reviews uses
National Bureau of Economic Research (NBER)
Topics: Working Papers, book chapters and other publications on NBER tend to be open access. Although they are not necessarily peer-reviewed (Working Papers are not), they are more likely to be methodologically sound than papers retrieved from random journals in a Google Search! Indeed, economists tend to use fairly rigorous scientific approaches. Further, this database contains some important and fairly seminal papers concerning evaluation and social research methodology. This database can therefore be a useful source of primary studies, reviews and examinations of evaluation methodologies if you fail to identify any systematic reviews that completely address your questions.
Topics: The OECD iLibrary is the online library of the Organisation for Economic Cooperation and Development (OECD) featuring its books, papers, data collections and statistics. It is also the gateway to OECD’s analysis and data. It replaced SourceOECD in July 2010. OECD iLibrary also contains content published by the International Energy Agency (IEA), the Nuclear Energy Agency (NEA), the OECD Development Centre, PISA (Programme for International Student Assessment), and the International Transport Forum (ITF). OECD iLibrary presents all content so users can find – and cite – tables and databases as easily as articles or chapters in any available format: PDF, WEB, XLS, ActiveChart, DATA, ePUB, READ.
Evidence Appraisal Frameworks
Study authors often forget or choose not to report important details. Omissions can be particularly difficult to spot if you’re a non-specialist: for example, an astronomer reading about a study in education. The Equator Network makes available field-standard reporting guidelines, which are used to promote transparency and quality in health research. These guidelines are designed through extensive research and consultation processes. The guidelines specify what should be reported in different types of studies (e.g. in systematic reviews, primary evaluations, descriptive studies and so on). These reporting guidelines can help you approach studies in health, development and education with a checklist so that you can more easily spot significant errors or gaps. The website also provides a guide on how to determine which guideline is appropriate for the article you are reading!
This link to a PDF provides a useful background piece explaining the use of global trial registries to support evidence-based practice across disciplines. Other trial databases and registries are listed below.
The American Economic Association’s Trial Registry
Started in 2012, this is the AEA’s registry for randomised controlled trials in economics. Trial registration is free. While researchers are encouraged to register before beginning a trial, completed trials can also be retrospectively registered (as noted on their registration). Evaluations must use an experimental design. Behavioural experiments are included. A search for “education” on 13 December 2016 yielded 10 results.
As of December 2016, keywords that can be searched include: education, agriculture, electoral, environment & energy, finance & microfinance, governance, health, labor, post-conflict and welfare.
3ie’s Registry for International Development Impact Evaluations (RIDIE)
RIDIE is a prospective registry of impact evaluations related to development in low- and middle- income countries. Researchers can register any development impact evaluation that rigorously attempts to estimate the causal impacts of a programme, including experimental as well as quasi-experimental designs”. Behavioural laboratory experiments are not included. A search for “education” on 13 December 2016 yielded 48 results.