Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Gerstein Science Information Centre

Knowledge syntheses: Systematic & Scoping Reviews, and other review types

Screening for articles

Searching for studies is different than screening for studies. When screening potential articles to be included in a knowledge synthesis, keep these pointers in mind:

  • Think specifically, and at a granular level
  • Decisions are made on the specific characteristics of each study, as defined by the scope of your topic, and that will answer your question. Eligibility criteria should mirror these characteristics
  • The process on how to handle uncertainty when screening should also be pre-determined, e.g. If the study is missing data, then include for now
  • Reviewers should have a high minimum of 90% agreement among them in their screening process

*Our librarians do not provide consultations on screening, however we're happy to provide you with the information and resources below.

First & second level of screening

First level of screening

In the first stage of screening, a pair of reviewers from the review team will independently scan titles and abstracts of articles that were retrieved from a comprehensive (i.e. multiple source) search, and make decisions whether to include or exclude articles. To do this in a streamlined, unbiased, and method-driven way, reviewers should adhere to the pre-defined eligibility criteria, or guidance form.

Keep these tips in mind during the first stage of screening:

  • Authors should generally be over-inclusive at this stage; a maybe is always a yes at this point
  • Screen for outcomes with caution if at all; they're poorly reported in the title and abstract. Save those for full-text!
  • Ask yourself: Is there enough information in the title and abstract to exclude this study at this stage, with absolute certainty?

Second level of screening

The second level of screening is a more rigourous, in-depth process in which the articles that were included in the first stage of screening are read in full-text. Similar to the first-level of screening, this is done independently by two reviewers from the review team, and the eligibility criteria that was used as a guideline for the first-level of screening is largely the same. However the second level of screening differs in these important ways:

  • The reason(s) for exclusion must be recorded and reported
  • You can now screen for outcome(s). Ask yourself: does the study report on the outcome(s) you're interested in?
  • Although the eligibility criteria is the same, it will require additional detail (clarifying questions may arise during the first stage of screening)

undefined

Internal & external challenges

Screening does not come without challenges. The process can often be confusing, and coming to a consensus on whether an article "fits" the eligibility criteria can be hard. These are some common challenges faced during screening:

Internal challenges:

  • Lack of methodological expertise
  • Belief that AI or technology will solve problems, or fix a lack of methodological knowledge
  • Poorly defined eligibility criteria
  • Lack of training among the team

External challenges:

  • Volume of the evidence
  • Poor reporting of research
  • Lack of clear how-to/practical guidance
  • Subjective nature of methods

Strategies for improvement

Guidance and training are two keys methods in mitigating challenges that arise during the screening process. 

Establishing a guidance form lessens reviewer confusion & anxiety during screening by providing a document that all team members can refer to. 

First level of screening - Steps to create a guidance form

  1. Determine all eligibility criteria for your study based on your question & scope. 
  2. Define each criteria. Extensively!
  3. Write out all variations (nomenclature, spelling, meaning, definitions) for all criteria with examples.
  4. Play the so what does that mean? game to a fault, until the the entire team is clear on all criteria and definitions. Be annoying!
  5. Once you feel confident on step 4, fill out the guidance form templates and adapt as necessary.

First level of screening - Pilot testing

  1. One person in the team should take the lead, usually the PI
  2. Select a set of articles retrieved by your search. You want a good variety of articles, some that are difficult/vague, some that are clear.
  3. Share articles among your team
  4. Using the screening guidance forms, ask your team to determine individually if each record should be included or excluded with reasons
  5. Collect the individual answers, and collate them blinding the name of each reviewer
  6. Meet to discuss the results of the pilot, and provide the correct answers, with reasons. This is the chance for the team to air out any confusion and solve it!
  7. Further clarify criteria based on the discussion. Clarify! Clarify! Clarify!
  8. Apply guidance consistently throughout the screening process

Second level of screening - Steps to update a guidance form

  1. Base guidance forms on the ones created for the first level of screening
  2. Enhance forms by adding (never subtracting!) details that are hard to screen for, or determine, at the first level
  3. Determine explicit steps to be taken should studies warrant further clarification, and who will be responsible for it
  4. Build a flowchart indicating in what order the reasons for exclusion will be determined, and reported

Second level of screening - Pilot testing

  1. One person in the team should take the lead, usually the PI
  2. Select a set of articles retrieved by your search. You want a good variety of articles, some that are difficult/vague, some that are clear.
  3. Share articles among your team
  4. Using the screening guidance forms, ask your team to determine individually if each record should be included or excluded with reasons
  5. Collect the individual answers, and collate them blinding the name of each reviewer
  6. Meet to discuss the results of the pilot, and provide the correct answers, with reasons. This is the chance for the team to air out any confusion and solve it!
  7. Further clarify criteria based on the discussion. Clarify! Clarify! Clarify!
  8. Apply guidance consistently throughout the screening process.

Screening webinar, examples, and templates

For a more comprehensive explanation on the process of screening, watch the webinar "Screening for studies in systematic reviews, scoping reviews, and other knowledge syntheses: Strategies for improvementcreated and presented by Patricia Ayala, Research Services Librarian at Gerstein Science Information Centre. To view the slides, examples, and screening templates, click here