EES3001 Professional Scientific Literacy

Q: Teamwork/Work Division

• How to divide up the work?
• Is there an easy way to do things in a group, or will one person be forced to do all the work?

A: This is something you’ll need to discuss and decide as a group – teamwork is a crucial skill in the workforce, including among research teams, so this is a great opportunity to get practice!

 

Q: Citation Chasing

• Once the articles have been established, could we then source articles that are referenced in the ones we’ve selected from our initial search?

A: Yes! This is a fantastic way to find additional research on your topic. For key articles on your topic, look at a) its reference list (review articles are great for this!), and b) who’s cited that particular paper in more recent publications. This is what’s known as “forward chaining” and “backward chaining” your references.

 

Q: Where to Search

• Does using the "Start your search" bar on the library homepage provide coverage for all articles within the university's databases?
• Not sure of the difference between using a database search vs. the regular library search?

A: Unfortunately no! Despite appearing to be an "all search" option, this tool does not capture everything and will often bring back loooots of irrelevant results, which is why we recommend searching within databases directly. Furthermore, many databases offer more thorough search abilities and filtering options, which allows you to create a more focused search. That's not to say you can't use it, just be aware of its limitations, especially for in-depth search methods such as rapid reviews.

 

Q: Number of References to Use/How Much Data to Include

• How many references is considered legitimate?
• How many papers is too many papers?
• How to further narrow down results if you still have thousands/hundreds?
• What is the most effective way to go about reporting on such a large amount of literature?
• When literature is obtained, how exactly do you narrow down to your specific needs besides using keywords in the search?
• Do I really need to include all data, even the ones I don’t like/do not find needful?

A: This will depend on your topic and how much or how little published literature there is on your topic. There is no f you’re getting an unmanageable number of results from your searches, you might consider:

• Refining your research question to make it more specific/narrow down the number of hits,
• Tighten up your inclusion/exclusion criteria and/or your critical appraisal criteria to help weed out the masses, or
• Set some search limits in place, such as publication years for the existing literature (e.g. last 5 years, last 10 years, etc.) – this tactic would need to be outlined and justified in your research protocol.

It’s certainly legitimate to exclude results that aren’t helpful based upon your evaluation and appraisal criteria, but be cautious about excluding results that don’t line up with your personal opinions and preferences – objectivity is key!
 

Q: How many articles is typically considered sufficient for a rapid review vs. a systematic review?

A: There is no magic number of studies distinguishing the two types of reviews; rather, the difference is in the goals and the process for how the review is conducted (e.g. search strategy, inclusion/exclusion criteria, and critical appraisal). Please refer to slide 5 on the EES3001 PowerPoint to compare the different types of knowledge synthesis and key differences between them.
 

Q: Rapid Review Documentation

• How thorough is the documentation that is necessary when conducting the research?

A: Be as thorough as possible! Recall that key elements of rapid reviews are that they are transparent and replicable. Slides 23 and 28 from the EES3001 PowerPoint posted at the start of this guide cover some key elements to include during documentation, as well as further guidelines from the Collaboration for Environmental Evidence (CEE).

 

Q: Combining Search Materials

• When pulling sources from large databases (e.g. Scopus + Web of Science), do we need to specify which articles were found from which database? 
• What is an organized way to report findings from different search outputs?
• How can we combine results from different databases?

A: You wouldn’t need to link individual articles to specific databases, no – it’s sufficient to report the number of results obtained in total from all of the databases searched as part of your documentation. These can be reported using flow diagrams such as PRISMA or ROSES. CEE guidelines also provide some helpful details for the level of depth required when documenting and reporting your searches (see section 5.3 of Conducting a Search). To combine results from different databases, you can export your search results into citation managers (e.g. Zotero) and then de-duplicate articles within the software. 

 

Q: Duplicate Results

• Will there be a lot of duplicate papers if I search for the papers using the same keywords, but on different search engines?

A: Potentially, depending on your research topic and the resources searched. So for example, the Web of Science and Scopus databases have a lot of content overlap, but also unique results that would be found in one and not the other. The same applies to search engines like Google and Duck Duck Go. Frustrating for the researcher, but important for avoiding resource bias! Hopefully Zotero's deduplication feature helps with this :)

 

Q: Do we keep track of all grey literature searches? Especially the ones which are not credible but has embedded links to peer reviewed journals or other credible sources?

A: Documenting the resource and search strategy you used to get to those embedded results should be sufficient. So for example, say I did a Google search and found some quality references linked on a Wikipedia page. I would document the search engine I used (Google), the search string I used, the number of hits the Google search turned up, etc., and make note that citation chasing was used to find embedded articles from the resulting list.

The EES3001 PowerPoint has some more details on what to include when documenting your search. 

 

Q: Should priority be given to articles with higher citation counts?

A: Citations can be great indicators of papers that are prominent in a field. Keep in mind that citation count is not necessarily an indication of quality and that time required to accumulate citations is also a factor. Ultimately your selections will be determined based on your inclusion/exclusion criteria and/or your critical appraisal criteria as determined in your protocol.

 

Q: Evaluating Results

• Subjectivity in screening process/exclusion & inclusion? How about confounders and studies that do not acknowledge them?

A: The type and degree of screening criteria will vary depending on your topic and research protocol; for examples, we recommend looking over some sample protocols and reviewing the resources outlined in the EES3001 PowerPoint.

 

Q: Evidence Synthesis

• Are you allowed to mix a quantitative and qualitative approach in a review?

A: Yes! Rapid Reviews may involve a mixed-methods approach.

 

Q: Remaining Unbiased/Relaying Implications of Research

• In essence, we do not interpret what we found based on all the evidence we found? (for instance, like a discussion on a journal article?)
• If they are to inform policy decisions, should the language remain innocuous? 

A: This is tricky! As noted in class, you can certainly relay the implications for existing research, but must aim to do so in an unbiased way – recall, the goal of a systematic review is to inform, not to advise. You are providing the best available evidence (and their implications) so that policy makers can make an informed decision.

 

Q: Reporting/Paper Formatting/Examples of Systematic Reviews

• How should we arrange the final project?
• What format is a rapid review presented in?
• A few examples of rapid reviews would be helpful.
• Is there a database of rapid reviews previously done on topic x? A systematic review of systematic reviews?

A: Meander through some completed reviews on the Environmental Evidence page to get an idea of formatting and general content; look for examples that have “systematic review” in the title.

 

Q: Citation Style

• What is the best method of sourcing for your systematic review?

A: It’s up to you, just be consistent!

 

Q: “Shelf-Life” of a Systematic Review

• Do systematic reviews have an expiry date? In fields with high publication rates, is it possible that a systematic review can, over time, fail to include a considerable proportion of relevant literature? How long might that take?

A: Great question! Once researchers conduct a systematic review, it’s not uncommon for that to become their “research baby” which they then tend to and care for as the literature continues to grow and evolve. So you may see researchers do an update on a previously-conducted SR (for example, 5 or 10 years down the line) to see what new literature has emerged on the topic – this is why documentation and transparency is so important, as the researcher must be able to recreate the search exactly as they did the first time!