Skip to Main Content

Research Guides

Framework for AI Literacy

Evaluating AI Tools Critically

Students should be equipped to assess the credibility, accuracy, and broader impact of AI tools: 

  • Promote critical thinking by having students analyze the limitations of AI, including outdated training data and potential biases. 
  • Design assignments that are less susceptible to AI-generated content by emphasizing: 
    • Process over product: Use scaffolded, step-by-step assignments. 
    • In-class work: Break large tasks into smaller, manageable parts. 
    • Personal reflection: Ask students to incorporate unique insights or experiences that AI cannot replicate. 


Example learning objectives

  • Critically evaluate AI's potential impact on the environment, labour, privacy, copyright, academic integrity, and other important issues 
  • Contrast different AI tools; identifying their advantages, disadvantages, risks and limitations 
  • Compare the outputs of AI tools with traditional tools to determine their value and validity 
  • Assess quality and accuracy of scholarly sources provided by generative AI  
  • Identify and critique biases and/or inaccurate information generated by AI tools 
  • Verify the authenticity and faithfulness of citations provided in AI-generated summaries 

Example Learning Activity #1

Learning objective: Contrast different AI tools, identifying their advantages, disadvantages, risks and limitations 

Context: All disciplines/years of study  

Format: Lecture and Demo

Explanation: Address the fundamentals of GenAI chatbots and prompt engineering. Choose two different tools to compare eg. ChatGPT and MS Copilot.

Demo using each tool to generate an output from the same prompt. Compare the outputs from each and look for discrepancies, relevance, and inaccuracies. Discuss why the tools provide different outputs when using the same prompt. 

Guide the students through their own comparison of the two tools, using pre-selected prompts, and have them reflect on and discuss the relative merits or disadvantages of each tool depending on their intended usage (e.g. generating practice questions for study).

When comparing a tool with MS Copilot, discuss the University of Toronto’s licensed version’s additional security features and ask students to reflect on how that might impact their choice of tools. 

Example Learning Activity #2

Learning objective: Verify the authenticity and faithfulness of citations provided in AI-generated summaries. 

Context: All disciplines/years of study  

Format: Activity/Worksheet

Explanation: Ask students to take the top 5 citations generated in the summary by an AI tool and look them up in the library catalogue. Provide the following questions for them to reflect on and answer:

  • Can you access the material referenced? Are they cited correctly? If there are mistakes in the citation, refer to the library citation guide on how to accurately cite your sources. 
  •  Is citation faithful? In other words, how accurately and factually does it represent the cited work?  
  • Is the citation for an authoritative source?
  •  Have students provide an overall reflection of evaluating their AI generated citations. How well did their tool do overall in providing accurate citations and what were any limitations?

Example Learning Activity #3

Learning objective: Compare the outputs of AI tools with traditional tools to determine their value and validity. 

Context: Library workshop for graduate students

Format: 2-part scaffolded activity on worksheet

Explanation:  
Part 1: Guide students through an evaluation of a Boolean search string generated by ScopusAI. Ask students to answer the following questions on a worksheet:  

  • Are there too few or too many concepts? 
  • Are there missing synonyms in any of the concepts? 
  • Are the concepts included in each concept accurate?  

Part 2: Have students improve the Boolean strategy based on their observations and then use that search string to search regular Scopus. Discuss the differences between ScopusAI results and regular standard Scopus results with a neighbour. Have students write down one pro and one con they notice about the results. Have students rank their level of confidence in the recall of results.  

Take up this final activity with a discussion about the differences in the results and the differences in their confidence using the two different types of tools. 

chat loading...