When making a Podcast, using audio affects can add depth to the storyline and listening experience. However, reviewing the Creative Commons guides to the reuse of audio for podcasting would be a good place to start, before adding audio produced by other individuals.
Creative Commons > Podcasting Tools
For more information about the legal usage of audio files you find and whether you can use them in your podcast, visit the Creative Commons Wiki on "The Podcasting Legal Guide"
Audio Libraries with free for use audio files available:
British Library Sounds - (Legal & Ethical Usage Statement, British Library of Sound)
Internet Archive - Community Audio
Wikimedia Commons - Audio Files