Rest can make you better!

You're thinking about it too much. My body isn't just being lazy on rest days, its rebuilding. Resting is part of your training! On my rest days when I'm training for an event I rest but not mentally. I am always analyzing my training routine and looking for places to improve. The workouts themselves only provide the stimulus for change. The change itself (hopefully an improvement in fitness level) takes place during the periods between workouts.