Problem with the Climate-Control: can't seem to shut it off.

When I turn the Climate-control off, the front passenger side would start to have loud air-blowing sound - even though no air comes out, only the sound; turn on the climate-control and this sound would go away - only now there is the sound of the A/C.
I just want some quietness sometimes, but the air-blowing sound is always there, one way or another.