check engine light/purge valve

My check engine light came on and the mechanic tells me I need a new
purge valve.  While waiting for the part, the check engine light is no longer
on.  Do I still need to do the repair?  Is this a warning that only goes off once
and then the mechanic resets it or is it a sensor that should still be on if the
problem is really a problem.  I am wondering if the gas cap was not
tightened (by me) properly?