Can I use stainless steel rivets instead of plastic push pins?

Hello good community! I'm not sure if this is a model thing, but my plastic
fender inner panels keep falling. I've had 3 replacements already. The plastic
push pins (OEM pins) fail often and I have to replace them else I lose the
whole thing. I was thinking of using stainless steel rivets (on steel body)
instead of the plastic pins. Is that wise or am I going to regret it?
Thanks!