Audi S4 B6 (2004) does not crank or start. How to fix?

My Audi S4 B6 (2004) (bought in 2003) will not start when I turn the key. When I
turn the key really nothing happens: the starter does not react and the car really
doesn’t produce any sound. The lights on the dashboard and the head lights do
work as they should. This issue only happens half of the times that I start the car. If
the car doesn’t start and I wait for 5-10 minutes (key in the off position), the engine
always does start. Another (possibly related) issue: when I turn the key to start the
engine in the cases that it does start, there is sometimes (a lot of) time (up to 20 or
more seconds) between the moment that I turn the key and the moment that the
starter reacts. The battery is new, so that cannot be the problem. Can anyone help
me with this/know what the cause is? Thank you!