Responding to salam with a hand gesture is disliked, but it does not invalidate the prayer. See: Maraqi al-Falah p.351.
If the previous answers are not appropriate, please send your question to the Mufti via