I say, and with God's guidance: You should reconsider your life with him so that you treat him well, adorn yourself for him, take care of him, and make him your top priority in life. As for neglecting him, it is not permissible according to Islamic law; it is a negative and incorrect way of thinking. And God knows best.