I say, and with God's success: He must leave in this case, but if he spends a portion of the time leaving his retreat, it invalidates his obligatory retreat which is the vowed one, and he must make it up. This is different from leaving for a person's need; because it is known to occur, it is considered an exception. However, in the case of the confirmed Sunnah retreat and the recommended one, it ends with his exit without an excuse. Refer to: Al-Tabyin 1: 351, and God knows best.