How do I tell him that it isn't the right time?

You are not responsible for how another person feels. You are not the guardian of his feelings or his reactions. The only thing any of us can control in this world is our choice of reactions. So you do what you feel you need to do. Whatever his reaction is his choice. He will survive.