Hi Am,
I'm not on milk diet yet, but have done the Cambridge Diet in the past that consisted of Milk shakes. When I first started it I didn't go to the loo for 7 days. I think that was just my body adjusting to it.
I always thought of it that if you are only taking in liquids and nothing soild then, yes you are likely to go to the a poo less often.
Also remember the quantity is less than when you are eating so there is less to pass.
As long as you are not feeling uncomfortable and don't feel constipated I wouldn't worry to much. When I was on a liquid only diet I never went to the loo daily, maybe every 2-3 days.
I'm sure one of the milk diet experts will correct me if I'm wrong.