I like your comments but question your statement about how women can wear men’s clothes without a problem so why not vice versa?
In my opinion, the trouble with this line of reasoning is that we cross dressers do not just limit our activity wearing women’s outerwear. Rather, we go for the full transformation with wigs, makeup, mannerisms, etc. And you can bet that women wearing trousers are not sporting boxers or tighty whities underneath – again, not so vice versa.
So I think this is crucial difference in “cross dressing” as practiced by the two sexes. Just my opinion, of course.