I do it all the time. Get that sweatshirt off and show the world. My experience is that people are too wrapped up in their own lives. Any comments I have had have all been complimentary.
I only wear womens jeans/pants now. Fit much better than shapeless bland male ones. Womens sneakers/shoes, tee shirts, sweatshirts, belts, jackets…
Loving it… fun fun fun.
When I do get the need to be overtly feminine-dress, skirt, heels etc I do get the hair and make up on, feel it completes the more formal look.