I say go for it. I can speak from my own experience that most individuals will notice, and those who do will not care. I regularly venture out in feminine attire while proudly presenting as a man. In fact, as I type this, I am sitting at Panera wearing gingham sundress and wedge sandles. I was complimented on my dress, and I am clearly a man (military short hair and neatly trimmed goatee both of which of my wife adores).