Calling it a red flag may have been too harsh. It's certainly not an immediate no.
However, like it or not, it is a signal because it means you deviate significantly from the mode of the distribution. And a sober application of Bayes suggests that if anything, all else equal that signal is a negative one.
I would go as far as to say being this hyper-focused on clothes rather than if the person is sociable and competent is a red flag itself. It is rather superficial. Vague platitudes about "culture" might get thrown out, but are we engineering and building things or are we putting on a fashion show?
Being hyper judgemental about the clothes people wear isn't productive