on the wearing of hats in the office

here’s a question, is it appropriate to wear a hat in the office?

yesterday one of the office admins gave me a light scolding about the fact that i have a habit of wearing a hat at my desk and around the office, but i’ve never thought twice about wearing it. i’m comfortable in my hat and i think it looks good so why not?

to be clear, the standard of dress most people abide by is pretty low, cotton dockers (or jeans) and polo shirts is the staple for most men on the floor i work on. me, i tend to wear nice jeans with button up shirts (ok, i’ll come clean, they’re usually pretty wrinkled) with the occasional decent tie. maybe one or two people in a hundred wear good pressed shirts and dress pants.

in that environment, is it such a faux pas to wear a hat? the office admin told me it was disrespectful to wear a hat, but i don’t get it, how so, it’s just a hat?