It became a habit for me due to attending a high school that had it as a dress code. We were required to wear polo shirts. (red, white or navy blue)
Most of my jobs I've had require business casual attire. All the years scrapping to get by, having "work attire" and "public attire" was too expensive so I just had "attire" universal the stuff I wore to work is the stuff I'd wear all the time anyways.
Also being a fat bastard with "teh dubblechin", and no muscle tone at all in the shoulder area. Polo shirts collar area gives me an actual fake appearance of a neck, I feel I look a bit better in a polo shirt than I do a T shirt.