It depends what and where you wear it IMO. If you wear an actual guernsey out of the house other then you're heading to a game you can come across as a bogan. I wear my nice Essendon polo sometimes (when going to watch my mates play sport) for example and don't see anything wrong with it. If you're happy wearing it and are a passionate support who gives a toss.