Why should people be made to feel like criminals for a personal choice?
You could say the same thing about owning illegal firearms. My uncle is a responsible man and a lifelong Liberal voter, but was disgusted when Howard forced him to hand in his rifles. It's a minority that causes the serious problems. 80/20 rule?
Of course there are users who consider themselves to be responsible and in control of their hobby, but the conduct of a minority ensures that all drug use is stigmatised. Plus, a certain amount will fall by the wayside, even through "responsible" use, due to a predisposition to addiction, psychosis etc. Are they to be regarded as collateral damage so that you can enjoy your fix?