Alcohol contributes to far more intoxication-related violence than any other drug. If the violence of intoxication is to be a factor, there's no reason at all to ban marijuana, and every reason to ban alcohol.
Yes, I believe the link between alcohol and increased levels of violence is very, very obvious and not contested by anyone. The link between drugs and violence is just as obvious.
So what about drugs that aren't associated with violence? Someone really high on pot not only doesn't want to fight, they're probably not going to be very effective at it. Likewise, those high on heroin are harmless at the time.