There’s already ways to give negative feedback on a comment. You can downvote or flag, or even just collapse a comment so it doesn’t bother you. It comes off as presumptuous when people point at rules, thereby pulling the thread into a tangential meta conversation.
Then all you get is people asking why they were downvoted. IMO downvoting should include a comment if one doesn't already exist sufficiently explaining your reason for downvoting.
That doesn't mean it needs to turn into a discussion if they defend themselves, but I think some indication of why is almost always warranted.