There's a time window for downvoting, once it has expired nobody can do it. I don't remember what the period is, maybe 12 hours.