Probably because when watchfaces were invented, the hour was much more important than the minute or the second.

People wanted to look up to the town hall or church clock and see what hour it was, maybe quarter hour.

Maybe keeping it more distinct from a distance during daylight hours was more important?