It's a good abstraction, because most of the time you care about perceived brightness. With gamma encoding, 255 looks twice as bright as 127. You are going to need to deal with that perceptual power law at some point no matter what representation you use, so you may as well pick the efficient one. It's probably not a good tradeoff to make everyone deal with a color scale that doesn't look right, and is inefficient, just to make intensity operations linear so that unwary programmers don't trip.