Yes, that's a very good point. The perfect example of this was Emanuel Lasker. He was an extremely talented mathematician, noted for work in commutative algebra, but he poured the vast majority of his efforts into chess. He was world champion for 27 years, easily the most dominant player of his era and amongst the best ever.

Just imagine if mathematics had been his passion and chess the side hobby!

http://en.wikipedia.org/wiki/Emanuel_Lasker