I don't think that's it. For me sometimes my phone compass is (roughly) right, but sometimes points in a completely wrong direction, for example 90 degrees off. It calibrates after a while when I start walking but I guess this is what OP had in mind, not a few degree difference.
iPhones can regain the compass fix if you move the phone along a figure eight path while holding it flat and level. Worked pretty well for me in the past.
