From a usability aspect at least, you could keep the same distortion until the interaction ends, and then tween the distortion values to the new orientation over a second or two. Technically it'd be 'wrong' for a short time, but it'd be nicer to use.

Awesome work regardless though.