Hard to say without looking at the complete setup - and probably just a side-effect of the underlying issue. The question is, why did you have such high RTTs? That already points to a different cause.

I would even argue that NODELAY for a VoIP solution makes no sense - why are you even using TCP instead of UDP in the first place?