The realtime API can be used to call tools [0], but I agree with your general point on the flexibility of working directly with text.

[0] https://github.com/openai/openai-realtime-agents