> But of course, doing that requires application level support, or at least low-level hooks into the desktop rendering system to extract video streams and treat them differently from everything else.
That's exactly what RDP does on Windows! I don't know the extent to which controls it supports rendering on the client, but the concept is that RDP is aware of things like text, buttons, title bars, etc., and is able to tell the client things like "draw a button at (20, 140)" instead of sending the raw pixel data. With enough engineering effort (and some standardization on the application UI side) I totally think Microsoft could make RDP the protocol of choice for virtual desktops.
It makes total sense; OTOH I'm kind of cracking up here because isn't this what a web browser has become? A portal for viewing a remote application state, that can be serialized into a blob of semi-structured XML and JSON?
It's hard to imagine that retrofitting such a serialization onto the total wild west that is desktop applications could be any more successful or consistent than what the web has already got.
> but the concept is that RDP is aware of things like text, buttons, title bars, etc
The fun bit, at least as I understand it, with the move to more and more Electron based apps, this is somewhat moot since those apps draw their own thing. There are still performance improvements RDP brings over a simple "blit everything" protocol, but they're smaller and smaller.
My biggest complaint with RDP in general is how many Windows apps react poorly to changing resolutions and DPI. I'm practically an expert now in rescuing windows that have decided "I was over at here on the right side of your ultra-wide monitor, so I'm still over here in this no man's land on this tiny RDP resolution."
That's exactly what RDP does on Windows! I don't know the extent to which controls it supports rendering on the client, but the concept is that RDP is aware of things like text, buttons, title bars, etc., and is able to tell the client things like "draw a button at (20, 140)" instead of sending the raw pixel data. With enough engineering effort (and some standardization on the application UI side) I totally think Microsoft could make RDP the protocol of choice for virtual desktops.